Behind the Scenes with Zimbra Quality Engineering

Zimbra 8 is the first major Zimbra release that embodies new practices around quality engineering. We thought we’d shine a light on these changes.

While quality has always been important to the development process, quality processes are now more formalized than in the past.  Starting with the Zimbra 8 release, our major release processes include:

• Complete localization and security testing as soon as feature development is completed
• Two development milestones focusing on quality for engineering and quality teams
• A quality focus in feature specifications

Expanded test coverage

The QE team runs both automated and manual tests against the code. We’ve been steadily increasing the test coverage over recent releases:

• For Zimbra 8, we ran 21% more tests than with Zimbra 7.
• We’ve increased the number of daily tests for the web client by 207%.

The team continues to develop tests, focusing on the highest severity and priority issues.

Release schedules and processes

There are many maintenance releases between the major releases. Each needs the same focus on quality – but there’s often less time for testing.

We’re working hard to balance the need for fully testing new features with the need to get new features in the product quickly.  One way we’re doing this is by moving to a quarterly release schedule, making critical fixes available by patches between maintenance releases.  This lets us focus getting features onto the schedule without compromising quality and stability.

Tracking support cases/bug fixes

We’re carefully tracking the bug fixes connected to support cases, for two reasons:

• We prioritize bug fixes connected with support cases.
• Tracking the overall trend helps us measure the effectiveness of our QE efforts.

The good news is that we’re seeing a downward trend in bugs attached to support cases – particularly in 2012. As of the end of Q3, fewer than 15% of support cases are connected with bugs.

Nobody’s perfect, of course.  You keep using Bugzilla to report those bugs that managed to get by our QE team, and we’ll keep trying to find them before you do.