Review of Karmic
Developer approach
In Karmic we changed the approach for developers to concentrate more on fixing bugs than on triaging new ones.
This was overall a success: The entire Ubuntu team in total (including community) increased bug fixing by 20% (see bugs fixed in jaunty and karmic. Canonical desktop team members got an increase as well.
So in total we had a good improvement, and given that Jaunty was a QA cycle and in Karmic we did a lot of upstream-ish work and did not really "mean" to fix a lot of bugs, the outcome was very satisfying.
Testing/Reporting
Many bugs only get reported after release (e. g. hardware/driver specific problems). This is the usual chicken-egg problem of too few testers before release.
Bryce reports that X driver testing is hard: most developers use Intel hardware because it is better supported. Then non-developers finally test NVIDIA upon release.
In Karmic we disabled the +filebug Launchpad page for non-ubuntu-bugcontrol-members, which led to richer bug reports, but not necessarily better descriptions nor fewer bug reports.
Things that take time for Desktop developers:
- Duplicates that cannot be automatically marked duplicate, because they have wildly different titles. These indicate the reporters are not finding the existing reports.
- Bug reports from prolific posters of low quality bugs
- Dead bugs that have been fixed but haven't been closed (clutters up bug list).
- Bug storms from automatic duplication (i. e. everyone getting noticed everytime there is a duplicate then people doing a reply-all to try and unsubscribe/complain).
Bug report quality
There is a large difference between true defect reports and support requests which are often "something happened, how do I actually write a useful defect report?". Most incoming reports are unfortunately in the latter class.
It is currently hard to separate high-quality from low-quality bug reports, as well as seeing bugs which affect a lot of people.
Expectations from different groups
Value of bug reports:
- Users: support tool
- Developers: improve Ubuntu, pool of potential issues to work on
- Marketing opportunity to help sell Ubuntu
- phone-a-friend to have your bug reproduced/confirmed or getting help for (connecting to local community)
- look at these other bugs about this package to try and confirm them to help out
- Expectations from both sides are not encoded anywhere
Goals for improvements
- Solve scalability issue
- Reduce incoming bugs
- Higher quality bug reports
Changes
Workflow
- Disable menu item for reporting a bug in the stable release; only leave it for the development release
- After a new Ubuntu release, set new bugs to incomplete and ask whether it's still an issue in the newer version; this goes hand in hand with automatic time out. This should be done on a per-package basis with a script invoked by the package maintainer.
- Dynamic is against the core team keeping up with incoming bugs, so stop trying.
Apport
- Auto-subscribe QA team by the apport retracer once it gets the tenth duplicate, for creating bug patterns.
- Change apport symptoms and appropriate package hooks to produce default bug titles, so that Launchpad's "proposed duplicates" becomes useful for bugs.
- Marc Tardiff to provide list of types of problems that we should make symptoms for.
Launchpad
- Automatically time out incomplete bugs without response for Ubuntu which do not have an assignee (needs to be enabled in Launchpad). This would work better with a new "Expired" state (which can't be set from the web UI), but "Invalid" will do for now.
- Change Launchpad bug interface to setting expectations and next steps (after a user has submitted a bug) (deryck)
- Improve/fine tune parameters for "bug heat".
Community
- Generalize hug day concept and involve the community more: "15 minutes a day ... please triage these three bugs"
- Adopt a package for triaging, give triagers a sense of ownership
- The qa team discussed this topic on the review of the bugsquad mentoring program.