Review of Karmic

Developer approach

In Karmic we changed the approach for developers to concentrate more on fixing bugs than on triaging new ones.

This was overall a success: The entire Ubuntu team in total (including community) increased bug fixing by 20% (see bugs fixed in jaunty and karmic. Canonical desktop team members got an increase as well.

So in total we had a good improvement, and given that Jaunty was a QA cycle and in Karmic we did a lot of upstream-ish work and did not really "mean" to fix a lot of bugs, the outcome was very satisfying.

Testing/Reporting

Many bugs only get reported after release (e. g. hardware/driver specific problems). This is the usual chicken-egg problem of too few testers before release.

Bryce reports that X driver testing is hard: most developers use Intel hardware because it is better supported. Then non-developers finally test NVIDIA upon release.

In Karmic we disabled the +filebug Launchpad page for non-ubuntu-bugcontrol-members, which led to richer bug reports, but not necessarily better descriptions nor fewer bug reports.

Things that take time for Desktop developers:

• Duplicates that cannot be automatically marked duplicate, because they have wildly different titles. These indicate the reporters are not finding the existing reports.
• Bug reports from prolific posters of low quality bugs
• Dead bugs that have been fixed but haven't been closed (clutters up bug list).
• Bug storms from automatic duplication (i. e. everyone getting noticed everytime there is a duplicate then people doing a reply-all to try and unsubscribe/complain).

Bug report quality

There is a large difference between true defect reports and support requests which are often "something happened, how do I actually write a useful defect report?". Most incoming reports are unfortunately in the latter class.

It is currently hard to separate high-quality from low-quality bug reports, as well as seeing bugs which affect a lot of people.

Expectations from different groups

Value of bug reports:

• Users: support tool
• Developers: improve Ubuntu, pool of potential issues to work on
• Marketing opportunity to help sell Ubuntu
  • phone-a-friend to have your bug reproduced/confirmed or getting help for (connecting to local community)
  • look at these other bugs about this package to try and confirm them to help out
• Expectations from both sides are not encoded anywhere

Goals for improvements

1. Solve scalability issue
2. Reduce incoming bugs
3. Higher quality bug reports

Changes

Workflow

• Disable menu item for reporting a bug in the stable release; only leave it for the development release
• After a new Ubuntu release, set new bugs to incomplete and ask whether it's still an issue in the newer version; this goes hand in hand with automatic time out. This should be done on a per-package basis with a script invoked by the package maintainer.
• Dynamic is against the core team keeping up with incoming bugs, so stop trying.

Apport

• Auto-subscribe QA team by the apport retracer once it gets the tenth duplicate, for creating bug patterns.
• Change apport symptoms and appropriate package hooks to produce default bug titles, so that Launchpad's "proposed duplicates" becomes useful for bugs.
• Marc Tardiff to provide list of types of problems that we should make symptoms for.

Launchpad

• Automatically time out incomplete bugs without response for Ubuntu which do not have an assignee (needs to be enabled in Launchpad). This would work better with a new "Expired" state (which can't be set from the web UI), but "Invalid" will do for now.
• Change Launchpad bug interface to setting expectations and next steps (after a user has submitted a bug) (deryck)
• Improve/fine tune parameters for "bug heat".

Community

• Generalize hug day concept and involve the community more: "15 minutes a day ... please triage these three bugs"
• Adopt a package for triaging, give triagers a sense of ownership
• The qa team discussed this topic on the review of the bugsquad mentoring program.