Ubuntu Wiki

• Launchpad entry: https://launchpad.net/products/malone/+spec/bug-workflow

• Created: 2006-11-09 by MatthewPaulThomas

• Contributors: MatthewPaulThomas, SimonLaw, BjornTillenius

Summary

In order to make the workflow of a bug more obvious, some statuses will be renamed, and some new statuses will be added. We won't force people to use a certain workflow, but we'll guide the user by only showing the relevant triaging statuses to bug triagers.

See also HighlightBugTriaging, which deals with making it more obvious how to triage bugs.

Use cases

The projects that use Launchpad for bug tracking can be classified into four broad types.

• Small upstream: Bugs typically go directly from reported to fixed, and there is no formal verification of fixes. Releases early and often (usually), but there is often still some lag between fixes being committed and appearing in a release. Example: Gobby.
• Large upstream: Large enough that vetting and clarifying new bug reports is a useful volunteer task in itself. Releases may have alphas and betas, so whether a bugfix has been "released" is a difficult question. There may be security fixes for past releases. Example: Firefox.
• Web application: A production server is updated semi-regularly from mainline or from a branch, with a combination of rollouts and cherrypicked urgent fixes. Therefore the lag between a fix landing in mainline and appearing on the server is highly variable. Example: Launchpad.
• Distribution: Includes software from many diverse sources, so the developers often do not get involved in fixing lower-severity bugs. Has releases every few months or few years, with security and other critical fixes provided for past releases. Example: Ubuntu.

User stories

Amanda develops a simple program and she wants to use Malone as her BugTrackingSystem. This is a small upstream project, she does all the triaging of the bugs, as well as fixing them. Her project's workflow looks like Figure 1, below.

Figure 1.

Ben is a developer for a large software project, a GNU/Linux distribution, or a website that uses Malone as its bug tracking system. He has a triaging team that do all the triaging work, and a QA team that makes sure that someone is assigned to fix a bug, and that the bug is indeed fixed. His distribution's workflow looks like Figure 2, below.

Figure 2.

Ben's could also have been working for a large upstream or a web application, which would have a similar workflow.

Catie is triaging bugs and she encounters many duplicate problem reports related to the X server breaking. She wants to know whether the bug is in a stable release, or in the development version.

Dimitri wants to become a developer and is interested in KDE bugs. He needs to find unclaimed bugs, and it should be clear which statuses he should use

Also see: https://blueprints.launchpad.net/launchpad/+spec/highlight-comments

Bug statuses

Rationale

Now that Ubuntu and a variety of upstreams are using Launchpad for bug tracking, we need to revisit the bug statuses available to see whether they allow and encourage useful bugtracking processes.

Currently a bug report has seven possible statuses.

• Unconfirmed: Initial status. Has not been looked at by a developer or QA volunteer.

• Needs Info: Needs clarification from the reporter, or someone else who can reproduce the problem.

• Rejected: Either it is not a bug, or it is a bug that will not be fixed in this particular place.

• Confirmed: Contains enough information, presented well enough, for a developer to start fixing the bug.

• In Progress: Currently being fixed.

• Fix Committed: A fix has been included in the code, but this code is not necessarily available in a released version.

• Fix Released: The bug is fixed in a released version.

(Whether a bug is a duplicate is orthogonal to its status.)

Current plans are to introduce a "Won't Fix" status, which would mean that the bug is valid but is not going to be fixed in that particular place. (For example, a bug in Windows that cannot be worked around by an application it affects; or a bug in Ubuntu that is never going to be fixed for Dapper.)

Problems identified with these statuses:

• The meaning of "Confirmed" is unclear. Ideally it should mean "contains all necessary information for a developer to fix the bug", but many people interpret it as "has been experienced by more than one person".
• "Needs Info" is ambiguous. Some people treat it as "needs clarification from the reporter", while others treat it as "needs information from someone other than the assignee" or "I have a question".
• When a bug should be moved from "Fix Committed" to "Fix Released" is unclear. Should it be done when a new package is uploaded? When an alpha is released containing the fix? When an OS version is released containing the fix?
• There is no process for verifying bugs, for those projects that want to do that.
• The statuses are too wordy generally.

Design

• Launchpad should have these bug statuses for triagers:

  • New: This bug report needs an experienced user or developer to evaluate the report and ensure it has enough information for a developer to reproduce the problem. This is the first state a report is in when filed and nobody has claimed responsibility.

  • Incomplete: The bug report doesn't contain the necessary information. It needs clarification from the reporter (and/or possibly other people) to provide sufficient information about the problem.

  • Invalid: Not actually a bug. It could be a feature request, designed behaviour, user error, or any number of explanations.

  • Triaged: An experienced user has verified that the problem described by this bug report is a valid bug. It also means that there should be enough preliminary information that a developer can reproduce or isolate the bug.

* And Launchpad should have theses statuses for developers:

• Won't Fix: A valid bug, but it should not be fixed by this particular project. It may be fixed in another release or forwarded to another project.

• Todo: The QA team or developer has confirmed that this is a valid bug, and that it should get fixed at some point.

• In Progress: A developer is fixing this bug.

• Committed: The fix for this bug is available somewhere, but isn't yet available in an official release.

• Fix Verified: In software that has a verification process, the fix for this bug has been tested to not break existing functionality and has fixed the appropriate faults. This will be an optionally state that can be enabled on a per-project basis.

• Released: launchpad-bazaar, buildd, or a human has noticed that the bugfix has been included in a release.

A triager is often not capable of deciding whether a bug really should be fixed, and he has no need of marking a bug as "In Progress", or "Committed", etc., so there's no need for him to have the option of setting the bug to a status he shouldn't use. Only priviliged users should be able to set the status to the bug to any of the developer statuses. For small projects, it has to be possible to allow anyone set any status, though, since they often don't need that level of control.

The Registrant for a distribution or a product should be able to choose whether bugfixes in that project need verifying; by default they should not (to make life easier when registering small upstream products). The "Fix Verified" state should be available only for projects that have chosen to require verification. If a project that required verification changes to no longer require it, all bugs currently in Fix Verified state should change to Committed (without sending any notification mail).

Implementation

Schema changes:

• Introduce "bug fixes need verifying" flag for distributions and products.
• Change bug statuses:

  • rename "Unconfirmed" -> "New"

  • rename "Rejected" -> "Invalid"

  • introduce "Won't Fix"

  • rename "Needs Info" -> "Incomplete"

  • rename "Confirmed" -> "Triaged"

  • rename "Fix Committed" -> "Committed"

  • introduce "Fix Verified"

  • rename "Fix Released" -> "Released".

Code changes:

• A checkbox for whether a project's bugfixes should be verified.
• Code that changes "Fix Verified" bugs to "Committed" if verification is turned off.
• Code that maps old names to new ones, so that existing links to searches are not broken.
• Add a QA team to products and distributions. If a QA team is set, normal users can only use a restricted set of bug statuses.

Searching for bug contacts

Rationale

Currently, development teams use various methods for tracking unassigned bugs. For instance, the GNOME desktop team assigns any GNOME desktop bug to their own team. But the kernel team does not want this, because it generates too many e-mail notifications.

We need to have some way of generating a list of those bugs that nobody is responsible for fixing, but that are related to a particular team or interest group.

Design

Each package in a distribution has a bug contact, which is usually the maintainer or a team of maintainers. Launchpad should allow searching for all bugs in packages with a particular bug contact, in the same way as it currently allows searching for bugs by reporter and/or assignee. Any such bug that is assigned to Nobody needs a volunteer to fix it.

This should be presented as a new "Bug contact" field in the "People" group, for the Advanced search in Malone.

Implementation

The +packagebugs page already has code for getting all the packages a Person is a bug contact for. This code will be moved to the general bugtask search code, which will be extended with to accept a bugcontact parameter and use the +packagebugs code.

Version hint

Rationale

When ResolutionForBug57153 was occurring, we noted a lot of duplicates in bugs being filed against the X server. However, triagers assumed that this was in the development (Edgy) version of X and not the stable version.

Design

Either the Description field should encourage people to enter the version/release where they experience the bug, or there should be a separate Version field. Which of these is appropriate will be determined as follows.

Implementation

We will start by sampling bug reports created with the current interface, measuring the proportion that should contain version numbers but don't.

Then we should tweak the bug-reporting form by encouraging people, in the caption for the "Description" field, to include the version(s) of the package and/or the distribution where the bug occurs. A month later, we will sample bug reports and measure again.

If that does not improve the situation enough, we should add a separate "Version or Release:" field to the bug-reporting form. The text entered in this field should be prepended to the description when the bug report is created. This will not involve any schema changes; it will be done entirely in browser code.

"Enough" will be defined at the discretion of Simon Law.

Potential drawback: The proportion might be increasing or decreasing over time for other reasons, in which case a comparison between only two or three datapoints will be misleading.

Future work

• Using bug contacts to group bugs that people might be interested in is a bit non-obvious, and doesn't allow overlapping groups. Perhaps in the future we should consider having package groups, so that (for example) Muine can be in both a "Mono" group and a "Multimedia" group, with bugs in it fixable by those watching either. (Cf. Google Code Hosting's use of project tags.)

Notes

1. Isolate the following categories of bugs, which should be addressed by Ubuntu developers:
   • Bugs which are specific to Ubuntu

     • Rationale: Either we created these bugs, and so they are our responsibility, or they are only manifest in our environment, in which case we are best equipped to diagnose them.

   • Other bugs which are very severe or relate to core goals of Ubuntu

     • Rationale: These bugs justify the direct attention of Ubuntu developers because they jeopardize our mission of producing high-quality releases

2. Pass on all other categories of bugs to the appropriate party:
   • Complete bug reports which affect upstream should be made the responsibility of upstream (forwarded)

     • Rationale: The upstream community is better equipped to fix these bugs, and the fix should eventually go upstream anyway, so it's best for it to be fixed there in the first place

   • Incomplete bug reports should be made the responsibility of the reporter by requesting further information ("Needs info")

     • Rationale: Bug reports must be reasonably complete in order to be diagnosed by a developer. Because the development team is relatively small compared to the community of bug reporters, the development team must focus on complete reports.

   • Improper bug reports should be rejected

     • Rationale: Some bug reports are hopeless, for example if the reporter is unresponsive to questions, and must be abandoned in order for the bug system to remain manageable.

Another view

I've many thoughts on this, and am bulleting to keep things brief. If anything is less than clear, just ask!

Assumptions

• Ubuntu process works pretty darned well: Needs adjustments, not radical change.
• These comments apply to Ubuntu process only, not generic "want to use Launchpad for project X" (though there should be no conflicts between them)
• Some labels are unclear, but status reasonably well defined
• When a bug is opened, it is obviously new and often incomplete/unconfirmed; we don't need two states to reflect that

• Verified label or status is not required: If a fix is released and doesn't work, we'll find out soon enough

• We need a useful indication that a fix didn't (didn't fix, that is, or regressed)
• Anyone can triage, providing sufficient instructions exists
• Assignment to a team, product, package or individual requires "insider knowledge" or a "degree of authority"
• Anyone can mark as fix released or rejected, providing sufficient evidence exists
• package hints are hints, and may be overused or over-relied-upon, as is (complicates search results when a bug appears multiple because it is confirmed in one, unconfirmed in anbother, etc.)
• reliable statistics require we know where the problems actually were, and also who (person or team) fixed 'em or managed 'em

Proposal

• Rename Unconfirmed to Incomplete

• Scrap Needs info

• Rename Confirmed to Well-founded - this is much closer to what we actually mean

• Introduce a bad fix state (regress or didn't fix or fix didn't take

• Introduce controlled state transitions, some of which require "authority", some of which require particular fields be filled

Some acronyms

State definitions/explanations

Workflow/state transitions

All bug reports start in Incomplete.

Figure: Proposed flow

Notes on the diagram

1. There is enough information to reject the bug
2. The bug report was rejected in error (open for debate, TBD whether it goes 1 or 3)
3. There is EITWTP
4. WKWTPI - assign to package, possibly to a person
5. Refer to fix committed, above
6. The problem has been fixed by a generally-available update
7. Refer to bad fix, above
8. "Bad fix" was incorrect - the observed problem was not a result of the fix, return to fix released
9. "Bad fix" may be correct, we need more information
10. "Bad fix" is correct, we have EITWTP
11. Just like #1. In other words, only if there is no bug. We DO NOT reject if it has misassigned or directed to the wrong package or person, we redirect. There is no state change, we just redirect. Assigning to the wrong package/team/person is not a reason for rejection, there is still a bug.

Other thoughts

• Rename rejected to not a bug? If it is a bug and isn't an Ubuntu bug, then either it is something Ubuntu wants to track or it isn't. If it is, the report is open and assigned (and not not a bug) or it is marked not a bug. In either case, tracker information pointing upstream/elsewhere is provided, either for our records or FYI.

• There is some overlap between assigned and package: Once it gets to the package WTPI, should we leave it to the package team to decide how to work the problem? Perhaps assigned should be replaced with "HTPWTP" (how the package works the problem): It would be up to the package team to fill it in with a person's ID, a package-specific process step, etc.

• Another status that might be useful is Monitoring for close (better name required). We would use this to mark reports that we think we can close, but that we're not quite ready to close yet. For example, we'd use this for bug reports in packages/products that have complex interactions, where the symptons were experienced widely over a span of time, but the symptons seem to no longer appear. We want the bug in an open state but one that the devs can ignore safely - the triagers will monitor and close after some time goes by (like needs info today) or after enough reports of asymptomatic behavior have been received.

Unresolved issues

• There are currently three conflicting proposals in this specification.
• How exactly do we control who should be able to set all the statuses?

Maybe add a QA team, that can use all statuses, as well as allowing the assignee to set the bug to any status? Do we need to add a developer team as well?

• "Committed", "Fixed Verified", and "Released"? That's a lot. The benefit of separating "Committed" and "Released" is that "Committed" bugs still show up in bug listings, so people are less likely to report duplicates of recently-fixed bugs. Is it worth the complexity? I suggest not, especially since the meaning of "Released" is ambiguous.

  • Arguably, we don't have an equivalent "Committed" state for "Invalid" which is why rejected bugs are often duplicated.

• Ten statuses is too many. Each might seem perfectly justified by itself, until you realize there are ten overall, and ... holy crap, ten bug statuses.

• What if a "triager" accidentally changes a bug report from one of the statuses limited only to developers, to one of the statuses allowed for triagers? Will it be impossible to change back?

• Compare with MozillaTeam/Bugs/States.

-- MatthewPaulThomas

CategoryBugSquad