BugWorkflow

Summary

Goal

Maximize the amount of bug fixing that occurs per unit time invested in bugs management by desktop team engineers.

Rationale

The current situation is:

  • Many desktop team engineers are swamped with bug reports. That is, there are more bug reports in their areas of responsibility than they can reasonably address.
  • The desktop team has a history of addressing each and every bug report.
  • Bug reports are a vital source of feedback regarding the quality of the product.
  • Users have an expectation that a bug report will result in a timely fix, and may even expect support via their bug report.

The goal state is:

  • Desktop team engineers can meet their obligations regarding bug triaging.
  • Desktop engineers can spend more time fixing bugs when appropriate.
  • End users have appropriate expectations regarding the impact of their bug reports.

User stories

End User

A user encounters a bug in program Foo. Based on previous experience they go to launchpad.net to log the bug. When click the link to add a bug, they are shown instructions to use ubuntu-bug. They run ubuntu-bug which gathers data relevant to their symptoms and logs the bug for them. When the bug is logged they see a message telling them that the bug may be triaged, but may not be fixed, and also tells them where to go for support.

Triaging Tool

seb128 starts work in the morning. He fires up an app that chugs away on launchpad loading bugs into a list in the areas of GNOME that he works on. He gets some coffee and works on email. When the list is finished loading, he is able to look over the list and select bugs in batches. He can click buttons to set the bugs into different states, such needs info, won't fix, etc.... When he is done, he knows that he has triaged all the new bugs in his area.

Doing an Upload

asac uploads a firefox security fix. An hour later he starts getting bug mails about a regression some users experience. The mails are marked as being related to his recent upload (reason=upload watch). He triages the bugs. He gets similar bug reports for the next four days. After four days, the emails stop coming related to the upload, though bug mails are sent in the normal fashion.

Assumptions

  1. Being more or less forced to use ubuntu-bugs will decrease the number of incoming bugs, and will increase the quality of the reports.

Design

A multi-pronged approach is being used to make progress towards this goal state:

  1. The launchpad team is changing launchpad so that it helps users use ubuntu-bug to report bugs. This should result in fewer but better quality bug reports.
  2. pitti is enhancing ubuntu-bug to be "symptom based" (see desktop-karmic-symptom-based-bug-reporting). This should result in even better bug reports.

  3. Work with the launchpad team to add verbiage to launchpad that could help set the users' expectations regarding bug reports.
  4. The launchpad team will add an "expired" state to older bugs, reducing the noise in the system.
  5. The desktop team engineers will limit bugs assigned to them to bugs which they can and will realistically work on, preferably in the current cycle
  6. Craft a tool that dramatically speeds up the bug triaging process.
  7. Craft a tool that "watches" for bugs in new uploads, allowing engineers to respond to regressions more quickly.

Implementation

Triaging Tool

This will likely be added to or derived from the pm-dashboard code:

This will require:

  1. a way to define a set of packages to query for bugs to triage
  2. A list view with enough information for assessing the impact of the bug
  3. A way to quickly select batches of bugs
  4. A needs info button that can edit bugs in bulk to need info
  5. A needs to be worked on button
  6. A would be nice to get worked on button

Needs Info

Clicking this button would set the status to "needs info". It would inject boiler plate "needs info" text into the comments.

Fix

Will set the bug priority to High and the bug status to Triaged, and will assign to the canonical-desktop-team.

Community

Will set the bug priority to Medium. Will not assign the bug. Will set the bug to triaged. Will add a comment that the canonical-desktop-team will not work on it, but would welcome help on the bug.

Triaging a Single Bug Task

Notice that the description is visible. The buttons "Need Info", "Community", and "Fix" will set the selected bug to the appropriate state.

  • single_select.png

Bulk Triaging

While multi-selected, the selected bug titles can be read, and the triaging buttons will apply to all selected bug tasks.

  • multi_select.png

UI Changes

  • pm-dashboard may need a different list view that includes description info
  • A way to multi-select bugs will need to be added to the list view
  • buttons for triaging will need to be added
  • perhaps a way to change boiler plate on the fly will be needed

Code Changes

  • new list view with more bug info and selectability
  • new tab pane with triaging buttons
  • ability to update bugs

Upload Watch

There is no UI for this per se. This is essentially a cron job that will run on rookery, and detect recent uploads, and direct bug mail appropriately. The key idea here is that after doing an upload, engineers must monitor their email for regressions related to the uploaded packages, and must sort those bug mails out from the bulk of other bug mails. This system will separate bug mails out as potential regressions if they are related to recently uploaded packages.

Code Changes

  1. a cron job that detects recently uploads
  2. a way to create a special subscription to bugs on packages related to those uploads
  3. a way to change the headers for those bug mails so that engineers can sort them seperately
  4. a way to unsubscribe after a set period of time, when the upload is no longer "hot"

Test/Demo Plan

Unresolved issues

BoF agenda and discussion

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

UDS Discussion Notes

pitti: Rick, would you mind drafting this? If not, I'm fine with doing that.

Note:

"Bug workflow for desktop engineers"

Problem according to Rick: "Too many freaking bugs (not code defects, just noise) with culture of addressing all reports (not feasible). This is causing burn out or supressing time to work on bug fixes"

  • Many engineers want to look at all bug reports, but can't
  • Too much time triaging, too little time for fixing, no way to know when you're done
  • You don't know when you have done enough, no queue to zero down
  • Users filing bugs have an expectation of them being fixed
  • Sound: triaging is done diligently, bug fixing is hard because of limited hw access

How do you mark a bug that I looked at it, but I want it off the new list?

  • Canonical team reviewed tag 'ct-rev' is intended for this
  • Assigning a bug to yourself is the way to put it in category

How can QA drive to 0?

  • You have lists and buckets that you can put it in (Getting Things Done)
  • One issue with doing this, like with tags:
    • You can't do a search excluding a tag, so 'dt-rev' isn't as useful as it could be
    • You can't get it out of your list
    • Searching for the absense of a tag is planned in LP

Seb's dream:

  • Open the new bugs list
  • Go quickly through the list
  • Categorize those as needinfo, need to be worked now, would be nice to get worked ("no time", "moreinfo", "want to work on")

Rick has a desktop tool for queuing multiple bugs for processing

Should also have a way that users can communicate with upstream quickly - then need to observe what is going on with communication

Bryce - several workflows

  • From upload forward, I care, don't care about bugs earlier than that

Pitti wants a way to have "gravity" that means that you can see key bugs quickly Workflow - regression watch after an upload (for particular version) Important that users use apport to make this work asac wants automatic temporary (3 days) package bug subscription for New bugs only

Bug upload-watch flow:

  • Do an upload
  • LP automatically sends you upload watch mails
  • Mail gets sent for new mails (activity?) on the package version with reason in header
  • Sponsor and contributor get added? How to sort it?
  • Prototype something using launchpadlib and the changelog mailing list(?)
    • This would not have the necessary e-mail header though

Upstreaming process of bugs:

  • It takes a long time for bryce and asac to forward bugs upstream
  • asac can't follow bugs and proxy communication
  • asac wants a strict policy of who can upstream to protect their time and our reputation
  • They want to know from us about our policies, can you cancel accounts, etc...
  • It's important to understand the throughput of the upstream wrt bugs so you don't overload them
  • robert_ancell: Only upstream if have a reproducible test case

How engineer's should prioritize bug work:

  1. Watch out/work on major regressions
  2. Work on bugs that are already assigned to me
  3. Already triaged bugs, that need to be worked on, upstreamed, etc.
  4. Triage incoming bugs

Must be honest about only assigning bugs to self that will be worked on; keep /+assignedbugs list clean Almost all x bugs are regressions - matter of degree needs to be determined

Need a "responded" state as opposed to Incomplete w/ Response - since this is not visibile anywhere in the web interface for launchpad

  • Use case? Incomplete w/ Response is a virtual status, only used in searches. Is it useful elsewhere?
    • - Users who have answered a bug sometimes get confused/frustrated that the bug is still marked Incomplete - Seems like "Answered" or "Replied" state might be better
  • When you are looking at a specific bug page there is no indication if it is Incomplete w/ or w/o response

If you don't have a triaging team, then the bug work flow related to state changes doesn't work for you

New: Not being looked at yet Incomplete: Waiting for more information from reporter(s) Invalid: Wont Fix: Triaged = enough information for a developer to understand the root cause and fix the issue (i.e. no more user discussion required) Confirmed = user has asked for all the information / another user also has the problem / Triager has been able to reproduce the issue

  • What is the difference between confirmed and triaged?
  • FWIW, the Launchpad development team have stopped using Confirmed. Bugs go from New to Triaged or Incomplete, etc.

In Progress: Fix Committed: Fix Released:

rickspencer3 thinks higher quality lower quantity bugs will be better

  • Problem: users commenting on bugs that are really different issues that they have
  • Solution: Require minimum karma level to comment on another bug
  • Solution: "archive" bugs as per the debian archivers
  • Lock comments/status to the bug reporter
  • Tell people not comment on bugs without apport collect

Apport and apport collect:

  • Will add data retroactively
  • Symptom based bug reporting will only work with (?)

Launchpad:

  • Change the link to "file a bug" link to tell people to use automated tools
  • Firefox file a bug plugin
  • Could users mark their own bugs as "wishlist"?
  • Smaller bug lists will encourage new community developers
  • lock bugs after a month of being closed: "This bug is closed, please open a new bug if you are still experiencing this after upgrading"
    • What about stable release updates?
  • Reporters should be able to set "wishlist" priority themselves

Can we mass close a large amount of bugs

  • launchpadlib Smile :)

Will add an "expired" state: LP bug janitor disaster

  • would affect bugs that are Incomplete w/o response and New old bugs

Roman Friesen input:

  • Make reporting of bugs more challenging
  • new bug page should be information for the user
    • what you can expect after they report a bug
    • can take time
    • could b required to provide additional information
    • Ubuntu is only a collection of software, so we have to wait until bugs are fixed upstream
    • One important point: bug fixing team is focused on critical applications( main and security bugs )
  • make thre process more challenging
    • not just one page, but steps, and critical information must be required, like please execute this command in the terminal to provide ubuntu version, package version, etc...
    • you can differentiate between users: collect points for users who are reporting bugs, can provide different ways to report the bugs, more challenging for new users, shorter for those proven to have quality reports


CategorySpec

DesktopTeam/Specs/Karmic/BugWorkflow (last edited 2009-06-17 16:08:12 by rick-rickspencer3)