BugWorkflow
Launchpad Entry: https://code.edge.launchpad.net/~rick-rickspencer3/pm-dashboard/sebified
Created:
Contributors: rickspencer3, pitti
Packages affected: all and none
Summary
Goal
Maximize the amount of bug fixing that occurs per unit time invested in bugs management by desktop team engineers.
Rationale
The current situation is:
- Many desktop team engineers are swamped with bug reports. That is, there are more bug reports in their areas of responsibility than they can reasonably address.
- The desktop team has a history of addressing each and every bug report.
- Bug reports are a vital source of feedback regarding the quality of the product.
- Users have an expectation that a bug report will result in a timely fix, and may even expect support via their bug report.
The goal state is:
- Desktop team engineers can meet their obligations regarding bug triaging.
- Desktop engineers can spend more time fixing bugs when appropriate.
- End users have appropriate expectations regarding the impact of their bug reports.
Approach
A multi-pronged approach is being used to make progress towards this goal state:
- The launchpad team is changing launchpad so that it helps users use ubuntu-bugs to report bugs. This should result in fewer but better quality bug reports.
- pitti is enhancing ubuntu-bugs to be "symptom based" (see ). This should result in even better bug reports.
- Work with the launchpad team to add verbiage to launchpad that could help set the users' expectations regarding bug reports.
- The launchpad team will add an "expired" state to older bugs, reducing the noise in the system.
- The desktop team engineers will limit bugs assigned to them to bugs that are targeted to be fixed in the current cycle.
- Craft a tool that dramatically speeds up the bug triaging process.
- Craft a tool that "watches" for bugs in new uploads, allowing engineers to respond to regressions more quickly.
User stories
End User
A user encounters a bug in program Foo. Based on previous experience they go to launchpad.net to log the bug. When click the link to add a bug, they are shown instructions to use ubuntu-bugs. They run ubuntu-bugs which gathers data relevant to their symptoms and logs the bug for them. When the bug is logged they see a message telling them that the bug will be triaged, but may not be fixed, and also tells them where to go for support.
Triaging Tool
seb128 starts work in the morning. He fires up an app that chugs away on launchpad loading bugs into a list in the areas of GNOME that he works on. He gets some coffee and works on email. When the list is finished loading, he is able to look over the list and select bugs in batches. He can click buttons to set the bugs into different states, such needs info, won't fix, etc.... When he is done, he knows that he has triaged all the new bugs in his area.
Doing an Upload
asac uploads a firefox security fix. An hour later he starts getting bug mails about a regression some users experience. The mails are marked as being related to his recent upload (reason=upload watch). He triages the bugs. He gets similar bug reports for the next 48 hours. After 48 hours, the emails stop coming related to the upload, though bug mails are sent in the normal fashion.
Assumptions
Design
Implementation
Triaging Tool
This will likely be added to or derived from the pm-dashboard code:
This will require:
- a way to define a set of packages to query for bugs to triage
- A list view with enough information for assessing the impact of the bug
- A way to quickly select batches of bugs
- A needs info button that can edit bugs in bulk to need info
- A needs to be worked on button
- A would be nice to get worked on button
Needs Info
Clicking this button would set the status to "needs info". It would inject boiler plate "needs info" text into the comments.
Community
Will set the bug priority to High and the bug status to Triaged, and will assign to the canonical-desktop-team.
Fix
Will set the bug priority to Medium. Will not assign the bug. Will set the bug to triaged. Will add a comment that the canonical-desktop-team will not work on it, but would welcome help on the bug.
Triaging a Single Bug Task
Notice that the description is visible. The buttons "Need Info", "Community", and "Fix" will set the selected bug to the appropriate state.
Bulk Triaging
While multi-selected, the selected bug titles can be read, and the triaging buttons will apply to all selected bug tasks.
UI Changes
- pm-dashboard my need a different list view that includes description info
- A way to multi-select bugs will need to be added to the list view
- buttons for triaging will need to be added
- perhaps a way to change boiler plate on the fly will be needed
Code Changes
- new list view with more bug info and selectability
- new tab pane with triaging buttons
- ability to update bugs
Upload Watch
There is no UI for this per se. This is essentially a cron job that will run on rookery, and detect recent uploads, and direct bug mail appropriately. The key idea here is that after doing an upload, engineers must monitor their email for regressions related to the uploaded packages, and must sort those bug mails out from the bulk of other bug mails. This system will separate bug mails out as potential regressions if they are related to recently uploaded packages.
Code Changes
- a cron job that detects recently uploads
- a way to create a special subscription to bugs on packages related to those uploads
- a way to change the headers for those bug mails so that engineers can sort them seperately
- a way to unsubscribe after a set period of time, when the upload is no longer "hot"
Test/Demo Plan
Unresolved issues
BoF agenda and discussion
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.
UDS Discussion Notes
pitti: Rick, would you mind drafting this? If not, I'm fine with doing that.
Note:
"Bug workflow for desktop engineers"
Problem according to Rick: "Too many freaking bugs (not code defects, just noise) with culture of addressing all reports (not feasible). This is causing burn out or supressing time to work on bug fixes"
- Many engineers want to look at all bug reports, but can't
- Too much time triaging, too little time for fixing, no way to know when you're done
- You don't know when you have done enough, no queue to zero down
- Users filing bugs have an expectation of them being fixed
- Sound: triaging is done diligently, bug fixing is hard because of limited hw access
How do you mark a bug that I looked at it, but I want it off the new list?
- Canonical team reviewed tag 'ct-rev' is intended for this
- Assigning a bug to yourself is the way to put it in category
How can QA drive to 0?
- You have lists and buckets that you can put it in (Getting Things Done)
- One issue with doing this, like with tags:
- You can't do a search excluding a tag, so 'dt-rev' isn't as useful as it could be
- You can't get it out of your list
- Searching for the absense of a tag is planned in LP
Seb's dream:
- Open the new bugs list
- Go quickly through the list
- Categorize those as needinfo, need to be worked now, would be nice to get worked ("no time", "moreinfo", "want to work on")
Rick has a desktop tool for queuing multiple bugs for processing
Should also have a way that users can communicate with upstream quickly - then need to observe what is going on with communication
Bryce - several workflows
- From upload forward, I care, don't care about bugs earlier than that
Pitti wants a way to have "gravity" that means that you can see key bugs quickly Workflow - regression watch after an upload (for particular version) Important that users use apport to make this work asac wants automatic temporary (3 days) package bug subscription for New bugs only
Bug upload-watch flow:
- Do an upload
- LP automatically sends you upload watch mails
- Mail gets sent for new mails (activity?) on the package version with reason in header
- Sponsor and contributor get added? How to sort it?
- Prototype something using launchpadlib and the changelog mailing list(?)
- This would not have the necessary e-mail header though
Upstreaming process of bugs:
- It takes a long time for bryce and asac to forward bugs upstream
- asac can't follow bugs and proxy communication
- asac wants a strict policy of who can upstream to protect their time and our reputation
- They want to know from us about our policies, can you cancel accounts, etc...
- It's important to understand the throughput of the upstream wrt bugs so you don't overload them
- robert_ancell: Only upstream if have a reproducible test case
How engineer's should prioritize bug work:
- Watch out/work on major regressions
- Work on bugs that are already assigned to me
- Already triaged bugs, that need to be worked on, upstreamed, etc.
- Triage incoming bugs
Must be honest about only assigning bugs to self that will be worked on; keep /+assignedbugs list clean Almost all x bugs are regressions - matter of degree needs to be determined
Need a "responded" state as opposed to Incomplete w/ Response - since this is not visibile anywhere in the web interface for launchpad
- Use case? Incomplete w/ Response is a virtual status, only used in searches. Is it useful elsewhere?
- - Users who have answered a bug sometimes get confused/frustrated that the bug is still marked Incomplete - Seems like "Answered" or "Replied" state might be better
- When you are looking at a specific bug page there is no indication if it is Incomplete w/ or w/o response
If you don't have a triaging team, then the bug work flow related to state changes doesn't work for you
New: Not being looked at yet Incomplete: Waiting for more information from reporter(s) Invalid: Wont Fix: Triaged = enough information for a developer to understand the root cause and fix the issue (i.e. no more user discussion required) Confirmed = user has asked for all the information / another user also has the problem / Triager has been able to reproduce the issue
- What is the difference between confirmed and triaged?
- FWIW, the Launchpad development team have stopped using Confirmed. Bugs go from New to Triaged or Incomplete, etc.
In Progress: Fix Committed: Fix Released:
rickspencer3 thinks higher quality lower quantity bugs will be better
- Problem: users commenting on bugs that are really different issues that they have
- Solution: Require minimum karma level to comment on another bug
- https://wiki.canonical.com/BryceHarrington/BetterBugs
- - Bryce, perhaps make this public?
- Solution: "archive" bugs as per the debian archivers
- Lock comments/status to the bug reporter
- Tell people not comment on bugs without apport collect
Apport and apport collect:
- Will add data retroactively
- Symptom based bug reporting will only work with (?)
Launchpad:
- Change the link to "file a bug" link to tell people to use automated tools
- Firefox file a bug plugin
- Could users mark their own bugs as "wishlist"?
- Smaller bug lists will encourage new community developers
- lock bugs after a month of being closed: "This bug is closed, please open a new bug if you are still experiencing this after upgrading"
- What about stable release updates?
- Reporters should be able to set "wishlist" priority themselves
Can we mass close a large amount of bugs
launchpadlib
Will add an "expired" state: LP bug janitor disaster
- would affect bugs that are Incomplete w/o response and New old bugs
Roman Friesen input:
- Make reporting of bugs more challenging
- new bug page should be information for the user
- what you can expect after they report a bug
- can take time
- could b required to provide additional information
- Ubuntu is only a collection of software, so we have to wait until bugs are fixed upstream
- One important point: bug fixing team is focused on critical applications( main and security bugs )
- make thre process more challenging
- not just one page, but steps, and critical information must be required, like please execute this command in the terminal to provide ubuntu version, package version, etc...
- you can differentiate between users: collect points for users who are reporting bugs, can provide different ways to report the bugs, more challenging for new users, shorter for those proven to have quality reports