UsefulBugMetrics
|
Size: 2057
Comment:
|
Size: 6537
Comment: tweaks
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| Line 5: | Line 3: |
| Identify bug statistics that will be useful to the Ubuntu team during the cycle to drive quality improvements and assist during release periods. == Release Note == ##This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.) ##It is mandatory. |
Identify additional bug statistics that will be useful to the Ubuntu team during the cycle to drive quality improvements and assist during release periods. |
| Line 15: | Line 7: |
| ##This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified. | We should use bug metrics not only to see where we are but also drive the development and QA effort. Existing bug statistics can be expanded and new ones created to take advantage of new data points being tracked in Launchpad. The bug tracking system of Launchpad has been expanded to record some additional date information regarding bug tasks. The information currently being recored includes the following: * date-created (the date the bug task was created) * date-left-new (the date the bug task's status was first changed) * date-incomplete (the date the bug task's status was last set to Incomplete) * date-confirmed (the date the bug task's status was last set to Confirmed) * date-triaged (the date the bug task's status was last set to Triaged) * date-inprogress (the date the bug task's status was last set to In Progress) * date-fixcommitted (the date the bug task's status was last set to Fix Committed) * date-fixreleased (the date the bug task's status was last set to Fix Released) * date-closed (the date the bug task's status was set to a end status) * date-assigned (the last date the bug was assigned to someone) Currently, this information is only available via the +text interface for a bug report and is not particularly useful by itself. Rather, these dates become more useful when considering the difference between any two dates and over a package as a whole. Subsequently, a new tool should be developed to find the differences and aggregate the information for packages or teams. |
| Line 18: | Line 25: |
* The Mozilla team wants to analyse their bug workflow and throughput from New to Fixed to identify the bottlenecks where they should focus their efforts. The useful bug metrics displayed on a Mozilla summary page lets them target their effort and monitor their progress. * Pedro is planning a bug day to focus on the bugs that have been left in the New and Incomplete states for the longest times. The new bughelper tools let's him produce the appropriate wiki pages. |
|
| Line 23: | Line 34: |
| ##You can have subsections that better describe specific parts of the issue. | Currently, information is being gathered on the quantities of bugs with a specific status or importance. It would be useful to expand this out to include status and importance. Additionally, the Incomplete should be more granular by breaking those numbers down into Incomplete w/ Response and Incomplete w/o Response. Another useful quantity are bugs reported via apport - those tagged as 'apport-bug' and 'apport-crash'. The bughelper suite of applications complements the bug tracking system by providing statistics regarding quantities of bug reports and specific lists of bug reports. The bugnumbers tool has the ability to determine the quantity of bug reports matching specific criteria. However, bugnumbers has an overwhelming number of options and is primarily used to create lists of bug numbers matching specific criteria. Subsequently, the statistics options should become a separate tool and be expanded to include new options. The new bug lifecycle analysis tool should also be included in the bughelper package. |
| Line 27: | Line 40: |
| ##This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like: === UI Changes === ##Should cover changes required to the UI, or specific UI that is required to implement this |
* A new lifecycle tool will be added to the bughelper suite which will also take over some of bugnumber's current features (leaving a simpler bugnumbers) * A range of new throughput statistics will be produced with the bug-lifecycle tool for various slices of time (a given week, month, from the start of the project). Examples: * '''Time in New:''' Sum(date-left-new - date-created) / Count(bugs_in_slice) * '''Time to confirmed''' Sum(date-confirmed - date-created) / Count(bugs_in_slice) * '''Time to triage:''' Sum(date-triaged - date-created) / Count(bugs_in_slice) * '''Time to assigned:''' Sum(date-assigned - date-created) / Count(bugs_in_slice) * '''Time to fixed:''' Sum(date-fixreleased - date-created) / Count(bugs_in_slice) * '''Time from Triaged to fixed:''' Sum(date-fixreleased - date-triaged) / Count(bugs_in_slice) * '''Time worked on:''' Sum(date-fixcommitted - date-inprogress) / Count(bugs_in_slice) * '''Oldest New/Triaged bug''' - display as a link to the current oldest and graphically as a function of time * Key statistics will be displayed for select packages (or rather package groups) See: [:QATeam/Specs/PackageStatus:PackageStatusPages]. |
| Line 35: | Line 54: |
| ##Code changes should include an overview of what needs to change, and in some cases even the specific details. | The statistics features of bugnumbers will become a separate tool from bugnumbers. This new tool's functionality will be expanded to return quantities of bugs in a state and importance. Quantities of Incomplete bug reports will also be broken down into Incomplete with and without a response. |
| Line 37: | Line 56: |
| == Test/Demo Plan == | The bughelper package relies on python-launchpad-bugs for the parsing of bug pages and accessing attributes of a bug report. Therefore python-launchpad-bugs will need to be extended to make these new bug task dates accessible. |
| Line 39: | Line 58: |
| ##It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. | After these new dates are accessible it will be possible to create a new application in the bughelper package to calculate the time for a change to occur for a bug task. This tool should accept any list of bugs, whether it be a package's bugs, a team's bugs, a project's bugs or a whole distribution's bugs, and calculate the time to any change. |
| Line 41: | Line 60: |
| ##This need not be added or completed until the specification is nearing beta. == Outstanding Issues == ##This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved. |
After this functionality has been added the most useful statistics should be made publicly available for packages and teams with a substantial quantity of bug reports. |
| Line 51: | Line 66: |
| MarkusKorn: * Implementation details for bughelper and python-launchpad-bugs * the new version of the +text-parser is able to parse all needed dates. This branch needs some final testing before being merged with the .main branch. [https://code.edge.launchpad.net/~thekorn/python-launchpad-bugs/new_text_parser branch in launchpad] * during UDS I started a [https://code.edge.launchpad.net/~bughelper-dev/bughelper/buggraphs bughelper branch] which adds a 'buggraphs' tool to the bughelper suite. This tool is supposed to be a first draft of the implementation of this spec in bughelper. There are some brief informations on how to use this tool at the top of the sourcecode of 'buggraphs'. |
Summary
Identify additional bug statistics that will be useful to the Ubuntu team during the cycle to drive quality improvements and assist during release periods.
Rationale
We should use bug metrics not only to see where we are but also drive the development and QA effort. Existing bug statistics can be expanded and new ones created to take advantage of new data points being tracked in Launchpad.
The bug tracking system of Launchpad has been expanded to record some additional date information regarding bug tasks. The information currently being recored includes the following:
- date-created (the date the bug task was created)
- date-left-new (the date the bug task's status was first changed)
- date-incomplete (the date the bug task's status was last set to Incomplete)
- date-confirmed (the date the bug task's status was last set to Confirmed)
- date-triaged (the date the bug task's status was last set to Triaged)
- date-inprogress (the date the bug task's status was last set to In Progress)
- date-fixcommitted (the date the bug task's status was last set to Fix Committed)
- date-fixreleased (the date the bug task's status was last set to Fix Released)
- date-closed (the date the bug task's status was set to a end status)
- date-assigned (the last date the bug was assigned to someone)
Currently, this information is only available via the +text interface for a bug report and is not particularly useful by itself. Rather, these dates become more useful when considering the difference between any two dates and over a package as a whole. Subsequently, a new tool should be developed to find the differences and aggregate the information for packages or teams.
Use Cases
- The Mozilla team wants to analyse their bug workflow and throughput from New to Fixed to identify the bottlenecks where they should focus their efforts. The useful bug metrics displayed on a Mozilla summary page lets them target their effort and monitor their progress.
- Pedro is planning a bug day to focus on the bugs that have been left in the New and Incomplete states for the longest times. The new bughelper tools let's him produce the appropriate wiki pages.
Assumptions
Design
Currently, information is being gathered on the quantities of bugs with a specific status or importance. It would be useful to expand this out to include status and importance. Additionally, the Incomplete should be more granular by breaking those numbers down into Incomplete w/ Response and Incomplete w/o Response. Another useful quantity are bugs reported via apport - those tagged as 'apport-bug' and 'apport-crash'.
The bughelper suite of applications complements the bug tracking system by providing statistics regarding quantities of bug reports and specific lists of bug reports. The bugnumbers tool has the ability to determine the quantity of bug reports matching specific criteria. However, bugnumbers has an overwhelming number of options and is primarily used to create lists of bug numbers matching specific criteria. Subsequently, the statistics options should become a separate tool and be expanded to include new options. The new bug lifecycle analysis tool should also be included in the bughelper package.
Implementation
- A new lifecycle tool will be added to the bughelper suite which will also take over some of bugnumber's current features (leaving a simpler bugnumbers)
- A range of new throughput statistics will be produced with the bug-lifecycle tool for various slices of time (a given week, month, from the start of the project). Examples:
Time in New: Sum(date-left-new - date-created) / Count(bugs_in_slice)
Time to confirmed Sum(date-confirmed - date-created) / Count(bugs_in_slice)
Time to triage: Sum(date-triaged - date-created) / Count(bugs_in_slice)
Time to assigned: Sum(date-assigned - date-created) / Count(bugs_in_slice)
Time to fixed: Sum(date-fixreleased - date-created) / Count(bugs_in_slice)
Time from Triaged to fixed: Sum(date-fixreleased - date-triaged) / Count(bugs_in_slice)
Time worked on: Sum(date-fixcommitted - date-inprogress) / Count(bugs_in_slice)
Oldest New/Triaged bug - display as a link to the current oldest and graphically as a function of time
Key statistics will be displayed for select packages (or rather package groups) See: [:QATeam/Specs/PackageStatus:PackageStatusPages].
Code Changes
The statistics features of bugnumbers will become a separate tool from bugnumbers. This new tool's functionality will be expanded to return quantities of bugs in a state and importance. Quantities of Incomplete bug reports will also be broken down into Incomplete with and without a response.
The bughelper package relies on python-launchpad-bugs for the parsing of bug pages and accessing attributes of a bug report. Therefore python-launchpad-bugs will need to be extended to make these new bug task dates accessible.
After these new dates are accessible it will be possible to create a new application in the bughelper package to calculate the time for a change to occur for a bug task. This tool should accept any list of bugs, whether it be a package's bugs, a team's bugs, a project's bugs or a whole distribution's bugs, and calculate the time to any change.
After this functionality has been added the most useful statistics should be made publicly available for packages and teams with a substantial quantity of bug reports.
BoF agenda and discussion
- Implementation details for bughelper and python-launchpad-bugs
the new version of the +text-parser is able to parse all needed dates. This branch needs some final testing before being merged with the .main branch. [https://code.edge.launchpad.net/~thekorn/python-launchpad-bugs/new_text_parser branch in launchpad]
during UDS I started a [https://code.edge.launchpad.net/~bughelper-dev/bughelper/buggraphs bughelper branch] which adds a 'buggraphs' tool to the bughelper suite. This tool is supposed to be a first draft of the implementation of this spec in bughelper. There are some brief informations on how to use this tool at the top of the sourcecode of 'buggraphs'.
QATeam/Specs/UsefulBugMetrics (last edited 2008-08-06 16:36:14 by localhost)