Diff for "ReleaseFutureMetrics"

ReleaseFutureMetrics

Differences between revisions 3 and 4

This is a brainstorm area to describe the metrics we'd like to put in place, and links to partial solutions that exist today (and sources when known).

Future Metrics

Approach

initially focus metrics on
- incoming & triage rate. - changes over time
- bug status for specific release - changes over time
- bug status for specific packages with aggregation across main in logical groups.
- extraction and reporting of release targetted bugs, broken into team focus areas.
time phase to use:
- Per week (minimum) for active development & incoming, possibly daily updates.
- Monthly for the stable, supported releases.
heuristics will need some missing knowledge to drive them, that will need to be kept "live".
- come up with heirarchy grouping related packages to report against.
- drill down to package level, but have a meaninful overview.
- groupings will probably flow out of some of the heuristic & -rdpends analysis.

Reports

Incoming & Triage Bug Rates

Need to graph summary of all new/open, over date range in launchpad (1 week or daily? )
1. Initially just use user reported severity (critical, hot, etc.) but then refine to use triage heuristics(TBD) to give better indicator of true priority to overall projects.
  1. Over date range in launchpad (1 week interval possibly daily) - list of all triaged, possibly do as rolling 6 month so trend lines become visible? 3 month interval? to discuss.
  2. In parallel for each week report the summary of where the analyzed ones have been triaged
2. by annotating release marked against (ie. 10.10, 10.4, 9.08, 8.04)
  1. possibly look at using color gradation to indicate severity within stacked bar chart per release. Probably to too busy with this detail? to discuss.
  2. over date range in launchpad (1 week interval) - total number untriaged.
3. see Incoming rate draft for illustration. This graph should be on the top of a WIKI page, where each of the color bars for this week's data is effectively a click to a list of bugs that were counted against each category (see legend in mockup for the idea of categories).
  1. Some implementation concerns:
    - need way in launchpad so that submitter marks where discovered (? is this here - double check, saw target fix against, but not source... - and based on source, could pull up area bootloader, kernel, toolchain, packages, based on release manifest - could let narrow down packages better if reporter knows ??)
Project behavior looking to see change by tracking this:
- Heuristics effective at steering focus to triaging the current release & high impact bugs to appropriate development teams quicker.
- New bugs == triaged bugs over time. then pull down backlog.
- Community wide looking to see improved awareness:
  - scope of problem of untriage'd bugs - community "rally to solve".
  - heuristics possibly find some key bugs that have been overlooked in triage process.

Release Specific Bugs

Requirements:
- For each specific release should be able to pull up chart of open/closed bugs per week. ie. 10.10beta vs. 10.10.a3
  - Need to distinguish pre and post release (track bugs through 6 month release development lifecycle)? track through lifetime of release, after issued. How handle LTS? to discuss.
  - Migrate targetted bugs from one release (not resolved) to next
  - Master summary for entire release 10.10 - summary of 10.10beta + 10.10a3 +... pre-release for 10.10. Need to be able to parameterize granularity of bugs you're looking at - like burndown charts permits)
  - Track over time, web page with color coding (heat) exists for open already - need to add summary component.
Need to graph:
- Over date range in launchpad (1 week interval, daily?)) - list of all release targetted open bugs
  - Bugs should of status: New, Triaged, Confirmed, In progress, Incomplete?
  - Use triaged severity (critical, hot, medium, low, undecided, wishlist) to summarize as color coded stacked bar.
- Over date range in launchpad (1 week interval, daily?) - list of all resolved bugs
  - Bugs should be of status: Fix Committed, Fix Released, Won't Fix
  - Use severity (critical, hot, medium, low, undecided, wishlist) to summarize as color coded stacked bar.
  - ? might want to consider "untriaged New flagged against release" as separate reportable?
Chart thoughts:
- show release milestone on X-axis timeline (think burndown chart) corresponding to granularity selected.
- want to be able to look at bugs at various granularities using tags (like burndown feature charts)
- superset of release (all teams, all alpha, beta,final releases) ie. 10.10 status
- specific release (all teams) ie. alpha3 overall status (timeline to correspond to alpha3)
- specific team for specific release ie. alpha3 status for kernel team (timeline to correspond to alpha3)
- specific team for entire release ie. kernel team across all of 10.10
- set of stacked open/close get added to status on weekly intervals.
- chart is at top of WIKI page (again, following burndown model), and all the bugs listed against open/closed for that week are reported in the WIKI page explicitly.
Project behaviour looking to see change:
- All critical closed by release.
- Last minute "surprises" not emerging from the untriaged queue - just before release.
- Not loose focus on bugs high/medium bugs.
Community wide looking to see improved awareness:
- wider developer and management awareness of actual bug status for release.
- more of the critical/hot bugs fixed earlier in release cycle.

Package-oriented Bug Overview dashboard

come up with mapping for each package into "logical grouping"
- see Release Notes for a good first approximation (except for desktops - they're special )
  - use Release Flavors to identify related desktop packages ie. GNOME family, KDE family, etc.
  - use -rdepends analysis to identify "common dependency groups"
  - goal is to find about 30-40 high level groups that the packages can logically "belong to".
  - code up the logical grouping into WIKI and machine readable form, so can be changed over time.
- need to discuss how often this should be updated & logged - weekly? monthly? -- to discuss.
- Set it up as summary WIKI table for open bugs (New, Triaged, Confirmed, In progress, Incomplete), where clicking on row, will drill down to actual package list for that grouping of on separate page in similar format.
Example:

Package Grouping	critical	high	med	low'	undecided	wish list
Installation	2	4	10	20	80	10
Upgrade	1	0	8	40	100	23
Kernel	..	..	..	..	..	..
Toolchain	..	..	..	..	..	..
GNOME desktop (needs further breakdown)	..	..	..	..	..	..
KDE desktop (needs further breakdown)	..	..	..	..	..	..
Server and Cloud	..	..	..	..	..	..
Miscellaneous packages	..	..	..	..	..	..

Page for "Installation" would have:

Package Grouping	critical	high	med	low'	undecided	wish list
grub2	1	2	1	1	15	1
installation-guide	1	0	3	1	5	0
language-selector	0	0	6	18	9	5
ubiquity	0	2	0	10	11	0
...	0	0	0	0	0	1
...	0	0	0	0	0	2
...	0	0	0	0	0	0
Misc.	0	1	0	0	40	1

Possibly we might want to have it so that if you click on package name - you can drill down to all the "active" version of the package, and breakdown of bugs per specific version in play? Not sure.
If you click on any of the numbers - you should be able to bring up a WIKI page with list of all the bugs that made up number. For example if you click on grub2 - high bug column you'd get:

package	bug number	description
grub2	576724	Only offer partitions containing /, /boot, or /boot/grub for grub-install; installing to other partitions may have harmful effects such as making Windows unbootable, and installing GRUB to every single partition is likely to result in confusion anyway
grub2	508173	When migrating from the old grub-pc/install_devices scheme, check if the old value indicated installation to a single partition on a single disk. In that case, if there is only one disk present and it has a matching partition number, then we can install to that partition without asking

Note: none of the details above are accurate... only for illustrative purposes.

Project behaviors looking to see change with this:
- Management has better understanding of areas that need improvement/resources at package level, across all releases
- Better opportunistic management of backlog and prioritization of community/development developers on fixing important areas
Community wide looking to see improved awareness:
- Wider development awareness of "quality of the active code base", rich fix areas.
- Identifiable points of pain for "focused" community activities to address
Note: when release is "end of life"d, and has active bugs, a scrub should be done to see if the package carries forward in knowledge base, and if bugs should be reassigned/tagged to other release.

Background

Some considerations to keep in mind for the above proposals:

reported bugs are good. Feedback is a gift. Avoid discouraging people from trying product and reporting bug. Keep any changes to Launchpad lightweight. (one click)
large number of incoming bugs, not sure of rates, critical ones being overlooked for releases, low priority ones ignored, etc. From July 2010 report: "There were 5,558 new bugs reported in July. 4,409 bugs were closed in the same period, resulting in a total of 78,228 open bugs. That is an increase of 1,149. The total number of bugs in the New state decreased by 77 to a total of 37,246." so based on this, we're probably seeing about 1000 bugs per week, and triaging is not keeping up.
need to automate as much as possible, to deal with volume & prioritization of triage focus.

Knowledge Base to Support Automation

Need heuristics to apply to incoming bug based on where discovered and target to funnel to queue of prioritized bugs to be triaged. This will be heuristic and our first attempts will probably get it wrong, ( If there are heuristics in place already, please point me at them. Haven't been able to find so far. )
- Factors to be considered in the triage automation heuristics (and knowledge bases to build up):
  - Target (and/or source) of fix is an upcoming development release.
  - Which package is it in, and who depends on it. (? this info should be installer for package dependencies and add some common sense for foundations, kernel, installer parts) weighting should be higher dependency "score", higher in triage list
  - How many flavors depend on the package (ubuntu main, kubuntu, ...)
  - Is there a designated package maintainer that the notification can be routed to directly to triage
  - During planning sprint/UDS for release tag packages that are key to release requirements -- and should get extra "love"
  - Severity given problem by the reporter.

Existing Pieces: Examples and Sources

QA's existing reports

Development Starting Points

Bug Hugger

Contacts:
- Rick Spencer
Access Info:
- TBF
Concern: speed of interface through launchpad
Sources:
- TBF

ReleaseFutureMetrics (last edited 2010-12-16 21:53:12 by c-98-246-63-231)

-  ⇤ ← Revision 3 as of 2010-09-22 17:10:12 → 
  Size: 15128
  Editor: ppp-71-139-1-225
  Comment:
+   ← Revision 4 as of 2010-09-22 17:14:55 → ⇥
  Size: 15133
  Editor: ppp-71-139-1-225
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-This is a brainstorm to describe the metrics we'd like to put in place, and the links to partial solution that exist today and their sources that may contribute.
+This is a brainstorm area to describe the metrics we'd like to put in place, and links to partial solutions that exist today (and sources when known).
 Line 143:
-Some considerations for following proposals:
+Some considerations to keep in mind for the above proposals:

Ubuntu Wiki