DeveloperWeatherReport
10135
Comment: Add notes from gobby session
|
10148
|
Deletions are marked like this. | Additions are marked like this. |
Line 56: | Line 56: |
* determine bugs of interest based on subscriber count (Launchpad / python-launchpad-bugs) | * determine bugs of interest based on subscriber or duplicate count (Launchpad / python-launchpad-bugs) |
Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.
Launchpad Entry: developer-weather-report
Packages affected:
Summary
Provide a dashboard covering all the variables that go into determining whether Ubuntu is release ready - bug count, RC bug count, archive analysis reports, iso testing summaries, etc.
Release Note
Rationale
This will provide the release managers and Distro Team an actual metric to help measure the quality of an upcoming release. A lot of this data already exists today but is spread amongst different locations. We want to gather the information into one central location.
Use Cases
- Release manager Bob wants to know what showstopping bugs exist and how many there are.
- Mary in the distro team wants to know about the iso testing status and what area could use more focus and help.
Assumptions
- The system should scale gracefully, i.e. it should be trivial to add a new data source and incorporate it into the dashboard display
- Visual cues (e.g. color) should be used to show at a glance whether anything is outstanding
- Should provide a link to the specific metric thus enabling the capability to drill down to the information
- A distinction could be made between critical indicators (blocking) and other metrics (e.g. bug counts)
Design
Ideas (Release readiness / critical path)
- display a list of high/critical milestoned (rc) bugs (Launchpad / python-launchpad-bugs)
- iso testing (qa tracker)
- isos up to date (archive, ISOs)
automatic ubiquity test (ubiquity-vmware-automation) results (qa tracker?)
- iso bug count (Launchpad / python-launchpad-bugs)
automatic upgrade testing result (http://people.ubuntu.com/~mvo/automatic-upgrade-testing/)
- packages which have no current build (archive or Launchpad?)
- failed builds (Launchpad?, include links to failed-build logs)
- oversize cd images - which images are of interest? (cdimage - an *.OVERSIZED file shows up)
pending uploads in the queue that need to be approved (Launchpad or http://people.ubuntu.com/~ubuntu-archive/queue/)
installability of *-desktop packages and other archive inconsistencies (http://people.ubuntu.com/~ubuntu-archive/testing/)
- Launchpad distrorelease status (Launchpad)
- livefs up-to-dateness (cdimage)
mirror publishing (https://launchpad.net/ubuntu/+archivemirrors, https://launchpad.net/ubuntu/+cdmirrors)
meta-release index (http://changelogs.ubuntu.com/meta-release)
- bittorrent (torrent.ubuntu.com:6969 ?) make sure it is seeding
Ideas (other metrics)
- quantity of duplicate bugs reported (Launchpad / python-launchpad-bugs)
- determine bugs of interest based on subscriber or duplicate count (Launchpad / python-launchpad-bugs)
- open bugs with patches attached (Launchpad / python-launchpad-bugs)
- percentage change of iso testing passes/fails
merge status (http://merges.ubuntu.com/stats.txt)
- Number of merges needed for different areas (e.g. Desktop, Xorg)
- sync status (during freeze periods, including link to diff) to see which Debian RC bugs could be fixed by a sync
http://popcon.ubuntu.com/ - more for bug statistics
upstream version currency (some code in http://dehs.alioth.debian.org/ and svn://svn.debian.org/qa/trunk/)
- outstanding archive requests: syncs, removals, ... (Launchpad / python-launchpad-bugs)
- Language pack status (update package versions should be identical to -base ones; IOW: update packages are empty)
GnomeAppInstallDesktopDatabaseUpdate: is http://people.ubuntu.com/~mvo/gnome-app-install/ up to date?
ReadaheadListUpdate: has the /etc/readahead/boot file in the readahead-list package been updated in the last week?
- a count of lintian errors of carefully selected error classes (not everything is worth worrying about)
- liw will be running lintian on Ubuntu, at lesat main, and will make such a selection list anyway
- a count of lintian errors of carefully selected error classes (not everything is worth worrying about)
Implementation
- all data should perhaps be tracked historically, but not in the first version
- but keep all output files (data files?) in bzr
- low-priority for implementation
- package build failures will probably want a separate report for main / restricted / universe / multiverse
https://launchpad.net/ubuntu/hardy/+builds?build_text=&build_state=failed
http://people.ubuntu.com/~ubuntu-archive/testing/hardy_outdate.txt has FTBFS+NBS+NEEDSBUILD for main
http://people.ubuntu.com/~ubuntu-archive/NBS/ has NBS for all components
https://launchpad.net/ubuntu/hardy/+queue?queue_state=0 - new packages to consider in release queue
https://launchpad.net/ubuntu/hardy/+queue?queue_state=1 - release queue packages which need to go through the 7 step SRU process
- have some sort of distinction in the reports (ex main, restricted, universe, multiverse)
- there should be an indication as to when the stat was last updated as not all reports are run at the same time
- XML-RPC interface? or scrape HTML
- Scraping HTML will be very brittle
- Have each source output javascript object notation (JSON)
Each data source can then be "syndicated" at given locations (e.g. http://.../.../datasource1.js, and update at whatever frequency makes most sense for that particular data source (hourly, daily, etc.).
- Can still use scraping for areas that do not output this format, but have those tools each output JSON for inclusion into the interface
- The weather report page can then re-load these objects asynchronously, so page will update itself without needing manual reload
- how is severity determined?
- package dependent
- restrict output to problems, hide "green status"
Metrics from http://qa.debian.org/developer.php?login=debian-gcc%40lists.debian.org&comaint=yes (and DEHS)
- upstream version
- debian version
- bug counts
- buildd status (including age of failed build)
https://launchpad.net/ubuntu/hardy/+builds et al (can select by state)
Example Prototype (mdz)
Metric |
Status |
Launchpad release status |
FROZEN |
Pending uploads |
Empty |
Failed package builds |
None |
Pending package builds |
5 |
Pending livefs builds |
None |
Archive inconsistencies |
3 |
Outdated installation media |
desktop, alternate |
Showstopper bugs |
#1234, #2345 |
Automated installation tests |
Successful (current build) |
Manual installation tests |
Successful (build 20071031.2) |
Pre-published mirrors |
0 |
Meta-release index |
Pending |
Bittorrent |
Pending |
Logistics
- How often do we want the reports generated?
- hourly for many components; sections regarding CDs should be more frequent
- how far back do we want these reports archived?
- will the site poll for data or is data pushed?
- Will we display results at the new qa website qa.stgraber.com or launchpad?
- more likely on qa.ubuntu.com
- Also, once we have test automation infrastructure running, we'll probably want to display this info as well.
- Ideally viewers should be able to filter the stats they want to view and save(?) preferences
UI Changes
There is a risk the interface could get very "cluttered", esp. if we wish to show historical info (graphs, etc.) One graphical technique for displaying info with minimal clutter is "SparkLines" http://en.wikipedia.org/wiki/Sparkline This displays historical trending in a concise amount of space
It may also help to have collapsed sections that can be expanded to get the next level of detail. For instance, one line may show total number of new, open, etc. bugs, but when expanded it shows a further breakdown (per milestone, per area, over time, or etc.)
Code Changes
Migration
Test/Demo Plan
Outstanding Issues
BoF agenda and discussion
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.
QATeam/Specs/DeveloperWeatherReport (last edited 2008-08-06 16:18:53 by localhost)