Ubuntu Wiki

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

• Launchpad Entry: developer-weather-report

Summary

The goal is to provide a dashboard covering all the variables that go into determining whether Ubuntu is release ready - bug count, RC bug count, archive analysis reports, iso testing summaries, etc.

Rationale

There currently exists numerous sites which display information regarding the state of Ubuntu. However, it is inconvenient having to remember multiple sites as well as navigate to multiple locations to obtain wanted information. It would be useful if this information were gathered into one central location. A dashboard will help showcase this infomration at a single location and provide the release managers and Distro Team a way to measure the quality of an upcoming release.

Use Cases

• Release manager Bob wants to know the current Distro release status and what showstopping bugs exist at this point in the release. He'd also like to drill down and see which of these bugs relate to the packages his team oversees. He currently has to visit multiple different locations to gather this information.
• Mary in the distro team wants to know about the iso testing status and the current number of iso-testing related bugs. This will help her determine which areas still need help testing as well as what areas are proving to be problematic. She also currently has to visit multiple locations to find this information.

Assumptions

• The system should scale gracefully, i.e. it should be trivial to add a new data source and incorporate it into the dashboard display
• Visual cues (e.g. color) should be used to show at a glance whether anything is outstanding
• Dashboard should provide a link to the specific metric thus enabling the capability to drill down to the information
• A distinction could be made between critical indicators (blocking) and other metrics (e.g. bug counts)

Design

Ideas (Release readiness / critical path)

The items in this table will be the initial target.

Need more info on the following two items? (specific source of the data and what info to parse)

• isos up to date (archive, ISOs)
• livefs up-to-dateness (cdimage)

Ideas (other metrics)

The items listed here will be taken into consideration after the initial implementation (above table) is satisfactory with the target audience.

• quantity of duplicate bugs reported (Launchpad / python-launchpad-bugs)
• determine bugs of interest based on subscriber or duplicate count (Launchpad / python-launchpad-bugs)
• open bugs with patches attached (Launchpad / python-launchpad-bugs)
• percentage change of iso testing passes/fails

• merge status (http://merges.ubuntu.com/stats.txt)

  • Number of merges needed for different areas (e.g. Desktop, Xorg)
• sync status (during freeze periods, including link to diff) to see which Debian RC bugs could be fixed by a sync

• http://popcon.ubuntu.com/ - more for bug statistics

• upstream version currency (some code in http://dehs.alioth.debian.org/ and svn://svn.debian.org/qa/trunk/)

• outstanding archive requests: syncs, removals, ... (Launchpad / python-launchpad-bugs)

  • https://launchpad.net/~ubuntu-archive/+subscribedbugs?field.searchtext=sync

  • https://launchpad.net/~ubuntu-archive/+subscribedbugs?field.searchtext=remov

• Language pack status (update package versions should be identical to -base ones; IOW: update packages are empty)

• GnomeAppInstallDesktopDatabaseUpdate: is http://people.ubuntu.com/~mvo/gnome-app-install/ up to date?

• ReadaheadListUpdate: has the /etc/readahead/boot file in the readahead-list package been updated in the last week?

  • a count of lintian errors of carefully selected error classes (not everything is worth worrying about)
    • liw will be running lintian on Ubuntu, at lesat main, and will make such a selection list anyway

• automatic ubiquity test (ubiquity-vmware-automation) results (qa tracker?)

• https://launchpad.net/ubuntu/hardy/+queue?queue_state=0 - new packages to consider in release queue

• https://launchpad.net/ubuntu/hardy/+queue?queue_state=1 - release queue packages which need to go through the 7 step SRU process

• Metrics from http://qa.debian.org/developer.php?login=debian-gcc%40lists.debian.org&comaint=yes (and DEHS)

  • upstream version
  • debian version
  • bug counts
• buildd status (including age of failed build)

  • https://launchpad.net/ubuntu/hardy/+builds et al (can select by state) - some already covered in table above

• Once we have test automation infrastructure running, we'll probably want to display this info as well.

Implementation

Example Prototype (mdz)

• all data should be tracked historically, but not in the first version
  • maybe keep all output files (data files?) in bzr
  • low-priority for implementation
• package specific data (ex. build failures) will probably want a separate report for main / restricted / universe / multiverse

  • https://launchpad.net/ubuntu/hardy/+builds?build_text=&build_state=failed

  • http://people.ubuntu.com/~ubuntu-archive/testing/hardy_outdate.txt has FTBFS+NBS+NEEDSBUILD for main

  • http://people.ubuntu.com/~ubuntu-archive/NBS/ has NBS for all components

• there should be an indication as to when the stat was last updated as not all reports are run at the same time
  • watermark/timestamp the reports
• XML-RPC interface? or scrape HTML
  • Scraping HTML will be very brittle
  • One suggestion was to have each source output javascript object notation (JSON)

    • Each data source can then be "syndicated" at given locations (e.g. http://.../.../datasource1.js, and update at whatever frequency makes most sense for that particular data source (hourly, daily, etc.).

    • Can still use scraping for areas that do not output this format, but have those tools each output JSON for inclusion into the interface
    • The weather report page can then re-load these objects asynchronously, so page will update itself without needing manual reload
• severity will be package dependent
• ideally viewers should be able to filter the stats they want to view and save(?) preferences - most likely won't happen in the initial version
• how often do we want the reports generated?
  • hourly for many components; sections regarding CDs should be more frequent
• how far back do we want these reports archived?
  • I imagine we want to be able to look back indefinitely
• will the site poll for data or is data pushed?
  • initially poll for the data as not every data source will have the capability to push data
• will display results at the new qa website, qa.ubuntu.com

UI Changes

• There is a risk the interface could get very "cluttered", esp. if we wish to show historical info (graphs, etc.)

  • One suggestion was to use "SparkLines" http://en.wikipedia.org/wiki/Sparkline . This displays historical trending in a concise amount of space

• It may also help to have collapsed sections that can be expanded to get the next level of detail. For instance, one line may show total number of new, open, etc. bugs, but when expanded it shows a further breakdown (per milestone, per area, over time, or etc.) - again, most likely wont' happen in the initial version

Test/Demo Plan

Outstanding Issues

BoF agenda and discussion

Discussions were held at UDS Boston 2007 and the comments have been encorporated into the above spec.

CategorySpec