ARMValidationDashboard

Differences between revisions 2 and 7 (spanning 5 versions)
⇤ ← Revision 2 as of 2010-05-03 16:02:30 →
Size: 2683
Editor: 75-27-138-126
Comment:
← Revision 7 as of 2010-05-18 16:07:03 → ⇥
Size: 6626
Editor: fdu90
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
 * '''Launchpad Entry''': UbuntuSpec:arm-m-validation-dashboard  * '''Launchpad Entry''': https://blueprints.launchpad.net/ubuntu-arm/+spec/arm-m-validation-dashboard
Line 5: Line 5:
 * '''Contributors''': PaulLarson  * '''Contributors''': PaulLarson, ZygmuntKrynicki
Line 22: Line 22:
Bob is a release manager for Ubuntu on a particular arm device. Bob wants to check the overall status of the image produced yesterday before releasing Alpha 1. Bob visits the dashboard to check for test failures. Bob marked some tests as expected to fail on this device as not all components are yet in place and some things are still broken. As all other tests have good results Bob can go forward with the release.

Jane is interested in basic performance metrics of current ubuntu image. Jane can check some synthetic benchmarks for CPU, GPU and IO performance. Jane can also check some end-to-end benchmarks for user applications (browser startup time, time to full desktop, time to render snapshot of key websites, etc). Jane can setup baseline for each metric and request to be notified of all variances that exceed given threshold.

Alice is on the Desktop QA team and would like to integrate some of the tests her team has created into the dashboard. QA Engineers quickly bootstrap a local installation of the dashboard and check the bundled documentation and examples. Within hours the dashboard can display results from some of the tests that local engineering team has already adapted.

Yung is a product manager in Big Corp ltd. Yung is building a product based on Ubuntu and wants to reuse our QA infrastructure. Yung instructs his engineers to deploy a local dashboard installation and run our tests on their new secret product. Engineers can also write an adapter that will take some of the results from the dashboard and push it to the internal QA system.
Line 23: Line 31:

 * Easy to deploy, including, outside of Canonical. All infrastructure components must be packaged and provided as a PPA, ready to install on a Lucid server. All device components must be packaged and uploaded to Maverick (TODO: which component? are we going straight-to-ppa or do we attempt to hit the main archive?)
 * One-way connectivity. Devices participating in the test can connect to the infrastructure services but reverse connection cannot be assumed. (TODO: what about IPv6?)
 * Distributed environment. Devices and infrastructure components are places in diverse geographical and administrative zones.
 * Launchpad integration for bug management. There is no plan to support third party bug trackers in the first release.
Line 26: Line 39:
You can have subsections that better describe specific parts of the issue. TODO:
 * UI design for each core use case
 * UI design vs UI design of other Canonical technologies
 * Design web pages we need to provide
Line 30: Line 46:
TODO:
 * Choose basic web technology
 * Choose database system
 * Design data model (including data ingest requirements, data presentation requirements, data transformations and storage)
 * Design web widgets/pieces/components that we will need and determine how each fits into the data model
Line 31: Line 53:

=== Architecture Overview ===

{{drawing:QADashboardArchitecture}}
Line 42: Line 68:
Include:
 * data migration, if any
 * redirects from old URLs to new ones, if any
 * how users will be pointed to the new way of doing things, if necessary.		 Currently there is no direct migration plan. Things we could consider is migrating some bits and pieces of technology already up and running either at Canonical or somewhere else in the community/other parties that is open source and integrate their tests into our framework. If that becomes true we might want to look at migration from qa-tool-foo to our new technology.
Line 55: Line 78:
This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved. * Do we roll our own technology or do we adapt existing frameworks (like Hudson)
Line 59: Line 82:
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected. Goal: define visualization interface for QA control that shows daily/snapshot/other summary of the 'health' of the image for a given platform

Different images based on a common base:
 * server
 * desktop
 * netbookcurrent

Stuff we want to show:
 * difference from yesterday
 * performance history
 * performance regressions
 * performance targets (as infrastructure to see if it works during the cycle)

Dashboard mockup:
 Two columns:
 1)
  - Current build status
    FTBFS count
    New package count, number of packages
    Latest build date/time
    
  - Test result
  - Build history
 2)
  - News
  - Performance Targets

Q: What about some UI for scheculing test runs:
A: we're not targeting this for the first release but we want have a UI for doing that in the future

Q: How does our project relate to other ubuntu QA projects
A:

Stuff to check:
  buildbot (python)
  hudson (java)

Action item: check hudson out? Zygmunt Krynicki
Hudson instance for Bzr at canonical: http://babune.ladeuil.net:24842/view/Ubuntu/job/selftest-jaunty/buildTimeTrend

We want to store the log file of each test run just in case (for unexpected successes)

Summary

As part of the automated testing efforts on ARM, we would also like to have a dashboard interface for visualizing the current state of the image. This interface should allow the user to see, at a glance, the state of functional tests as well as performance tests, and any other data that may be useful.

Release Note

No user visible changes

Rationale

We would like to easily see how things are shaping up over time on arm, and how changes are impacting performance. A dashboard interface would help us visualize, in one place, the results of testing from multiple machines. It could also be used to generate graphs of performance metrics across different image build dates to quickly see if things are improving, or getting worse, and how far we are away from any goals that may exist.

User stories

Bob is a release manager for Ubuntu on a particular arm device. Bob wants to check the overall status of the image produced yesterday before releasing Alpha 1. Bob visits the dashboard to check for test failures. Bob marked some tests as expected to fail on this device as not all components are yet in place and some things are still broken. As all other tests have good results Bob can go forward with the release.

Jane is interested in basic performance metrics of current ubuntu image. Jane can check some synthetic benchmarks for CPU, GPU and IO performance. Jane can also check some end-to-end benchmarks for user applications (browser startup time, time to full desktop, time to render snapshot of key websites, etc). Jane can setup baseline for each metric and request to be notified of all variances that exceed given threshold.

Alice is on the Desktop QA team and would like to integrate some of the tests her team has created into the dashboard. QA Engineers quickly bootstrap a local installation of the dashboard and check the bundled documentation and examples. Within hours the dashboard can display results from some of the tests that local engineering team has already adapted.

Yung is a product manager in Big Corp ltd. Yung is building a product based on Ubuntu and wants to reuse our QA infrastructure. Yung instructs his engineers to deploy a local dashboard installation and run our tests on their new secret product. Engineers can also write an adapter that will take some of the results from the dashboard and push it to the internal QA system.

Assumptions

  • Easy to deploy, including, outside of Canonical. All infrastructure components must be packaged and provided as a PPA, ready to install on a Lucid server. All device components must be packaged and uploaded to Maverick (TODO: which component? are we going straight-to-ppa or do we attempt to hit the main archive?)
  • One-way connectivity. Devices participating in the test can connect to the infrastructure services but reverse connection cannot be assumed. (TODO: what about IPv6?)
  • Distributed environment. Devices and infrastructure components are places in diverse geographical and administrative zones.
  • Launchpad integration for bug management. There is no plan to support third party bug trackers in the first release.

Design

TODO:

  • UI design for each core use case
  • UI design vs UI design of other Canonical technologies
  • Design web pages we need to provide

Implementation

TODO:

  • Choose basic web technology
  • Choose database system
  • Design data model (including data ingest requirements, data presentation requirements, data transformations and storage)
  • Design web widgets/pieces/components that we will need and determine how each fits into the data model

This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

Architecture Overview

UI Changes

Should cover changes required to the UI, or specific UI that is required to implement this

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Migration

Currently there is no direct migration plan. Things we could consider is migrating some bits and pieces of technology already up and running either at Canonical or somewhere else in the community/other parties that is open source and integrate their tests into our framework. If that becomes true we might want to look at migration from qa-tool-foo to our new technology.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

* Do we roll our own technology or do we adapt existing frameworks (like Hudson)

BoF agenda and discussion

Goal: define visualization interface for QA control that shows daily/snapshot/other summary of the 'health' of the image for a given platform

Different images based on a common base:

  • server
  • desktop
  • netbookcurrent

Stuff we want to show:

  • difference from yesterday
  • performance history
  • performance regressions
  • performance targets (as infrastructure to see if it works during the cycle)

Dashboard mockup:

  • Two columns: 1)
    • - Current build status
      • FTBFS count New package count, number of packages Latest build date/time
      - Test result - Build history
    2)
    • - News - Performance Targets

Q: What about some UI for scheculing test runs: A: we're not targeting this for the first release but we want have a UI for doing that in the future

Q: How does our project relate to other ubuntu QA projects A:

Stuff to check:

  • buildbot (python) hudson (java)

Action item: check hudson out? Zygmunt Krynicki Hudson instance for Bzr at canonical: http://babune.ladeuil.net:24842/view/Ubuntu/job/selftest-jaunty/buildTimeTrend

We want to store the log file of each test run just in case (for unexpected successes)

CategorySpec

Specs/M/ARMValidationDashboard (last edited 2010-06-04 12:08:03 by fdu90)