ARMValidationDashboard
Size: 2722
Comment:
|
Size: 5427
Comment:
|
Deletions are marked like this. | Additions are marked like this. |
Line 5: | Line 5: |
* '''Contributors''': PaulLarson | * '''Contributors''': PaulLarson, ZygmuntKrynicki |
Line 22: | Line 22: |
Bob is a release manager for Ubuntu on a particular arm device. Bob wants to check the overall status of the image produced yesterday before releasing Alpha 1. Bob visits the dashboard to check for test failures. Bob marked some tests as expected to fail on this device as not all components are yet in place and some things are still broken. As all other tests have good results Bob can go forward with the release. Jane is interested in basic performance metrics of current ubuntu image. Jane can check some synthetic benchmarks for CPU, GPU and IO performance. Jane can also check some end-to-end benchmarks for user applications (browser startup time, time to full desktop, time to render snapshot of key websites, etc). Jane can setup baseline for each metric and request to be notified of all variances that exceed given threshold. Alice is on the Desktop QA team and would like to integrate some of the tests her team has created into the dashboard. QA Engineers quickly bootstrap a local installation of the dashboard and check the bundled documentation and examples. Within hours the dashboard can display results from some of the tests that local engineering team has already adapted. Yung is a product manager in Big Corp ltd. Yung is building a product based on Ubuntu and wants to reuse our QA infrastructure. Yung instructs his engineers to deploy a local dashboard installation and run our tests on their new secret product. Engineers can also write an adapter that will take some of the results from the dashboard and push it to the internal QA system. |
|
Line 23: | Line 31: |
* Easy to deploy, including, outside of Canonical. All infrastructure components must be packaged and provided as a PPA, ready to install on a Lucid server. All device components must be packaged and uploaded to Maverick (TODO: which component? are we going straight-to-ppa or do we attempt to hit the main archive?) * One-way connectivity. Devices participating in the test can connect to the infrastructure services but reverse connection cannot be assumed. (TODO: what about IPv6?) * Distributed environment. Devices and infrastructure components are places in diverse geographical and administrative zones. * Launchpad integration for bug management. There is no plan to support third party bug trackers in the first release. |
|
Line 26: | Line 39: |
You can have subsections that better describe specific parts of the issue. | TODO: * UI design for each core use case * UI design vs UI design of other Canonical technologies * Design web pages we need to provide |
Line 29: | Line 45: |
TODO: * Choose basic web technology * Choose database system * Design data model (including data ingest requirements, data presentation requirements, data transformations and storage) * Design web widgets/pieces/components that we will need and determine how each fits into the data model |
Launchpad Entry: https://blueprints.launchpad.net/ubuntu-arm/+spec/arm-m-validation-dashboard
Created: PaulLarson
Contributors: PaulLarson, ZygmuntKrynicki
Packages affected:
Summary
As part of the automated testing efforts on ARM, we would also like to have a dashboard interface for visualizing the current state of the image. This interface should allow the user to see, at a glance, the state of functional tests as well as performance tests, and any other data that may be useful.
Release Note
No user visible changes
Rationale
We would like to easily see how things are shaping up over time on arm, and how changes are impacting performance. A dashboard interface would help us visualize, in one place, the results of testing from multiple machines. It could also be used to generate graphs of performance metrics across different image build dates to quickly see if things are improving, or getting worse, and how far we are away from any goals that may exist.
User stories
Bob is a release manager for Ubuntu on a particular arm device. Bob wants to check the overall status of the image produced yesterday before releasing Alpha 1. Bob visits the dashboard to check for test failures. Bob marked some tests as expected to fail on this device as not all components are yet in place and some things are still broken. As all other tests have good results Bob can go forward with the release.
Jane is interested in basic performance metrics of current ubuntu image. Jane can check some synthetic benchmarks for CPU, GPU and IO performance. Jane can also check some end-to-end benchmarks for user applications (browser startup time, time to full desktop, time to render snapshot of key websites, etc). Jane can setup baseline for each metric and request to be notified of all variances that exceed given threshold.
Alice is on the Desktop QA team and would like to integrate some of the tests her team has created into the dashboard. QA Engineers quickly bootstrap a local installation of the dashboard and check the bundled documentation and examples. Within hours the dashboard can display results from some of the tests that local engineering team has already adapted.
Yung is a product manager in Big Corp ltd. Yung is building a product based on Ubuntu and wants to reuse our QA infrastructure. Yung instructs his engineers to deploy a local dashboard installation and run our tests on their new secret product. Engineers can also write an adapter that will take some of the results from the dashboard and push it to the internal QA system.
Assumptions
- Easy to deploy, including, outside of Canonical. All infrastructure components must be packaged and provided as a PPA, ready to install on a Lucid server. All device components must be packaged and uploaded to Maverick (TODO: which component? are we going straight-to-ppa or do we attempt to hit the main archive?)
- One-way connectivity. Devices participating in the test can connect to the infrastructure services but reverse connection cannot be assumed. (TODO: what about IPv6?)
- Distributed environment. Devices and infrastructure components are places in diverse geographical and administrative zones.
- Launchpad integration for bug management. There is no plan to support third party bug trackers in the first release.
Design
TODO:
- UI design for each core use case
- UI design vs UI design of other Canonical technologies
- Design web pages we need to provide
Implementation
TODO:
- Choose basic web technology
- Choose database system
- Design data model (including data ingest requirements, data presentation requirements, data transformations and storage)
- Design web widgets/pieces/components that we will need and determine how each fits into the data model
This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:
UI Changes
Should cover changes required to the UI, or specific UI that is required to implement this
Code Changes
Code changes should include an overview of what needs to change, and in some cases even the specific details.
Migration
Include:
- data migration, if any
- redirects from old URLs to new ones, if any
- how users will be pointed to the new way of doing things, if necessary.
Test/Demo Plan
It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.
This need not be added or completed until the specification is nearing beta.
Unresolved issues
This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.
BoF agenda and discussion
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.
Specs/M/ARMValidationDashboard (last edited 2010-06-04 12:08:03 by fdu90)