ARMValidationDashboard
Size: 2683
Comment:
|
Size: 12049
Comment:
|
Deletions are marked like this. | Additions are marked like this. |
Line 3: | Line 3: |
* '''Launchpad Entry''': UbuntuSpec:arm-m-validation-dashboard | * '''Launchpad Entry''': https://blueprints.launchpad.net/ubuntu/+spec/arm-m-validation-dashboard |
Line 5: | Line 5: |
* '''Contributors''': PaulLarson | * '''Contributors''': PaulLarson, ZygmuntKrynicki |
Line 22: | Line 22: |
Bob is a release manager for Ubuntu on a particular arm device. Bob wants to check the overall status of the image produced yesterday before releasing Alpha 1. Bob visits the dashboard to check for test failures. Bob marked some tests as expected to fail on this device as not all components are yet in place and some things are still broken. As all other tests have good results Bob can go forward with the release. Jane is interested in basic performance metrics of current ubuntu image. Jane can check some synthetic benchmarks for CPU, GPU and IO performance. Jane can also check some end-to-end benchmarks for user applications (browser startup time, time to full desktop, time to render snapshot of key websites, etc). Jane can setup baseline for each metric and request to be notified of all variances that exceed given threshold. Alice is on the Desktop QA team and would like to integrate some of the tests her team has created into the dashboard. QA Engineers quickly bootstrap a local installation of the dashboard and check the bundled documentation and examples. Within hours the dashboard can display results from some of the tests that local engineering team has already adapted. Yung is a product manager in Big Corp ltd. Yung is building a product based on Ubuntu and wants to reuse our QA infrastructure. Yung instructs his engineers to deploy a local dashboard installation and run our tests on their new secret product. Engineers can also write an adapter that will take some of the results from the dashboard and push it to the internal QA system. |
|
Line 24: | Line 32: |
* Easy to deploy, including, outside of Canonical. All infrastructure components must be packaged and provided as a PPA, ready to install on a Lucid server. All device components must be packaged and uploaded to Maverick (TODO: which component? are we going straight-to-ppa or do we attempt to hit the main archive?) * Focused on one device. FIXME - is this _really_ true? - if not how to avoid multiple devices and benchmarks There seems to be a conflict of interests - we'd like to see this for >1 kind of device but vendors will not enjoy any device benchmark information being displayed, especially alongside competing devices * One-way connectivity. Devices participating in the test can connect to the infrastructure services but reverse connection cannot be assumed. (TODO: what about IPv6?) * Distributed environment. Devices and infrastructure components are places in diverse geographical and administrative zones. * Launchpad integration for bug management. There is no plan to support third party bug trackers in the first release. |
|
Line 26: | Line 43: |
You can have subsections that better describe specific parts of the issue. | TODO: * UI design for each core use case (TODO) (we want a list of tasks users have to perform to get to the goal they are after (with regard to the use case lists above) * UI design vs UI design of other Canonical technologies (TODO) * Design web pages we need to provide (DONE) Dashboard will feature the following pages/views: === Project Timeline === Recurring component of each page, shown at the top. Key aspects: * Shows milestones * Shows number of days till next milestone * Shows number of days/weeks? till final milestone * Allows to click at a past day to see historical records Project timeline could also hold global image/project properties action menu: * Edit (add/remove/modify) test suites and benchmarks === Day Overview === Main view contains daily summary of key aspects influencing upcoming release. This is also the default view for the application. The page contains the following components: * packages * summary indicators, mostly numbers and links for detail pages) (could be a horizontal bar like in test cases below) * total number and details link * newly added * modified (version change) * packages that failed to build * action links: * see package details * test cases * total tests suites and test cases * progress indicator: horizontal bar with the following components * skipped tests * successful tests * failed tests * pending tests (there is no indicator of a 'running' test) * (never ran tests) - this is optional and will be displayed for historic entries, not the current day * action links: * see all tests (details) * edit skipped tests [optional] * benchmarks * selected benchmark results (value + spark line) * synthetic benchmarks * CPU * FPU * GPU (if possible) * IO: * USB thumb drive * USB 2.5" HDD * SD card * Network * NAND [optional] * end-user / application benchmarks * time to boot * time to website/cached * time to ... (etc) * notable changes (value, spark line, delta) (things that are not included by default but have changed radically since last test) * action links: * see all benchmarks (details) * define selected items * devices * all devices we have at our disposal * percentage of time devoted to: * running tests * being 'owned' by someone * being idle * being offline * bugs [optional] * all bugs filed yesterday that affect this image (could use specific tag, project to detect) === Image Details === This page can be reached from the 'daily overview' pages. It should contain basic information about all packages that were used to build the image. If possible each package should be a link to a launchpad page. For packages that are not tracked by launchpad (PPAs, custom packages) and to packages that are a part of a private PPA no link will be provided. [Optional]. This view could also provide a diff against any other day. Such difference views could be easy accessible from the project timeline. === Test Suite Details === This page can be reached from test suite list and test day overview (selected suites) Test suite details provides access to the following components: * Test suite URL: bzr branch that contains this test suite * Description * List of test cases * Summary of historical results (pass/fail/skipped) Actions: * Disable whole suite for specific hardware * Disable specific tests for specific hardware === Test Case Details === This page can be reached from test suite details. Test case details provides access to the following components: * Historical results (pass/fail/skip) * Preview of the relevant part of the log file that was harvested to get this result. === Benchmark Details === TODO. Mostly similar to test suite, except for some presentation differences. === Benchmark probe (single benchmark item) Details === TODO Mostly similar to test case, except for some presentation differences. |
Line 30: | Line 162: |
This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like: | Choose basic web technology: DONE (django) Choose database system: DONE (PSQL) Design data model (including data ingest requirements, data presentation requirements, data transformations and storage): TODO Design web widgets/pieces/components that we will need and determine how each fits into the data model: TODO? (is this required in the spec?) === Architecture Overview === {{drawing:QADashboardArchitecture}} The dashboard is a part of a larger set of blueprints/projects that together provide the QA infrastructure. The main components are: * Automatic Test Framework * Test Cases (dealing with providing actual tests) * WebKit testing/benchmarking * Dashboard: * frontend - web pages, interaction with launchpad, etc * backend - database model, processes, interactions with other components ==== Database Model ==== Make database model image: TODO |
Line 42: | Line 193: |
Include: * data migration, if any * redirects from old URLs to new ones, if any * how users will be pointed to the new way of doing things, if necessary. |
Currently there is no direct migration plan. Things we could consider is migrating some bits and pieces of technology already up and running either at Canonical or somewhere else in the community/other parties that is open source and integrate their tests into our framework. If that becomes true we might want to look at migration from qa-tool-foo to our new technology. |
Line 55: | Line 203: |
This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved. | * Do we roll our own technology or do we adapt existing frameworks (like Hudson) * How and where do we store the user database? We need at least two types of users: viewers and editors. Editors could alter per-image/per-device state such as which tests should not be run. Editors could add more test suites, news, milestones etc. This could be, to a certain degree, avoided by pulling all non-volatile non-test information from external sources (feeds/rsr). * How do we provision devices (bind particular instance of the dashboard with a particular device and ensure it can be identified across upgrades/reflashes/etc)? * Where is the test scheduler (is it in this spec or in the automated test framework spec) |
Line 59: | Line 211: |
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected. | Goal: define visualization interface for QA control that shows daily/snapshot/other summary of the 'health' of the image for a given platform Different images based on a common base: * server * desktop * netbookcurrent Stuff we want to show: * difference from yesterday * performance history * performance regressions * performance targets (as infrastructure to see if it works during the cycle) Dashboard mockup: Two columns: 1) - Current build status FTBFS count New package count, number of packages Latest build date/time - Test result - Build history 2) - News - Performance Targets Q: What about some UI for scheculing test runs: A: we're not targeting this for the first release but we want have a UI for doing that in the future Q: How does our project relate to other ubuntu QA projects A: Stuff to check: buildbot (python) hudson (java) Action item: check hudson out? Zygmunt Krynicki Hudson instance for Bzr at canonical: http://babune.ladeuil.net:24842/view/Ubuntu/job/selftest-jaunty/buildTimeTrend We want to store the log file of each test run just in case (for unexpected successes) |
Launchpad Entry: https://blueprints.launchpad.net/ubuntu/+spec/arm-m-validation-dashboard
Created: PaulLarson
Contributors: PaulLarson, ZygmuntKrynicki
Packages affected:
Summary
As part of the automated testing efforts on ARM, we would also like to have a dashboard interface for visualizing the current state of the image. This interface should allow the user to see, at a glance, the state of functional tests as well as performance tests, and any other data that may be useful.
Release Note
No user visible changes
Rationale
We would like to easily see how things are shaping up over time on arm, and how changes are impacting performance. A dashboard interface would help us visualize, in one place, the results of testing from multiple machines. It could also be used to generate graphs of performance metrics across different image build dates to quickly see if things are improving, or getting worse, and how far we are away from any goals that may exist.
User stories
Bob is a release manager for Ubuntu on a particular arm device. Bob wants to check the overall status of the image produced yesterday before releasing Alpha 1. Bob visits the dashboard to check for test failures. Bob marked some tests as expected to fail on this device as not all components are yet in place and some things are still broken. As all other tests have good results Bob can go forward with the release.
Jane is interested in basic performance metrics of current ubuntu image. Jane can check some synthetic benchmarks for CPU, GPU and IO performance. Jane can also check some end-to-end benchmarks for user applications (browser startup time, time to full desktop, time to render snapshot of key websites, etc). Jane can setup baseline for each metric and request to be notified of all variances that exceed given threshold.
Alice is on the Desktop QA team and would like to integrate some of the tests her team has created into the dashboard. QA Engineers quickly bootstrap a local installation of the dashboard and check the bundled documentation and examples. Within hours the dashboard can display results from some of the tests that local engineering team has already adapted.
Yung is a product manager in Big Corp ltd. Yung is building a product based on Ubuntu and wants to reuse our QA infrastructure. Yung instructs his engineers to deploy a local dashboard installation and run our tests on their new secret product. Engineers can also write an adapter that will take some of the results from the dashboard and push it to the internal QA system.
Assumptions
- Easy to deploy, including, outside of Canonical. All infrastructure components must be packaged and provided as a PPA, ready to install on a Lucid server. All device components must be packaged and uploaded to Maverick (TODO: which component? are we going straight-to-ppa or do we attempt to hit the main archive?)
- Focused on one device.
- FIXME - is this _really_ true? - if not how to avoid multiple devices and benchmarks
There seems to be a conflict of interests - we'd like to see this for >1 kind of device but vendors will not enjoy any device benchmark information being displayed, especially alongside competing devices
- FIXME - is this _really_ true? - if not how to avoid multiple devices and benchmarks
- One-way connectivity. Devices participating in the test can connect to the infrastructure services but reverse connection cannot be assumed. (TODO: what about IPv6?)
- Distributed environment. Devices and infrastructure components are places in diverse geographical and administrative zones.
- Launchpad integration for bug management. There is no plan to support third party bug trackers in the first release.
Design
TODO:
- UI design for each core use case (TODO)
- (we want a list of tasks users have to perform to get to the goal they are after (with regard to the use case lists above)
- UI design vs UI design of other Canonical technologies (TODO)
- Design web pages we need to provide (DONE)
Dashboard will feature the following pages/views:
Project Timeline
Recurring component of each page, shown at the top. Key aspects:
- Shows milestones
- Shows number of days till next milestone
- Shows number of days/weeks? till final milestone
- Allows to click at a past day to see historical records
Project timeline could also hold global image/project properties action menu:
- Edit (add/remove/modify) test suites and benchmarks
Day Overview
Main view contains daily summary of key aspects influencing upcoming release. This is also the default view for the application. The page contains the following components:
- packages
- summary indicators, mostly numbers and links for detail pages)
- (could be a horizontal bar like in test cases below)
- total number and details link
- newly added
- modified (version change)
- packages that failed to build
- action links:
- see package details
- summary indicators, mostly numbers and links for detail pages)
- test cases
- total tests suites and test cases
- progress indicator: horizontal bar with the following components
- skipped tests
- successful tests
- failed tests
- pending tests (there is no indicator of a 'running' test)
- (never ran tests) - this is optional and will be displayed for historic entries, not the current day
- action links:
- see all tests (details)
- edit skipped tests [optional]
- benchmarks
- selected benchmark results (value + spark line)
- synthetic benchmarks
- CPU
- FPU
- GPU (if possible)
- IO:
- USB thumb drive
- USB 2.5" HDD
- SD card
- Network
- NAND [optional]
- end-user / application benchmarks
- time to boot
- time to website/cached
- time to ... (etc)
- synthetic benchmarks
- notable changes (value, spark line, delta)
- (things that are not included by default but have changed radically since last test)
- action links:
- see all benchmarks (details)
- define selected items
- selected benchmark results (value + spark line)
- devices
- all devices we have at our disposal
- percentage of time devoted to:
- running tests
- being 'owned' by someone
- being idle
- being offline
- percentage of time devoted to:
- all devices we have at our disposal
- bugs [optional]
- all bugs filed yesterday that affect this image
- (could use specific tag, project to detect)
- all bugs filed yesterday that affect this image
Image Details
This page can be reached from the 'daily overview' pages. It should contain basic information about all packages that were used to build the image. If possible each package should be a link to a launchpad page. For packages that are not tracked by launchpad (PPAs, custom packages) and to packages that are a part of a private PPA no link will be provided.
[Optional]. This view could also provide a diff against any other day. Such difference views could be easy accessible from the project timeline.
Test Suite Details
This page can be reached from test suite list and test day overview (selected suites)
Test suite details provides access to the following components:
- Test suite URL: bzr branch that contains this test suite
- Description
- List of test cases
- Summary of historical results (pass/fail/skipped)
Actions:
- Disable whole suite for specific hardware
- Disable specific tests for specific hardware
Test Case Details
This page can be reached from test suite details.
Test case details provides access to the following components:
- Historical results (pass/fail/skip)
- Preview of the relevant part of the log file that was harvested to get this result.
Benchmark Details
TODO.
Mostly similar to test suite, except for some presentation differences.
Benchmark probe (single benchmark item) Details
TODO
Mostly similar to test case, except for some presentation differences.
Implementation
Choose basic web technology: DONE (django) Choose database system: DONE (PSQL) Design data model (including data ingest requirements, data presentation requirements, data transformations and storage): TODO Design web widgets/pieces/components that we will need and determine how each fits into the data model: TODO? (is this required in the spec?)
Architecture Overview
The dashboard is a part of a larger set of blueprints/projects that together provide the QA infrastructure. The main components are:
- Automatic Test Framework
- Test Cases (dealing with providing actual tests)
WebKit testing/benchmarking
- Dashboard:
- frontend - web pages, interaction with launchpad, etc
- backend - database model, processes, interactions with other components
Database Model
Make database model image: TODO
UI Changes
Should cover changes required to the UI, or specific UI that is required to implement this
Code Changes
Code changes should include an overview of what needs to change, and in some cases even the specific details.
Migration
Currently there is no direct migration plan. Things we could consider is migrating some bits and pieces of technology already up and running either at Canonical or somewhere else in the community/other parties that is open source and integrate their tests into our framework. If that becomes true we might want to look at migration from qa-tool-foo to our new technology.
Test/Demo Plan
It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.
This need not be added or completed until the specification is nearing beta.
Unresolved issues
* Do we roll our own technology or do we adapt existing frameworks (like Hudson) * How and where do we store the user database?
- We need at least two types of users: viewers and editors. Editors could alter per-image/per-device state such as which tests should not be run. Editors could add more test suites, news, milestones etc. This could be, to a certain degree, avoided by pulling all non-volatile non-test information from external sources (feeds/rsr).
* How do we provision devices (bind particular instance of the dashboard with a particular device and ensure it can be identified across upgrades/reflashes/etc)? * Where is the test scheduler (is it in this spec or in the automated test framework spec)
BoF agenda and discussion
Goal: define visualization interface for QA control that shows daily/snapshot/other summary of the 'health' of the image for a given platform
Different images based on a common base:
- server
- desktop
- netbookcurrent
Stuff we want to show:
- difference from yesterday
- performance history
- performance regressions
- performance targets (as infrastructure to see if it works during the cycle)
Dashboard mockup:
- Two columns: 1)
- - Current build status
- FTBFS count New package count, number of packages Latest build date/time
- - News - Performance Targets
- - Current build status
Q: What about some UI for scheculing test runs: A: we're not targeting this for the first release but we want have a UI for doing that in the future
Q: How does our project relate to other ubuntu QA projects A:
Stuff to check:
- buildbot (python) hudson (java)
Action item: check hudson out? Zygmunt Krynicki Hudson instance for Bzr at canonical: http://babune.ladeuil.net:24842/view/Ubuntu/job/selftest-jaunty/buildTimeTrend
We want to store the log file of each test run just in case (for unexpected successes)
Specs/M/ARMValidationDashboard (last edited 2010-06-04 12:08:03 by fdu90)