Ubuntu Wiki

• Launchpad Entry: qa-maverick-mago-daily

• Created: 2010-04-23

• Contributors: apulido

• Packages affected: mago, checkbox

Summary

Mago needs to be run on a daily (or bi-weekly) basis to be useful. In the past, we tried using checkbox, but this had several problems.

We will discussing how to improve this implementation or other alternatives to run Mago tests daily.

Release Note

This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)

It is mandatory.

Rationale

Right now, Mago tests are running exclusively by the testers themselves. To make the most of this effort we need to run tests in a daily basis and in an automated fashion and results publicly available.

Having the tests run daily will be also useful for the tests maintainers, who will know that a test broke as soon as it happened.

User stories

• Dave is a tester-developer who is in charge of maintaining GUI tests for Update Manager. Update Manager changed the UI in the latest upload. He will notice that the tests were broken the day after the upload.
• Michael is the maintainer of Update Manager. He can check on a daily basis if an upload of a dependency broke his application.

Design

We will be using a Hudson server. Some other teams are using a Hudson server for automated builds and tests. We won't duplicate efforts using another different technology.

We will use Xvfb in order to run Mago tests in the server. The DX team is already using this to run mago tests.

Implementation

Action items

• [apulido] Set a Hudson server locally
• [apulido] Investigate how the DX team run mago tests in Xvfb
• [apulido] Create a test project that runs some of the available tests
• [apulido] Request IS for a server to run these tests
• [apulido] Move configuration to the server

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

BoF agenda and discussion

Agenda

• Mago & Checkbox: current problems

• AT-SPI & GTK: current problems

• Infrastructure
• Where to report to
• How to gain feedback to maintain the tests (test error v.s. failures)

Discussion

 * Mago & Checkbox: current problems
 * AT-SPI & GTK: current problems
 * Infrastructure
  * ACTION: Ara to get a machine with KVM to test there
  * Using a live session instead of installing
 * Where to report to
  * Evan is looking into Hudson as a server to run installation, upgrade tests
  ACTION: Ara to follow-up with Evan on Hudson
 * How to gain feedback to maintain the tests (test error v.s. failures)
  * A web site with a summary
 * VFB - can it be leveraged to decouple mago from the running display

Software Centre

 * Part of the source code

 * Turning a11y on by default?
   - Would bring more attention to existing bugs
   - compiz performance problems with it
   
 * Start with checkbox-cli to avoid the GTK problems
  ACTION: Javier to look at checkbox-cli to integrate mago tests
    
ACTION: James to send GTK sample to desktop testing list
ACTION: Ara to prepare an SRU for LDTP2

CategorySpec