DailyDesktopTesting

Differences between revisions 4 and 5
⇤ ← Revision 4 as of 2009-06-08 14:52:43 →
Size: 3153
Editor: cpc4-oxfd8-0-0-cust39
Comment:
← Revision 5 as of 2009-06-09 23:02:42 → ⇥
Size: 2760
Editor: cpc4-oxfd8-0-0-cust39
Comment:
Deletions are marked like this. Additions are marked like this.
Line 18: Line 18:
 * Run Ubiquity installation from a live CD.
Line 34: Line 33:
 * The image will remain prestine as much as possible, the test harness will be executed from a working directory with all dependencies being local.  * The image will remain pristine as much as possible, the test harness will be executed from a working directory with all dependencies being local.
 * A boot parameter will be required to run tests directly on a Live CD.
Line 46: Line 46:
 * Run Ubiquity installation from a live CD.
Line 47: Line 48:
== BoF agenda and discussion ==

'''Remaining interesting notes, not integrated:'''

 * Might be useful to test software only under one image (rather than with every *buntu image)
  * Thus we get the software testing performed once per machine per day

==== How can we test the LiveCD environment automatically? ====
 * What we want to do:
  1. Boot KVM image from LiveCD
  1. Run tests (checkbox?)
  1. Install from LiveCD
  1. Run (more) tests


==== Action Items ====		 == Action Items ==

Summary

Add desktop tests to the daily hardware testing schedule on laptops and netbooks.

Rationale

The automated desktop testing framework and suites allows us to repeat a large volume of UI tests on a regular bases. Integrating these tests into the existing checkbox-based hardware certification testing setup will allow us to take advantage of it's schedule and results reporting system. Ubuntu's quality will greatly increase if we regularly run automated desktop tests.

Use Cases/Requirements

  • Run all automated desktop test suites every day on a daily (installed) image.
  • Run all automated desktop test suites every day on live CD.

Design

  • The implementation will rely on Checkbox.
  • It will use the certification website for reporting test results.
  • There will be a method of tagging certain tests as hardware dependent, this will assure the tests are run on relevant target machines, and not on incompatible hardware or virtualized targets.
  • The solution will be usable on both physical and virtual hardware, and will take advantages of the strengths of both:
    • PXE boot on hardware.
    • ISO loopback on VMs.
    • Snapshot mode it VMs.
    • Quick ghosting.

Implementation

  • The checkbox certification plugin set will be expanded to include a Ubuntu Desktop Testing (UDT) plugin that will execute the whole gamut of included automated test suites.
  • A Ubiquity suite will be authored for automating install tests.
  • Parallelizing test execution in a cloud will be investigated, and perhaps implemented.
  • The image will remain pristine as much as possible, the test harness will be executed from a working directory with all dependencies being local.
  • A boot parameter will be required to run tests directly on a Live CD.

Test/Demo Plan

The tests will be tested once the tests are run on a regular testing schedule, and the test results will be displayed in a human-readable fashion on the current hardware test website.

Unresolved issues

The Live CD needs a boot option for executing a command on startup.

Future work

  • Parallelize test suites across multiple targets to reduce test duration, perhaps with cloud-like elasticity.
  • Run Ubiquity installation from a live CD.

Action Items

  • determine 1-2 duplicate machines that can be removed from the PPA pool and used as permanent VM hosts (Ronald)
    • coordinate with Eitan
  • talk to installer team and find out best way to get LiveCD tests to execute (Marc)

CategorySpec

QATeam/Specs/DailyDesktopTesting (last edited 2009-06-09 23:02:42 by cpc4-oxfd8-0-0-cust39)