DailyDesktopTesting

Differences between revisions 2 and 3
⇤ ← Revision 2 as of 2009-05-26 08:48:42 →
Size: 4935
Editor: 80
Comment: Added notes from UDS discussion
← Revision 3 as of 2009-05-27 15:57:34 → ⇥
Size: 5150
Editor: 80
Comment:
Deletions are marked like this. Additions are marked like this.
Line 14: Line 14:
This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)

It is mandatory.		 Ubuntu's quality will greatly increase if we regularly run automated desktop tests.
Line 19: Line 17:
The automated desktop testing framework and suites allows us to repeat a large volume of UI tests on a regular bases. Integrating these tests into the existing checkbox-based hardware certification testing setup will allow us to take advantage of it's schedule and results reporting system.
Line 20: Line 19:
This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified.

== User stories ==		 == Use Cases/Requirements ==
 * Run all automated desktop test suites every day on a daily (installed) image.
 * Run all automated desktop test suites every day on live CD.
 * Run Ubiquity installation from a live CD.
 * Parallelize test suites across multiple targets to reduce test duration, perhaps with cloud-like elasticity.
Line 27: Line 28:

You can have subsections that better describe specific parts of the issue.		  * The implementation will rely on Checkbox.
 * It will use the certification website for reporting test results.
 * There will be a method of tagging certain tests as hardware dependent, this will assure the tests are run on relevant target machines, and not on incompatible hardware or virtualized targets.
 * The solution will be usable on both physical and virtual hardware, and will take advantages of the strengths of both:
  * PXE boot on hardware.
  * ISO loopback on VMs.
  * Snapshot mode it VMs.
  * Quick ghosting.
Line 31: Line 38:

This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

=== UI Changes ===

Should cover changes required to the UI, or specific UI that is required to implement this

=== Code Changes ===

Code changes should include an overview of what needs to change, and in some cases even the specific details.

=== Migration ===

Include:
 * data migration, if any
 * redirects from old URLs to new ones, if any
 * how users will be pointed to the new way of doing things, if necessary.		  * The checkbox certification plugin set will be expanded to include a Ubuntu Desktop Testing (UDT) plugin that will execute the whole gamut of included automated test suites.
 * A Ubiquity suite will be authored for automating install tests.
 * Parallelizing test execution in a cloud will be investigated, and perhaps implemented.
 * The image will remain prestine as much as possible, the test harness will be executed from a working directory with all dependencies being local.
Line 50: Line 44:

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.		 The tests will be tested once the tests are run on a regular testing schedule, and the test results will be displayed in a human-readable fashion on the current hardware test website.
Line 56: Line 47:

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.		 The Live CD needs a boot option for executing a command on startup.

Summary

Add desktop tests to the daily hardware testing schedule on laptops and netbooks.

Release Note

Ubuntu's quality will greatly increase if we regularly run automated desktop tests.

Rationale

The automated desktop testing framework and suites allows us to repeat a large volume of UI tests on a regular bases. Integrating these tests into the existing checkbox-based hardware certification testing setup will allow us to take advantage of it's schedule and results reporting system.

Use Cases/Requirements

  • Run all automated desktop test suites every day on a daily (installed) image.
  • Run all automated desktop test suites every day on live CD.
  • Run Ubiquity installation from a live CD.
  • Parallelize test suites across multiple targets to reduce test duration, perhaps with cloud-like elasticity.

Assumptions

Design

  • The implementation will rely on Checkbox.
  • It will use the certification website for reporting test results.
  • There will be a method of tagging certain tests as hardware dependent, this will assure the tests are run on relevant target machines, and not on incompatible hardware or virtualized targets.
  • The solution will be usable on both physical and virtual hardware, and will take advantages of the strengths of both:
    • PXE boot on hardware.
    • ISO loopback on VMs.
    • Snapshot mode it VMs.
    • Quick ghosting.

Implementation

  • The checkbox certification plugin set will be expanded to include a Ubuntu Desktop Testing (UDT) plugin that will execute the whole gamut of included automated test suites.
  • A Ubiquity suite will be authored for automating install tests.
  • Parallelizing test execution in a cloud will be investigated, and perhaps implemented.
  • The image will remain prestine as much as possible, the test harness will be executed from a working directory with all dependencies being local.

Test/Demo Plan

The tests will be tested once the tests are run on a regular testing schedule, and the test results will be displayed in a human-readable fashion on the current hardware test website.

Unresolved issues

The Live CD needs a boot option for executing a command on startup.

BoF agenda and discussion

  • Checkbox is running daily across multiple machines
    • Is there any reason to *not* use Checkbox to perform daily desktop testing?
    • What is necessary to add to the LDTP plugin in Checkbox to make this work?

  • As desktop testing is mostly software there is no need to run it under real HW -> virtualization

  • Might be useful to test software only under one image (rather than with every *buntu image)
    • Thus we get the software testing performed once per machine per day

Are there compelling reasons to use virtualization?

  • Ability to test live ISO image
  • Totally clean environment for each test
  • Can snap off copies of installed image rather than doing a wipe and install
    • saves time and avoids potential archive changes in the meantime
    • provides consistent environment for parallel tests
  • Greater protection in the case of potential insecure code
    • Kees and James Troup are good resources to discuss security of network

Issues with virtualization

  • EC2 can't boot ISOs; need to create an image
    • but if we do KVM testing on our hardware this is not an issue
  • Difficult to test X (e.g. compositing)
  • Some issues are related to real hardware
    • E.g. testing X drivers from different vendors

  • This allows us to schedule tests that need to be run on physical HW to be run there and others on VMs

Proposal

  • do a LiveCD install on a physical machine, then do a KVM CD install on this install, then perform desktop tests inside the KVM image
  • Some tests can be performed on the (physical) host for the tests that require physical hardware
  • We should be able to set aside some duplicate hardware to use as a host (and never tear it down to do physical HW testing)

How can we test the LiveCD environment automatically?

  • What we want to do:
    1. Boot KVM image from LiveCD
    2. Run tests (checkbox?)
    3. Install from LiveCD
    4. Run (more) tests

Dependency Issues

  • Testing from packages has a potential issue if the test package(s) depend on other packages
  • You might pull in a dependency for the test package(s) that aren't geting pulled in but should be for tested programs
    • Tests should not change the package state
    • wget a tarball or something instead

Action Items

  • determine 1-2 duplicate machines that can be removed from the PPA pool and used as permanent VM hosts (Ronald)
    • coordinate with Eitan
  • talk to installer team and find out best way to get LiveCD tests to execute (Marc)

CategorySpec

QATeam/Specs/DailyDesktopTesting (last edited 2009-06-09 23:02:42 by cpc4-oxfd8-0-0-cust39)