DailyDesktopTesting
Differences between revisions 3 and 4
|
Size: 5150
Comment:
|
Size: 3153
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 12: | Line 12: |
| == Release Note == Ubuntu's quality will greatly increase if we regularly run automated desktop tests. |
|
| Line 17: | Line 13: |
| The automated desktop testing framework and suites allows us to repeat a large volume of UI tests on a regular bases. Integrating these tests into the existing checkbox-based hardware certification testing setup will allow us to take advantage of it's schedule and results reporting system. | The automated desktop testing framework and suites allows us to repeat a large volume of UI tests on a regular bases. Integrating these tests into the existing checkbox-based hardware certification testing setup will allow us to take advantage of it's schedule and results reporting system. Ubuntu's quality will greatly increase if we regularly run automated desktop tests. |
| Line 23: | Line 19: |
| * Parallelize test suites across multiple targets to reduce test duration, perhaps with cloud-like elasticity. == Assumptions == |
|
| Line 49: | Line 42: |
== Future work == * Parallelize test suites across multiple targets to reduce test duration, perhaps with cloud-like elasticity. |
|
| Line 50: | Line 48: |
| * Checkbox is running daily across multiple machines * Is there any reason to *not* use Checkbox to perform daily desktop testing? * What is necessary to add to the LDTP plugin in Checkbox to make this work? * As desktop testing is mostly software there is no need to run it under real HW -> virtualization |
'''Remaining interesting notes, not integrated:''' |
| Line 58: | Line 53: |
==== Are there compelling reasons to use virtualization? ==== * Ability to test live ISO image * Totally clean environment for each test * Can snap off copies of installed image rather than doing a wipe and install * saves time and avoids potential archive changes in the meantime * provides consistent environment for parallel tests * Greater protection in the case of potential insecure code * Kees and James Troup are good resources to discuss security of network ==== Issues with virtualization ==== * EC2 can't boot ISOs; need to create an image * but if we do KVM testing on our hardware this is not an issue * Difficult to test X (e.g. compositing) * Some issues are related to real hardware * E.g. testing X drivers from different vendors ==== Test definitions need to be aware of whether they are specifically hardware related ==== * This allows us to schedule tests that need to be run on physical HW to be run there and others on VMs ==== Proposal ==== * do a LiveCD install on a physical machine, then do a KVM CD install on this install, then perform desktop tests inside the KVM image * Some tests can be performed on the (physical) host for the tests that require physical hardware * We should be able to set aside some duplicate hardware to use as a host (and never tear it down to do physical HW testing) |
|
| Line 90: | Line 61: |
| ==== Dependency Issues ==== * Testing from packages has a potential issue if the test package(s) depend on other packages * You might pull in a dependency for the test package(s) that aren't geting pulled in but should be for tested programs * Tests should not change the package state * wget a tarball or something instead |
Launchpad Entry: karmic-qa-daily-desktop-testing
Created:
Contributors:
Packages affected:
Summary
Add desktop tests to the daily hardware testing schedule on laptops and netbooks.
Rationale
The automated desktop testing framework and suites allows us to repeat a large volume of UI tests on a regular bases. Integrating these tests into the existing checkbox-based hardware certification testing setup will allow us to take advantage of it's schedule and results reporting system. Ubuntu's quality will greatly increase if we regularly run automated desktop tests.
Use Cases/Requirements
- Run all automated desktop test suites every day on a daily (installed) image.
- Run all automated desktop test suites every day on live CD.
- Run Ubiquity installation from a live CD.
Design
- The implementation will rely on Checkbox.
- It will use the certification website for reporting test results.
- There will be a method of tagging certain tests as hardware dependent, this will assure the tests are run on relevant target machines, and not on incompatible hardware or virtualized targets.
- The solution will be usable on both physical and virtual hardware, and will take advantages of the strengths of both:
- PXE boot on hardware.
- ISO loopback on VMs.
- Snapshot mode it VMs.
- Quick ghosting.
Implementation
- The checkbox certification plugin set will be expanded to include a Ubuntu Desktop Testing (UDT) plugin that will execute the whole gamut of included automated test suites.
- A Ubiquity suite will be authored for automating install tests.
- Parallelizing test execution in a cloud will be investigated, and perhaps implemented.
- The image will remain prestine as much as possible, the test harness will be executed from a working directory with all dependencies being local.
Test/Demo Plan
The tests will be tested once the tests are run on a regular testing schedule, and the test results will be displayed in a human-readable fashion on the current hardware test website.
Unresolved issues
The Live CD needs a boot option for executing a command on startup.
Future work
- Parallelize test suites across multiple targets to reduce test duration, perhaps with cloud-like elasticity.
BoF agenda and discussion
Remaining interesting notes, not integrated:
- Might be useful to test software only under one image (rather than with every *buntu image)
- Thus we get the software testing performed once per machine per day
How can we test the LiveCD environment automatically?
- What we want to do:
- Boot KVM image from LiveCD
- Run tests (checkbox?)
- Install from LiveCD
- Run (more) tests
Action Items
- determine 1-2 duplicate machines that can be removed from the PPA pool and used as permanent VM hosts (Ronald)
- coordinate with Eitan
- talk to installer team and find out best way to get LiveCD tests to execute (Marc)
QATeam/Specs/DailyDesktopTesting (last edited 2009-06-09 23:02:42 by cpc4-oxfd8-0-0-cust39)