Launchpad Entry: karmic-qa-daily-desktop-testing
Add desktop tests to the daily hardware testing schedule on laptops and netbooks.
The automated desktop testing framework and suites allows us to repeat a large volume of UI tests on a regular bases. Integrating these tests into the existing checkbox-based hardware certification testing setup will allow us to take advantage of it's schedule and results reporting system. Ubuntu's quality will greatly increase if we regularly run automated desktop tests.
- Run all automated desktop test suites every day on a daily (installed) image.
- Run all automated desktop test suites every day on live CD.
- The implementation will rely on Checkbox.
- It will use the certification website for reporting test results.
- There will be a method of tagging certain tests as hardware dependent, this will assure the tests are run on relevant target machines, and not on incompatible hardware or virtualized targets.
- The solution will be usable on both physical and virtual hardware, and will take advantages of the strengths of both:
- PXE boot on hardware.
- ISO loopback on VMs.
- Snapshot mode it VMs.
- Quick ghosting.
- The checkbox certification plugin set will be expanded to include a Ubuntu Desktop Testing (UDT) plugin that will execute the whole gamut of included automated test suites.
- A Ubiquity suite will be authored for automating install tests.
- Parallelizing test execution in a cloud will be investigated, and perhaps implemented.
- The image will remain pristine as much as possible, the test harness will be executed from a working directory with all dependencies being local.
- A boot parameter will be required to run tests directly on a Live CD.
The tests will be tested once the tests are run on a regular testing schedule, and the test results will be displayed in a human-readable fashion on the current hardware test website.
The Live CD needs a boot option for executing a command on startup.
- Parallelize test suites across multiple targets to reduce test duration, perhaps with cloud-like elasticity.
- Run Ubiquity installation from a live CD.
- determine 1-2 duplicate machines that can be removed from the PPA pool and used as permanent VM hosts (Ronald)
- coordinate with Eitan
- talk to installer team and find out best way to get LiveCD tests to execute (Marc)