ServerKarmicAutomatedKvmTesting
Differences between revisions 2 and 4 (spanning 2 versions)
|
Size: 4013
Comment: updated for Lucid
|
Size: 4377
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 6: | Line 6: |
| * '''Packages affected''': qemu-kvm | * '''Packages affected''': qemu-kvm, libvirt |
| Line 10: | Line 10: |
| We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm, and various configurations. | We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm and libvirt, and various configurations. |
| Line 14: | Line 14: |
| Ubuntu 10.04's KVM virtualization stack has undergone regular, automated testing. | Ubuntu 10.04's virtualization infrastrucure has undergone regular, automated testing. |
| Line 30: | Line 30: |
| == Design/Implementation/Test/Demo Plan == | == Design/Implementation == |
| Line 32: | Line 32: |
| This specification is in fact mostly a Test Plan. | === How to Test === * [[http://autotest.kernel.org|Autotest]], [[http://www.linux-kvm.org/page/KVM-Autotest|KVM-Autotest]] * Wide support with Google, Red Hat, IBM, etc. * Use Step Files for different distros. * Currently use automated scripts to test for certain end products. * Ubuntu community should build Step Files, and get them committed to Autotest. * Autotest currently doesn't have a large number of fucntion tests. * Ubuntu could greatly expand these tests. === What to Test === * Functionality, Performance, Stability, Regression * QEMU-KVM * Already tested by [[http://www.linux-kvm.org/page/KVM-Autotest|KVM-Autotest]] * We need to integrate these tests into Canonical's automated testing infrastructure (checkbox?) * Could also enhance the functional testing coverage * Libvirt * Do libvirt testing as well as KVM testing, as UEC users are affected by both of these * Add libvirt tests to Autotest. * libvirt-sim does some tests with Autotests. * Security team has some libvirt tests * Guest OSes * Linux * Ubuntu (and flavors), Debian, Fedora, CentOS, OpenSUSE * amd64 and i386 * desktop and server * Windows * Windows 2003, XP, Vista, 7 === Procedure === * Build a collection of images that we care about testing * For Ubuntu, we could perhaps use daily UEC images (?) * If Windows is interesting, can use cygwin to get a bash shell * Guests attach a bash shell to the serial port * Test scripts interact with the shell * Run KVM-Autotest's functional tests * May need to enhance these (working with the upstream project) to ensure we cover the functionality that's important to us * e.g. dynamic block storage attach is something that we need for UEC, but it's not currently tested by upstream * Every test has a timeout * Do performance testing by timing certain processes * Network throughput * Disk throughput * CPU intensive calculations * Memory intensive calculations * Track results over time to identify if there is a regression * Integrate KVM-Autotest framework into hardware certification testing to "certify" KVM as a support hardware platform for Ubuntu releases * Treat as a native hardware platform. * Currently VMware is treated in this fashion. === When to Test === * Alpha1, Alpha2, Alpha3, Beta1, Beta2, RC * At each upload of qemu-kvm, libvirt * Perhaps one day link this and test against the daily-builds === Upstream Contribution === * Should submit "Step Files" to upstream kvm-autotest for supported Ubuntu releases (i386 and amd64) * These are more or less "preseed" files == Test/Demo Plan == * Once the automation is in place, execute the automated tests whenever a new qemu-kvm or libvirt package is uploaded to the archive. * Provide the results in a web hosted manner. * Act upon the results by confirming any issues found, and filing appropriate bugs. |
| Line 36: | Line 98: |
| == UDS/Karmic Raw Notes == | * Need a named QA resource dedicated to this project * Need agreement on hardware in the Canonical lab where these tests can run |
| Line 38: | Line 101: |
| * Use Autotest * Wide support with Google, Red Hat, IBM, etc. * Use Step Files for different distros. * Currently use automated scripts to test for certain end products. * Ubuntu community should build Step Files, and get them committed to Autotest. * Autotest currently doesn't have a large number of fucntion tests. * Ubuntu could greatly expand these tests. * Do libvirt testing as well as KVM testing. * Add libvirt tests to Autotest. * libvirt-sim does some tests with Autotests. * Need much more functional testing for KVM. * From a User perspective, should we be testing Windows VMs. * What are Ubuntu's requirements for testing. * Guests to Support: * Supported versions of Ubuntu Desktop and Server i386, amd64 (Hardy, Jaunty, etc). * Secondarily Kubuntu, Xubuntu, etc. * RHEL, Fedora * SUSE, OpenSUSE * Windows -- Functional not so much performance. * Windows 2003 Server * Windows 2003 Server * Windows XP * Windows 7 * Tests should hopefully find regressions. * Automated test procedure: * Start out with a guest that has a bash shell on a serial port. * Scripts to interact with the shell. * Can put cygwin on Windows to provide a bash shell. * Build a collection of images that we care about testing. * Determine which functional tests are important. * Every test has a timeout. * Do performance testing by timing the entire process. * Should determine very quickly if there is a regression. * Integrate current hardware certification testing into Autotest framework. * The framework is Open Source. * Some tests are provided by specific hardware vendor, and there may be issues integrating with Autotest. * Guest certification -- What level of testing is appropriate for an Ubuntu guest running on another distro like Fedora? * Would be treated as a native hardware platform. * Currently VMware is treated in this fashion. * With upstream release kvm-XX.XX is supported on Ubuntu if it passes a standard set of tests. * Most of the work needs to be done regarding developing the tests, not so much on which framework to use. * Test Times: * Alpha, Beta * Test at the commit level. * Deliverables: * Owe upstream Step Files for supported releases (i386 and amd64). * Preseed data suppmitted in conjunction with Step Files. * Where to run tests: * Upstream and Canonical will run tests concurrently. * Test suite that user's can run to test their hardware. * How to test KVM plus the stack. * KVM with libvirt * libvirt with virt-manager * Determine better bug reporting process. * Reporting bugs upstream to KVM. |
== Discussion == |
Launchpad Entry: server-lucid-automated-kvm-testing
Created: 2009-11-23
Contributors: DustinKirkland
Packages affected: qemu-kvm, libvirt
Summary
We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm and libvirt, and various configurations.
Release Note
Ubuntu 10.04's virtualization infrastrucure has undergone regular, automated testing.
Rationale
KVM is the virtualization hypervisor upon which we build the Ubuntu Enterprise Cloud, and as such, a cornerstone technology. We can, and should automate the testing of KVM.
User stories
- As an administrator of a local installation of UEC, Katie wants to ensure the stability her users' instances, which run in KVM virtual machines.
Assumptions
- Canonical IS will provide some VT-capable hardware which can run these automated tests.
- Canonical QA will instrument checkbox or some other testing framework to test KVM and report the results.
- Canonical Server Team will respond and fix bugs found/filed by QA.
Design/Implementation
How to Test
- Wide support with Google, Red Hat, IBM, etc.
- Use Step Files for different distros.
- Currently use automated scripts to test for certain end products.
- Ubuntu community should build Step Files, and get them committed to Autotest.
- Autotest currently doesn't have a large number of fucntion tests.
- Ubuntu could greatly expand these tests.
What to Test
- Functionality, Performance, Stability, Regression
- QEMU-KVM
Already tested by KVM-Autotest
- We need to integrate these tests into Canonical's automated testing infrastructure (checkbox?)
- Could also enhance the functional testing coverage
- Libvirt
- Do libvirt testing as well as KVM testing, as UEC users are affected by both of these
- Add libvirt tests to Autotest.
- libvirt-sim does some tests with Autotests.
- Security team has some libvirt tests
- Guest OSes
- Linux
- Ubuntu (and flavors), Debian, Fedora, CentOS, OpenSUSE
- amd64 and i386
- desktop and server
- Windows
- Windows 2003, XP, Vista, 7
- Linux
Procedure
- Build a collection of images that we care about testing
- For Ubuntu, we could perhaps use daily UEC images (?)
- If Windows is interesting, can use cygwin to get a bash shell
- Guests attach a bash shell to the serial port
- Test scripts interact with the shell
- Run KVM-Autotest's functional tests
- May need to enhance these (working with the upstream project) to ensure we cover the functionality that's important to us
- e.g. dynamic block storage attach is something that we need for UEC, but it's not currently tested by upstream
- Every test has a timeout
- May need to enhance these (working with the upstream project) to ensure we cover the functionality that's important to us
- Do performance testing by timing certain processes
- Network throughput
- Disk throughput
- CPU intensive calculations
- Memory intensive calculations
- Track results over time to identify if there is a regression
- Integrate KVM-Autotest framework into hardware certification testing to "certify" KVM as a support hardware platform for Ubuntu releases
- Treat as a native hardware platform.
- Currently VMware is treated in this fashion.
When to Test
- Alpha1, Alpha2, Alpha3, Beta1, Beta2, RC
- At each upload of qemu-kvm, libvirt
- Perhaps one day link this and test against the daily-builds
Upstream Contribution
- Should submit "Step Files" to upstream kvm-autotest for supported Ubuntu releases (i386 and amd64)
- These are more or less "preseed" files
Test/Demo Plan
- Once the automation is in place, execute the automated tests whenever a new qemu-kvm or libvirt package is uploaded to the archive.
- Provide the results in a web hosted manner.
- Act upon the results by confirming any issues found, and filing appropriate bugs.
Unresolved issues
- Need a named QA resource dedicated to this project
- Need agreement on hardware in the Canonical lab where these tests can run
Discussion
ServerKarmicAutomatedKvmTesting (last edited 2010-01-26 10:31:44 by 63)