ServerKarmicAutomatedKvmTesting

Revision 3 as of 2009-11-23 21:39:00

Clear message

Summary

We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm, and various configurations.

Release Note

Ubuntu 10.04's KVM virtualization stack has undergone regular, automated testing.

Rationale

KVM is the virtualization hypervisor upon which we build the Ubuntu Enterprise Cloud, and as such, a cornerstone technology. We can, and should automate the testing of KVM.

User stories

  • As an administrator of a local installation of UEC, Katie wants to ensure the stability her users' instances, which run in KVM virtual machines.

Assumptions

  • Canonical IS will provide some VT-capable hardware which can run these automated tests.
  • Canonical QA will instrument checkbox or some other testing framework to test KVM and report the results.
  • Canonical Server Team will respond and fix bugs found/filed by QA.

Design/Implementation

How to Test

  • Autotest, KVM-Autotest

    • Wide support with Google, Red Hat, IBM, etc.
    • Use Step Files for different distros.
    • Currently use automated scripts to test for certain end products.
    • Ubuntu community should build Step Files, and get them committed to Autotest.
    • Autotest currently doesn't have a large number of fucntion tests.
      • Ubuntu could greatly expand these tests.

What to Test

  • Functionality, Performance, Stability, Regression
  • QEMU-KVM
    • Already tested by KVM-Autotest

    • We need to integrate these tests into Canonical's automated testing infrastructure (checkbox?)
    • Could also enhance the functional testing coverage
  • Libvirt
    • Do libvirt testing as well as KVM testing, as UEC users are affected by both of these
    • Add libvirt tests to Autotest.
    • libvirt-sim does some tests with Autotests.
    • Security team has some libvirt tests
  • Guest OSes
    • Linux
      • Ubuntu (and flavors), Debian, Fedora, CentOS, OpenSUSE
      • amd64 and i386
      • desktop and server
    • Windows
      • Windows 2003, XP, Vista, 7

Procedure

  • Build a collection of images that we care about testing
    • For Ubuntu, we could perhaps use daily UEC images (?)
    • If Windows is interesting, can use cygwin to get a bash shell
  • Guests attach a bash shell to the serial port
  • Test scripts interact with the shell
  • Run KVM-Autotest's functional tests
    • May need to enhance these (working with the upstream project) to ensure we cover the functionality that's important to us
      • e.g. dynamic block storage attach is something that we need for UEC, but it's not currently tested by upstream
    • Every test has a timeout
  • Do performance testing by timing certain processes
    • Network throughput
    • Disk throughput
    • CPU intensive calculations
    • Memory intensive calculations
    • Track results over time to identify if there is a regression
  • Integrate KVM-Autotest framework into hardware certification testing to "certify" KVM as a support hardware platform for Ubuntu releases
    • Treat as a native hardware platform.
    • Currently VMware is treated in this fashion.

When to Test

  • Alpha1, Alpha2, Alpha3, Beta1, Beta2, RC
  • At each upload of qemu-kvm, libvirt
  • Perhaps one day link this and test against the daily-builds

Upstream Contribution

  • Should submit "Step Files" to upstream kvm-autotest for supported Ubuntu releases (i386 and amd64)
    • These are more or less "preseed" files

Test/Demo Plan

  • Once the automation is in place, execute the automated tests whenever a new qemu-kvm or libvirt package is uploaded to the archive.
  • Provide the results in a web hosted manner.
  • Act upon the results by confirming any issues found, and filing appropriate bugs.

Unresolved issues

  • Need a named QA resource dedicated to this project
  • Need agreement on hardware in the Canonical lab where these tests can run

Discussion


CategorySpec