ServerKarmicAutomatedKvmTesting

Differences between revisions 6 and 7
Revision 6 as of 2009-11-23 22:41:10
Size: 4828
Editor: cpe-66-69-232-158
Comment:
Revision 7 as of 2009-11-25 18:42:45
Size: 5250
Editor: cpe-66-69-232-158
Comment: updated user stories
Deletions are marked like this. Additions are marked like this.
Line 24: Line 24:
 * As an administrator of a local installation of UEC, Katie wants to ensure the stability her users' instances, which run in KVM virtual machines.
 * As a developer of UEC, Victor wants to ensure that the KVM and Libvirt virtualization stack are stable and bug free, such that he can trust the foundation and focus his efforts on debugging UEC.
 * As an administrator of a local installation of UEC, I wants to ensure the stability her users' instances, which run in KVM virtual machines.  With Canonical/Ubuntu having regularly and heavily tested its virtualization stack, I can rest assured that my KVM/Libvirt virtual machines will be reliable and high-performance
 * As a developer of UEC, I wants to ensure that the KVM and Libvirt virtualization stack are stable and bug free. I can trust the virtualization foundation and focus my efforts on debugging UEC.
 * As a system administrator interested in migrating my infrastructure to virtualization-based solutions, I chose Ubuntu/KVM/Libvirt based on the extensive testing performed by Canonical and Ubuntu in the Lucid LTS cycle.
Line 30: Line 31:
 * Canonical QA will instrument checkbox or some other testing framework to enhance the test coverage, automatically trigger these tests, and report the results.  * Canonical QA will instrument checkbox or some other testing framework to enhance the test coverage, automatically trigger these tests, review the results, and report bugs accordingly.

Summary

We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm and libvirt, and various configurations.

Release Note

Ubuntu 10.04's virtualization infrastrucure has undergone regular, automated testing.

Rationale

KVM is the Ubuntu's virtualization hypervisor, and Libvirt is the abstraction layer by which Eucalyptus communicates with KVM. These fundamental technologies are the foundation upon which we build the Ubuntu Enterprise Cloud.

We can, and should automate the testing of KVM and Libvirt on a regular basis.

User stories

  • As an administrator of a local installation of UEC, I wants to ensure the stability her users' instances, which run in KVM virtual machines. With Canonical/Ubuntu having regularly and heavily tested its virtualization stack, I can rest assured that my KVM/Libvirt virtual machines will be reliable and high-performance
  • As a developer of UEC, I wants to ensure that the KVM and Libvirt virtualization stack are stable and bug free. I can trust the virtualization foundation and focus my efforts on debugging UEC.
  • As a system administrator interested in migrating my infrastructure to virtualization-based solutions, I chose Ubuntu/KVM/Libvirt based on the extensive testing performed by Canonical and Ubuntu in the Lucid LTS cycle.

Assumptions

  • Canonical IS will provide some VT-capable hardware which can run these automated tests.
  • Canonical QA will instrument checkbox or some other testing framework to enhance the test coverage, automatically trigger these tests, review the results, and report bugs accordingly.
  • Canonical Server Team will respond and fix bugs found/filed by QA.

Design/Implementation

Details in the sections below.

How to Test

  • Autotest, KVM-Autotest

    • Wide support with Google, Red Hat, IBM, etc.
    • Use Step Files for different distros.
    • Currently use automated scripts to test for certain end products.
    • Ubuntu community should build Step Files, and get them committed to Autotest.
    • Autotest currently doesn't have a large number of fucntion tests.
      • Ubuntu could greatly expand these tests.

What to Test

  • Functionality, Performance, Stability, Regression
  • QEMU-KVM
    • Already tested by KVM-Autotest

    • We need to integrate these tests into Canonical's automated testing infrastructure (checkbox?)
    • Could also enhance the functional testing coverage
  • Libvirt
    • Do libvirt testing as well as KVM testing, as UEC users are affected by both of these
    • Add libvirt tests to Autotest.
    • libvirt-sim does some tests with Autotests.
    • Security team has some libvirt tests
  • Guest OSes
    • Linux
      • Ubuntu (and flavors), Debian, Fedora, CentOS, OpenSUSE
      • amd64 and i386
      • desktop and server
    • Windows
      • Windows 2003, XP, Vista, 7

Procedure

  • Build a collection of images that we care about testing
    • For Ubuntu, we could perhaps use daily UEC images (?)
    • If Windows is interesting, can use cygwin to get a bash shell
  • Guests attach a bash shell to the serial port
  • Test scripts interact with the shell
  • Run KVM-Autotest's functional tests
    • May need to enhance these (working with the upstream project) to ensure we cover the functionality that's important to us
      • e.g. dynamic block storage attach is something that we need for UEC, but it's not currently tested by upstream
    • Every test has a timeout
  • Do performance testing by timing certain processes
    • Network throughput
    • Disk throughput
    • CPU intensive calculations
    • Memory intensive calculations
    • Track results over time to identify if there is a regression
  • Integrate KVM-Autotest framework into hardware certification testing to "certify" KVM as a support hardware platform for Ubuntu releases
    • Treat as a native hardware platform.
    • Currently VMware is treated in this fashion.

When to Test

  • Alpha1, Alpha2, Alpha3, Beta1, Beta2, RC
  • At each upload of qemu-kvm, libvirt
  • Perhaps one day link this and test against the daily-builds

Upstream Contribution

  • Should submit "Step Files" to upstream kvm-autotest for supported Ubuntu releases (i386 and amd64)
    • These are more or less "preseed" files
  • Additional upstream tests

Test/Demo Plan

  • Once the automation is in place, execute the automated tests whenever a new qemu-kvm or libvirt package is uploaded to the archive.
  • Provide the results in a web hosted manner.
  • Act upon the results by confirming any issues found, and filing appropriate bugs.

Unresolved issues

  • Need a named QA resource dedicated to this project
  • Need agreement on hardware in the Canonical lab where these tests can run

Discussion


CategorySpec

ServerKarmicAutomatedKvmTesting (last edited 2010-01-26 10:31:44 by 63)