ServerKarmicAutomatedKvmTesting

Differences between revisions 1 and 9 (spanning 8 versions)
Revision 1 as of 2009-06-03 18:35:09
Size: 4743
Editor: cpe-66-69-254-183
Comment:
Revision 9 as of 2009-11-25 18:58:02
Size: 5515
Editor: cpe-66-69-232-158
Comment: added a bit more about guest testing
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
 * '''Launchpad Entry''': UbuntuSpec:server-karmic-automated-kvm-testing
 * '''Created''': 2009-06-03
 * '''Launchpad Entry''': UbuntuSpec:server-lucid-automated-kvm-testing
 * '''Created''': 2009-11-23
Line 6: Line 6:
 * '''Packages affected''': qemu-kvm  * '''Packages affected''': qemu-kvm, libvirt
Line 10: Line 10:
We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm. We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm and libvirt, and various configurations.
Line 14: Line 14:
Ubuntu 9.10's KVM virtualization stack has undergone regular, automated testing. Ubuntu 10.04's virtualization infrastrucure has undergone regular, automated testing.
Line 18: Line 18:
This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified. KVM is the Ubuntu's virtualization hypervisor, and Libvirt is the abstraction layer by which Eucalyptus communicates with KVM. These fundamental technologies are the foundation upon which we build the Ubuntu Enterprise Cloud.

We can, and should automate the testing of KVM and Libvirt on a regular basis.
Line 22: Line 24:
 * As an administrator of a local installation of UEC, I wants to ensure the stability her users' instances, which run in KVM virtual machines. With Canonical/Ubuntu having regularly and heavily tested its virtualization stack, I can rest assured that my KVM/Libvirt virtual machines will be reliable and high-performance
 * As a developer of UEC, I wants to ensure that the KVM and Libvirt virtualization stack are stable and bug free. I can trust the virtualization foundation and focus my efforts on debugging UEC.
 * As a system administrator interested in migrating my infrastructure to virtualization-based solutions, I chose Ubuntu/KVM/Libvirt based on the extensive testing performed by Canonical and Ubuntu in the Lucid LTS cycle.
Line 24: Line 30:
== Design ==  * Canonical IS will provide some VT-capable hardware which can run these automated tests.
 * Canonical QA will instrument checkbox or some other testing framework to enhance the test coverage, automatically trigger these tests, review the results, and report bugs accordingly.
 * Canonical Server Team will respond and fix bugs found/filed by QA.
Line 26: Line 34:
You can have subsections that better describe specific parts of the issue. == Design/Implementation ==
Line 28: Line 36:
== Implementation == Details in the sections below.
Line 30: Line 38:
This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like: === How to Test ===
Line 32: Line 40:
=== UI Changes ===  * [[http://autotest.kernel.org|Autotest]], [[http://www.linux-kvm.org/page/KVM-Autotest|KVM-Autotest]]
  * Wide support with Google, Red Hat, IBM, etc.
  * Use Step Files for different distros.
  * Currently use automated scripts to test for certain end products.
  * Ubuntu community should build Step Files, and get them committed to Autotest.
  * Autotest currently doesn't have a large number of fucntion tests.
   * Ubuntu could greatly expand these tests.
Line 34: Line 48:
Should cover changes required to the UI, or specific UI that is required to implement this === What to Test ===
 * Functionality, Performance, Stability, Regression
 * QEMU-KVM
  * Already tested by [[http://www.linux-kvm.org/page/KVM-Autotest|KVM-Autotest]]
  * We need to integrate these tests into Canonical's automated testing infrastructure (checkbox?)
  * Could also enhance the functional testing coverage
 * Libvirt
  * Do libvirt testing as well as KVM testing, as UEC users are affected by both of these
  * Add libvirt tests to Autotest.
  * libvirt-sim does some tests with Autotests.
  * Security team has some libvirt tests
 * Guest OSes
  * Linux
    * Ubuntu 8.04 LTS, 10.04 LTS, Desktop, Server
    * Debian 5.0
    * Fedora 12
    * CentOS 5
    * OpenSUSE 11.2
   * amd64 and i386
   * desktop and server
  * Windows
   * Windows 2003, XP, Vista, 7
  * Ensure that these guest OSes
   * Install
   * Boot
   * Have network connectivity
   * Sound (less important)
   * USB passthrough (less important)
Line 36: Line 77:
=== Code Changes === === Procedure ===
 * Build a collection of images that we care about testing
  * For Ubuntu, we could perhaps use daily UEC images (?)
  * If Windows is interesting, can use cygwin to get a bash shell
 * Guests attach a bash shell to the serial port
 * Test scripts interact with the shell
 * Run KVM-Autotest's functional tests
  * May need to enhance these (working with the upstream project) to ensure we cover the functionality that's important to us
   * e.g. dynamic block storage attach is something that we need for UEC, but it's not currently tested by upstream
  * Every test has a timeout
 * Do performance testing by timing certain processes (using existing checkbox performance tests)
  * Network throughput
  * Disk throughput
  * CPU intensive calculations
  * Memory intensive calculations
  * Track results over time to identify if there is a regression
 * Integrate KVM-Autotest framework into hardware certification testing to "certify" KVM as a support hardware platform for Ubuntu releases
  * Treat as a native hardware platform.
  * Currently VMware is treated in this fashion.
Line 38: Line 97:
Code changes should include an overview of what needs to change, and in some cases even the specific details. === When to Test ===
 * Alpha1, Alpha2, Alpha3, Beta1, Beta2, RC
 * At each upload of qemu-kvm, libvirt
 * Perhaps one day link this and test against the daily-builds
Line 40: Line 102:
=== Migration ===

Include:
 * data migration, if any
 * redirects from old URLs to new ones, if any
 * how users will be pointed to the new way of doing things, if necessary.
=== Upstream Contribution ===
 * Should submit "Step Files" to upstream kvm-autotest for supported Ubuntu releases (i386 and amd64)
  * These are more or less "preseed" files
 * Additional upstream tests
Line 49: Line 109:
It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.
 * Once the automation is in place, execute the automated tests whenever a new qemu-kvm or libvirt package is uploaded to the archive.
 * Provide the results in a web hosted manner.
 * Act upon the results by confirming any issues found, and filing appropriate bugs.
Line 55: Line 115:
This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.  * Need a named QA resource dedicated to this project
 * Need agreement on hardware in the Canonical lab where these tests can run
Line 57: Line 118:
== UDS Raw Notes ==

 * Use Autotest
  * Wide support with Google, Red Hat, IBM, etc.
 * Use Step Files for different distros.
 * Currently use automated scripts to test for certain end products.
 * Ubuntu community should build Step Files, and get them committed to Autotest.
 * Autotest currently doesn't have a large number of fucntion tests.
  * Ubuntu could greatly expand these tests.
 * Do libvirt testing as well as KVM testing.
  * Add libvirt tests to Autotest.
 * libvirt-sim does some tests with Autotests.
 * Need much more functional testing for KVM.
 * From a User perspective, should we be testing Windows VMs.
 * What are Ubuntu's requirements for testing.
 * Guests to Support:
  * Supported versions of Ubuntu Desktop and Server i386, amd64 (Hardy, Jaunty, etc).
   * Secondarily Kubuntu, Xubuntu, etc.
  * RHEL, Fedora
  * SUSE, OpenSUSE
  * Windows -- Functional not so much performance.
   * Windows 2003 Server
   * Windows 2003 Server
   * Windows XP
   * Windows 7
 * Tests should hopefully find regressions.
 * Automated test procedure:
  * Start out with a guest that has a bash shell on a serial port.
  * Scripts to interact with the shell.
  * Can put cygwin on Windows to provide a bash shell.
  * Build a collection of images that we care about testing.
  * Determine which functional tests are important.
  * Every test has a timeout.
 * Do performance testing by timing the entire process.
  * Should determine very quickly if there is a regression.
 * Integrate current hardware certification testing into Autotest framework.
  * The framework is Open Source.
  * Some tests are provided by specific hardware vendor, and there may be issues integrating with Autotest.
 * Guest certification -- What level of testing is appropriate for an Ubuntu guest running on another distro like Fedora?
  * Would be treated as a native hardware platform.
  * Currently VMware is treated in this fashion.
  * With upstream release kvm-XX.XX is supported on Ubuntu if it passes a standard set of tests.
 * Most of the work needs to be done regarding developing the tests, not so much on which framework to use.
 * Test Times:
  * Alpha, Beta
  * Test at the commit level.
 * Deliverables:
  * Owe upstream Step Files for supported releases (i386 and amd64).
   * Preseed data suppmitted in conjunction with Step Files.
 * Where to run tests:
  * Upstream and Canonical will run tests concurrently.
 * Test suite that user's can run to test their hardware.
 * How to test KVM plus the stack.
  * KVM with libvirt
  * libvirt with virt-manager
 * Determine better bug reporting process.
  * Reporting bugs upstream to KVM.
== Discussion ==

Summary

We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm and libvirt, and various configurations.

Release Note

Ubuntu 10.04's virtualization infrastrucure has undergone regular, automated testing.

Rationale

KVM is the Ubuntu's virtualization hypervisor, and Libvirt is the abstraction layer by which Eucalyptus communicates with KVM. These fundamental technologies are the foundation upon which we build the Ubuntu Enterprise Cloud.

We can, and should automate the testing of KVM and Libvirt on a regular basis.

User stories

  • As an administrator of a local installation of UEC, I wants to ensure the stability her users' instances, which run in KVM virtual machines. With Canonical/Ubuntu having regularly and heavily tested its virtualization stack, I can rest assured that my KVM/Libvirt virtual machines will be reliable and high-performance
  • As a developer of UEC, I wants to ensure that the KVM and Libvirt virtualization stack are stable and bug free. I can trust the virtualization foundation and focus my efforts on debugging UEC.
  • As a system administrator interested in migrating my infrastructure to virtualization-based solutions, I chose Ubuntu/KVM/Libvirt based on the extensive testing performed by Canonical and Ubuntu in the Lucid LTS cycle.

Assumptions

  • Canonical IS will provide some VT-capable hardware which can run these automated tests.
  • Canonical QA will instrument checkbox or some other testing framework to enhance the test coverage, automatically trigger these tests, review the results, and report bugs accordingly.
  • Canonical Server Team will respond and fix bugs found/filed by QA.

Design/Implementation

Details in the sections below.

How to Test

  • Autotest, KVM-Autotest

    • Wide support with Google, Red Hat, IBM, etc.
    • Use Step Files for different distros.
    • Currently use automated scripts to test for certain end products.
    • Ubuntu community should build Step Files, and get them committed to Autotest.
    • Autotest currently doesn't have a large number of fucntion tests.
      • Ubuntu could greatly expand these tests.

What to Test

  • Functionality, Performance, Stability, Regression
  • QEMU-KVM
    • Already tested by KVM-Autotest

    • We need to integrate these tests into Canonical's automated testing infrastructure (checkbox?)
    • Could also enhance the functional testing coverage
  • Libvirt
    • Do libvirt testing as well as KVM testing, as UEC users are affected by both of these
    • Add libvirt tests to Autotest.
    • libvirt-sim does some tests with Autotests.
    • Security team has some libvirt tests
  • Guest OSes
    • Linux
      • Ubuntu 8.04 LTS, 10.04 LTS, Desktop, Server
      • Debian 5.0
      • Fedora 12
      • CentOS 5
      • OpenSUSE 11.2
      • amd64 and i386
      • desktop and server
    • Windows
      • Windows 2003, XP, Vista, 7
    • Ensure that these guest OSes
      • Install
      • Boot
      • Have network connectivity
      • Sound (less important)
      • USB passthrough (less important)

Procedure

  • Build a collection of images that we care about testing
    • For Ubuntu, we could perhaps use daily UEC images (?)
    • If Windows is interesting, can use cygwin to get a bash shell
  • Guests attach a bash shell to the serial port
  • Test scripts interact with the shell
  • Run KVM-Autotest's functional tests
    • May need to enhance these (working with the upstream project) to ensure we cover the functionality that's important to us
      • e.g. dynamic block storage attach is something that we need for UEC, but it's not currently tested by upstream
    • Every test has a timeout
  • Do performance testing by timing certain processes (using existing checkbox performance tests)
    • Network throughput
    • Disk throughput
    • CPU intensive calculations
    • Memory intensive calculations
    • Track results over time to identify if there is a regression
  • Integrate KVM-Autotest framework into hardware certification testing to "certify" KVM as a support hardware platform for Ubuntu releases
    • Treat as a native hardware platform.
    • Currently VMware is treated in this fashion.

When to Test

  • Alpha1, Alpha2, Alpha3, Beta1, Beta2, RC
  • At each upload of qemu-kvm, libvirt
  • Perhaps one day link this and test against the daily-builds

Upstream Contribution

  • Should submit "Step Files" to upstream kvm-autotest for supported Ubuntu releases (i386 and amd64)
    • These are more or less "preseed" files
  • Additional upstream tests

Test/Demo Plan

  • Once the automation is in place, execute the automated tests whenever a new qemu-kvm or libvirt package is uploaded to the archive.
  • Provide the results in a web hosted manner.
  • Act upon the results by confirming any issues found, and filing appropriate bugs.

Unresolved issues

  • Need a named QA resource dedicated to this project
  • Need agreement on hardware in the Canonical lab where these tests can run

Discussion


CategorySpec

ServerKarmicAutomatedKvmTesting (last edited 2010-01-26 10:31:44 by 63)