ServerKarmicAutomatedKvmTesting

Revision 1 as of 2009-06-03 18:35:09

Clear message

Summary

We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm.

Release Note

Ubuntu 9.10's KVM virtualization stack has undergone regular, automated testing.

Rationale

This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified.

User stories

Assumptions

Design

You can have subsections that better describe specific parts of the issue.

Implementation

This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

UI Changes

Should cover changes required to the UI, or specific UI that is required to implement this

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Migration

Include:

  • data migration, if any
  • redirects from old URLs to new ones, if any
  • how users will be pointed to the new way of doing things, if necessary.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

UDS Raw Notes

  • Use Autotest
    • Wide support with Google, Red Hat, IBM, etc.
  • Use Step Files for different distros.
  • Currently use automated scripts to test for certain end products.
  • Ubuntu community should build Step Files, and get them committed to Autotest.
  • Autotest currently doesn't have a large number of fucntion tests.
    • Ubuntu could greatly expand these tests.
  • Do libvirt testing as well as KVM testing.
    • Add libvirt tests to Autotest.
  • libvirt-sim does some tests with Autotests.
  • Need much more functional testing for KVM.
  • From a User perspective, should we be testing Windows VMs.
  • What are Ubuntu's requirements for testing.
  • Guests to Support:
    • Supported versions of Ubuntu Desktop and Server i386, amd64 (Hardy, Jaunty, etc).
      • Secondarily Kubuntu, Xubuntu, etc.
    • RHEL, Fedora
    • SUSE, OpenSUSE
    • Windows -- Functional not so much performance.
      • Windows 2003 Server
      • Windows 2003 Server
      • Windows XP
      • Windows 7
  • Tests should hopefully find regressions.
  • Automated test procedure:
    • Start out with a guest that has a bash shell on a serial port.
    • Scripts to interact with the shell.
    • Can put cygwin on Windows to provide a bash shell.
    • Build a collection of images that we care about testing.
    • Determine which functional tests are important.
    • Every test has a timeout.
  • Do performance testing by timing the entire process.
    • Should determine very quickly if there is a regression.
  • Integrate current hardware certification testing into Autotest framework.
    • The framework is Open Source.
    • Some tests are provided by specific hardware vendor, and there may be issues integrating with Autotest.
  • Guest certification -- What level of testing is appropriate for an Ubuntu guest running on another distro like Fedora?
    • Would be treated as a native hardware platform.
    • Currently VMware is treated in this fashion.
    • With upstream release kvm-XX.XX is supported on Ubuntu if it passes a standard set of tests.
  • Most of the work needs to be done regarding developing the tests, not so much on which framework to use.
  • Test Times:
    • Alpha, Beta
    • Test at the commit level.
  • Deliverables:
    • Owe upstream Step Files for supported releases (i386 and amd64).
      • Preseed data suppmitted in conjunction with Step Files.
  • Where to run tests:
    • Upstream and Canonical will run tests concurrently.
  • Test suite that user's can run to test their hardware.
  • How to test KVM plus the stack.
    • KVM with libvirt
    • libvirt with virt-manager
  • Determine better bug reporting process.
    • Reporting bugs upstream to KVM.

CategorySpec