Starting with Lucid, we've been doing daily, automated ISO testing of Ubuntu Server.

The following test cases are currently performed automatically:

The cartesian product of:

• Task selection:
  • Default Install
  • Bind9
  • LAMP
  • Mail server
  • MySQL
  • PostgreSQL
• Partitioning:
  • LVM
  • Plain, old partitioning
• Architectures
  • i386
  • amd64

All in all 24 tests, each taking around 10 minutes (I install from ISO's on disk to a ramdisk).

Implementation

Our automated ISO testing setup is based on the http://autotest.kernel.org Autotest framework, specifically the KVM subtest.

The KVM tests were designed to test KVM. It assumed a static set of OS's and ran a set of tests on them using the latest KVM from git to see if it had changed unexpectedly. We, however, assume a stable, functioning kvm, and use that to check whether our installs change unexpectedly.

Step files

The core of the test system is a set of step files. Here are the first few lines of one of our step files:

# --------------------------------
step 9.45
screendump 20100111_155511_313075295a7ce5b66a500a46c154ca70.ppm
# Choose "English" install
barrier_2 117 34 79 303 de7e18c10594ab288855a570dee7f159 47
# Sending keys: ret
key ret

The important parts are the "barrier_2" and "key" lines.

• The barrier_2 line tells the test system that a particular part of the screen (the rectangle starting at (79, 303) with size (117,34)) should look a specific way (the md5sum of the ppm data of the rectangle should be de7e18c10594ab288855a570dee7f159) within a specific timeframe (47 seconds) before moving on.
  • The "key" line tells the test system to send a keypress. In this case, "return".

Together, these two lines define this step as: Wait until the list of languages turns up and English is highlighted, and then press "return".

Creating step files

In the basic mode, step files are created by performing an install through a component known as "stepmaker". It allows you to pause the VM when there's something that needs to look a certain way (adding barriers) and respond with input (such as key presses or mouse movement and clicks). The output is a long series of steps like the above.

There are some caveats to this creation process. It works by running a VM and letting you pause it at relevant times (when there's something you want to match with a barrier, typically), accepts input from you, and then resumes the VM. If you were to mistype anything, you can't rewind (since you cannot rewind a VM) and forget that you mistyped. Instead you just have to keep going, well knowing that your mistyping has been saved for eternity, unless you want to start all over. Yours truly somehow managed to always mistype something at one of the very last steps, so I added an option to adjust existing step files. You can now configure the test runner to fall back to stepmaker if a test fails. So, to fix up your mistakes, you would open the step file and change something in there so that it would fail (usually the md5sum at the step where you want to start editing) and rerun the test. When it failed, it would show you a special version step maker which lets you load steps from the old step file and adjust them to the way the screen looks now that you are appropriately careful not to mistype.

Ok, so each test has its own step file..

Step file generator

..or at least they used to. It turns out that there was a lot of duplication in them. They all started out by choosing "English" in the beginning, then "Install Ubuntu Server", etc. Later on, they'd all give the same user info, and they'd all sit around waiting for the install to finish and tell the installer to reboot.. Then they'd all wait for the login prompt and log in. This is tedious, and if part of the UI changed, all step files needed to be changed.

To address this I wrote a "step file generator". It basically stitches together a step file based on the configured partitioning scheme and task selection. Creating step files is slightly more complicated now (you have to look at step files and consider how they differ and how they're the same and split them up accordingly), but it allows me to run all the different combinations of tests and thus get better test coverage.

Getting involved

The code lives on https://code.launchpad.net/~soren/autotest/automated-ubuntu-server-tests Launchpad.

To get started, you simply check out this branch:

        bzr branch lp:~soren/autotest/automated-ubuntu-server-tests

There interesting parts of that tree is client/tests/kvm. To not fiddle too hard with the upstream tests, this stuff ships with a separate list of tests in "kvm_tests_ubuntu.cfg" and a separate control file: "ubuntu-isotest-control".

To run it:

       cd <topDir>/automated-ubuntu-server-tests/client/tests/kvm
       sudo ./bin/autotest ubuntu-isotest-control