• Launchpad Entry: qa-lucid-cloud-test

• Created: 2009-11-25

• Contributors: Scott Moser

• Packages affected: ec2-init euca2ools (tested)

Summary

In the Karmic cycle, we began producing ubuntu-server machine images that would run on ec2 or on UEC. The images would be tested running inside UEC and inside EC2. However, the test cases (here and here) were manual.

In the Lucid cycle, we would like to have automated testing of the daily build images, including features added to them in ServerLucidCloudConfig and ServerLucidCloudBoothooks.

Release Note

The Ubuntu Server cloud images are now tested in an automated fashion. This has helped to produce higher quality images, and catch regressions earlier.

Rationale

The testing done for our Ubuntu server cloud images in Karmic was all manual. This included all the shortcomings of manual test. The tests are expensive in terms of human resources to run, generally had poor coverage, and missed failures were common.

Automated test will provide us the ability to more thoroughly test the images and also to catch errors or regressions early. When running these tests on UEC, they will also serve to test the UEC installation. Additionally, if the test infrastructure is written to use the euca2ools those tools will also be being tested.

User stories

• As an Ubuntu developer I have added function to ec2-init and want to ensure that that function works as expected and doesn't cause any regressions.
• As a user of the Ubuntu Cloud images I want the images to be bug free.

Assumptions

• All test will inherently rely on the following, as such, these key features must be functional to further test:

  • The ability to correctly pass user-data from command line tools to the cloud (ie, 461156 must be fixed)

  • The ability for the image to consume that user data (this is how the tests will be started)

Design

Overall, the intent is for these tests to be done in a manner which is most consistent with other automated tests run by the QA team. In the UDS Session, we came up with a list (see 'BoF agenda and discussion') of the things that we would like to test. Additionally it seemed like the most consistent way to test these was to consider each region (us-east-1 and us-west-1) and instance-type to be a "machine". Then, the test suite would be run on each of these "machines" and test results recorded.

Implementation

• There will be one system from which all tests are launched.
• daily build images will be published to ec2 and to UEC. The test launching system will poll availability of images and launch tests based on new images found. It is my understanding/expectation that the same poll-ing mechanism is used for testing ISO images.
• Each combination of region and instance-type is considered a "machine" upon which tests should be run
  • i386: { us-east-1, eu-west-1 } / { m1.small, c1.medium }
  • amd64: { us-east-1, eu-west-1 } / { m1.large, m1.xlarge, c1.xlarge }
• The test infrastructure will launch a new instance of a given type. It will pass a script that will run on first boot of the instance. This script will cover:
  • ensuring dependencies are installed
  • obtaining tests (maybe in from bzr)
  • running tests
  • publishing results
• Tests will require the test-launching system to look at euca-console-output and verify things (such as ssh-fingerprints)

Questions

• How do we test different ServerLucidCloudConfig ? This seems to break the idea of instance-type/region as being a machine. Maybe it can be considered to be different install configurations on the same hardware. How are different install options tested in "normal ISO testing"?

• Where will test results be published to?

Migration

The ISO tracker tests and wiki tests should point to automated daily build test results for the given images. We probably will want to still do some human sniff testing of these images, but generally will rely on the automated tests.

Risks

The resources for UEC testing are not in place in time

Category: IS resources
Probability: 60%
Risk threat level: High

Mitigation plan

Mitigation Strategy:
Use two spare machines in the hardware certification labs to set up this testing.

Contingency:

1. Scott Moser: identify minimal requirements of the UEC for testing
2. Marc Tardif: identify two spares machines that matches the requirements
3. Marc Tardif: Give access to Scott and Ara to those machines.
4. Scott Moser: Set up a cloud for testing in the machines offered by the HW certification team.
5. Ubuntu QA Server: Use the specified cloud for the testing

The Ubuntu QA Server position is not filled in time to address all the blueprints in Lucid

Category: Human resources
Probability: 80%
Risk threat level: High

Mitigation plan

Mitigation Strategy:
Use an already available QA resource, working together with Scott Moser, to achieve as many work items as possible

Contingency:

1. Scott Moser: identify minimal requirements of the UEC for testing
2. Ubuntu QA Manager: decide which work items are going to be addressed and postpone the rest of them
3. Ara Pulido: work closely with Scott to address some of the most important work items

Test/Demo Plan

This spec generally covers testing of other features.

Unresolved issues

None.

BoF agenda and discussion

ISO tracker tests for karmic UEC images

• http://testcases.qa.ubuntu.com/System/EC2CloudImages

• https://wiki.ubuntu.com/Testing/EC2/TestCases

UEC

• where can we test images on a UEC
  • canonical data center
  • start publishing our daily builds to data center

EC2

• run tests on EC2
  • in each region
  • in each availability zone?
  • in each instance type?

What do we want to test

• initital boot and network connectivity
• can be reached by ssh
• fingerprints in console log (e.g., ec2-get-console-output)
• fingerprints on ssh
• fingerprints in syslog
• mount points are correct
• /etc/apt/sources.list are correct
• $LANG is correct
• timezone is correct
• kernel is correct
• correct modules are loaded
• ubuntu user exists
• root account locked
• ubuntu user can sudo without password
• ssh keys are correct in /root/.ssh/authorized_keys and /home/ubuntu/.ssh/authorized_keys
• root user gets the message that they can't ssh in
• possibly test ec2-currency
• landscape presense and function : ACTION: Ask landscape people
• boot hooks
  • need to test boot hooks are generally functional
• config data
  • need to test each of the config options that we support
• user-data
  • is ignored if it doesn't start with #!
  • is run on first boot if it does start with #!
  • is not run on reboot
  • sends output to console log for debugging
• post build tests
  • have post-vmbuild tests run (work on an image)
• rebundle the running instance and then run all of the above tests on the new image
• Make sure that all test instances are shut down
  • even if they failed at some point in the testing process
  • otherwise it's costing real money

TODO

• [QA] write framework for automated tests (use checkbox?)
• [QA] write tests to run (testing the config and ec2-init)
• [QA] get satellite machine up
  • smoser will help with everything

• Marc Tardif as point of contact, Ronald McCollam

---

CategorySpec