OBSOLETE!

This page is obsolete, and only being kept for historical reasons (linked to Blueprint). Please see http://packaging.ubuntu.com/html/auto-pkg-test.html for an intro about Ubuntu CI autopkgtests, and the autopkgtest packages' /usr/share/doc/autopkgtest/ for detailled documentation.

-- pitti 2014-10-28 16:41:31

Original Page

Contents

OBSOLETE!
Original Page
Introduction
Testing infrastructure
1. Pre-requistes
2. Configuration Files
Testing environment setup
1. Running the test
2. Trigerring the test
Integration with Jenkins
1. Lab Configuration
Adding support to a package with an existing testsuite

Introduction

The tool/infrasctructure described in this document is part of the deliverables of UDS-P blueprint: https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-intermediary

This tool determines what tests need to be run via depends, setup a clean testing environment, drives said tests, and outputs in a format that another tool can consumes and a human can understand

The tool is made of the following components

prepare-testbed: Create and setup a clean testing VM using a cloud image of the latest development release from http://cloud-images.ubuntu.com/
run-adt-test: Drives run of autopkgtest in the testbed from the host.
trigger-adt-test: Check periodically the changes in the archive and trigger the tests when newest version of a package or its dependencies are available.

The project is available from launchpad at https://launchpad.net/auto-package-testing

DEP8 documentation:

Testing infrastructure

Pre-requistes

The server hosting this project requires the following features:

kvm enabled - the testing framework using hardware virtualisation features; these need to be enabled (normally done in the BIOS setup screens and be done by anyone via the kvm console)
http access to cloud-images.ubuntu.com
http access to archive.ubuntu.com and external resources required by the tests

Configuration Files

There are 2 configuration files:

config: General setup. This file can be located in ~/.adtrc. It defines various locations and default settings.
credentials: Credentials file for jenkins. This file can be located anywhere and passed as argument to trigger-adt-test. It contains credentials to access jenkins.

Testing environment setup

The testing environment is the VM created by the script prepare-testbed

$ ./prepare-testbed --help
Usage: prepare-testbed [OPTIONS] <arch>
Build a base testbed from the latest dev release image

  arch      Target architecture. i386 or amd64

Options:
  -k,--sshkey     path to private key used to ssh to the VM. Public key is
                  copied to the VM and must be located in the same directory
                  than the private key and a .pub extension.
  -r,--release    release (default:quantal)
  -S,--disksize   disksize of the VM
  -d,--debug      enable debug mode

The script downloads the latest cloud image for the dev release and uses cloud-init to initialize it with the required components to run autopkgtest. It prepares the VM to run autopkgtest and creates a link to the most recent testing environment available.

This script is called when a new image is available from cloud-images.ubuntu.com i.e when the URL http://cloud-images.ubuntu.com/quantal/current/MD5SUMS changes. It takes approximately 20 minutes to creates fresh amd64 and i386 testing environments.

Once the environment is setup, pristine VM are created in ${DISKDIR}/pristine-${RELEASE}-${ARCH}.img

Example:

$ ./prepare-testbed amd64

Running the test

The script responsible for running the test is run-adt-test.

$ ./run-adt-test -h
Usage: run-adt-test [OPTIONS...] <PKGNAME>
Run auto-package-test for a speficied package on a base VM

  -h,--help    This help
  -a,--arch    Architecture (Default:amd64)
  -r,--release Release (Default:quantal)
  -k,--keep    Don't shutdown and remove testbed after test
  -s,--use-shm Write snapshot files to /dev/shm
  -S,--source  Run from source instead of dsc
  -d,--debug   Enable debug mode
  -P,--proposed 
               Enable -proposed in testbed
  -p,--ppa     Include a PPA to search packages

This script creates a new VM using the pristine VM as backing store, update it to the latest version of the packages, download the source package of the package to be tested and run auto-pkg-test for this package. The results are copied to the local server and the environment is destroyed.

For debugging purpose, you can pass the argument '-k' and the testing environment will be preserved after the run so you can login and manually debug. You'll have to manually clean-up the file when you're done debugging. The snapshot file can be created in memory to speed up the run with '-s'.

The result files are all the files generated by autopkgtest, a summary file, and a json file that saves the status of the run. The structure of the json file is:

{ 
    'status':None,   # Status of the run PASS,FAIL,RUNNING
    'release':None,  # Release being tested
    'arch':None,     # Architecture
    'package':None,  # Name of the package
    'version':None,  # Version of the package
    'depends':{},    # List of the dependencies package:version
}

Example:

$ ./run-adt-test -a i386 apport

Trigerring the test

The tests are triggered by the script trigger-adt-test. This script periodically updates its package cache and compares the status of the packages with a dep8 control file to their version and dependencies in the archive. A run is triggered if a new version of the package has been uploaded or if a new dependency or a new version of a dependency is available.

Currently, this script triggers a remote jenkins job.

Packages with a header "XS-Testsuite: autopkgtest" are auto-discovered and added to jenkins automatically.

Integration with Jenkins

Tests are run through jenkins and results are reported to there. There is one matrix job per source package. This job triggers a test for 2 architectures i386 and amd64.

The jenkins job is started remotely by the trigger script.

Jenkins jobs for Quantal: https://jenkins.qa.ubuntu.com/view/Quantal/view/AutoPkg%20Test/

Lab Configuration

In the lab the tests run on albali as user 'usit' and scripts are located in '$HOME/auto-package-testing'

Adding support to a package with an existing testsuite

Prepare a fresh testbed:

$ bin/prepare-testbed amd64

Start a run for the package you want to add support to. It will fail because there is no dep8 control file, but will setup the environment for this package:

$ ./bin/run-adt-test -a amd64 -Sdk aptdaemon

SSH to your testbed. You can use the command line displayed at the ends of the invocation of run-adt-test

$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -i /tmp/adt/disks/adtkey -p 54322 -l ubuntu localhost

cd to aptdaemon*/debian/ and create a tests directory
in the test directory create a control file with the following content:

Tests: run-test
Depends: @, python3-nose

Create the file debian/tests/run-test with the following content, and make it executable:

set -e

# clean up old crash reports
rm -rf /var/crash/*

# succeeding test must not write anything to stderr, as per DEP-8
# work around LP #972324
env -u TMPDIR  python3 setup.py test 2>&1

Run the test:

$ export ADTLOG_PATH=$HOME/adt-log
$ export ADTOPTS="-d --paths-host --summary $ADTLOG_PATH/summary.log --tmp-dir=$ADTLOG_PATH/ --leave-lang"
$ sudo adt-run -d --paths-host --summary $ADTLOG_PATH/summary.log --tmp-dir=$ADTLOG_PATH/ --leave-lang --unbuilt-tree . --- adt-virt-null

This command will build the package, install the test dependencies and run the test script run-test. Check the everything is right after the run, add any missing test dependencies to the test control file and run again until the test passes.

Add a XS-Testsuite header to the source record for autodiscovery. Edit the file debian/control and add the entry "XS-Testsuite: autopkgtest" to the source record. It should look like:

Source: aptdaemon
Section: admin
Priority: extra
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
XSBC-Original-Maintainer: Julian Andres Klode <jak@debian.org>
Build-Depends: debhelper (>= 7.3),
               debconf-i18n,
               dbus,
               lintian,
               dh-translations,
               python-all,
               python-setuptools,
               python-distutils-extra,
               python-nose,
               python3-all,
               python3-setuptools,
               python3-distutils-extra,
               python3-nose,
               gir1.2-glib-2.0,
               gir1.2-packagekitglib-1.0,
               python3-apt (>= 0.8.5~ubuntu1),
               python3-defer (>= 1.0.6),
               python3-dbus,
               python3-gi,
               python3-mock,
               python3-pkg-resources,
               python-apt (>= 0.8.5~ubuntu1),
               python-defer (>= 1.0.6),
               python-dbus,
               python-gi,
               python-mock,
               python-pkg-resources,
               iso-codes,
Standards-Version: 3.9.1
Homepage: https://launchpad.net/aptdaemon
Vcs-Bzr: lp:~aptdaemon-developers/aptdaemon/ubuntu-quantal
Vcs-Browser: https://code.launchpad.net/~aptdaemon-developers/aptdaemon/ubuntu-quantal
XS-Original-Vcs-Bzr: nosmart+http://bzr.debian.org/bzr/apt/aptdaemon/debian-sid
XS-Original-Vcs-Browser: http://bzr.debian.org/loggerhead/apt/aptdaemon/debian-sid
X-Python-Version: >= 2.7
X-Python3-Version: >= 3.2
XS-Testsuite: autopkgtest

And you're all set

QATeam/AutomatedTesting/AutoPackageTesting (last edited 2014-10-28 16:41:31 by pitti)