Adding a test to run under the CI Infrastructure

Guidelines

Here are some basic guidelines for creating a test that can run under CI:

• Ensure that the test is as self-contained as possible.
  • For tests running on phones, for example, it's best if it can run using phablet-test-run without any extra setup provided outside the test itself
• Tests should output results in subunit (preferred) or binary fail/pass
• Ensure that the tests run reliably without random failures as this makes it difficult to determine if failures are real.
• For tests that look at performance, the raw performance data should be part of the results stream, but baselines should be compared to also record a clear pass/fail for each testcase.

Requesting CI to add the test

As soon as you are ready for the CI team to start looking at adding the tests, open a bug at https://bugs.launchpad.net/ubuntu-ci-services-itself and provide the following information:

• Where the tests can be found
• Intended targets the test should run on (arch, vm/hardware, device types, etc)
• Any special hardware required
• Any special setup that has not yet been added to the setup of the test itself

Future Changes

• The CI team is investigating the use of dep8 testing as a broad solution to test setup and execution problem. This includes extensions to cover specific needs such as specifying a machine type.