CertificationTestSuite

Differences between revisions 2 and 3
Revision 2 as of 2011-09-19 13:17:05
Size: 4683
Editor: 188-223-22-218
Comment:
Revision 3 as of 2011-09-19 19:02:23
Size: 4879
Editor: 188-223-22-218
Comment:
Deletions are marked like this. Additions are marked like this.
Line 14: Line 14:
'''bluetooth-detect''': Check that applicable Bluetooth radios can still be detected after the SRU is applied. Runs hcitool to show detected Bluetooth devices, failing if none are detected. '''bluetooth-detect''': Check that Bluetooth radios can still be detected after the SRU is applied. Runs hcitool to show detected Bluetooth devices, failing if none are detected.
Line 69: Line 69:


== Glossary ==

'''hcitool''': A program which can be used to perform operations with Bluetooth radios connected to the system, including querying available devices, scanning and creating connections

Hardware Certification Testing of proposed kernels (SRU testing)

As part of the certification testing process, the Hardware Certification team runs a suite of automated tests on each system that has been certified for a stock Ubuntu release. This test suite has some requirements based on the fact that it must be run on a wide range of hardware in a short period of time (a few days).

  1. All tests must be automated. It is not feasible for an engineer to supervise even one test on over one hundred systems every few weeks.

  2. The test suite runtime must be bounded to no more than a few hours at most (at present less than 20 minutes). If problems occur then this time is precious.

There are currently 21 tests in the entire suite (3 more non-test Checkbox jobs are included in the tests displayed on the results page, and additonally the disk_read_performance test is run once for each drive, so the total can be more on any given system). The focus is on testing each hardware component adequately but not (at the moment) in depth, primarily due to the constraints listed above.

Bluetooth

bluetooth-detect: Check that Bluetooth radios can still be detected after the SRU is applied. Runs hcitool to show detected Bluetooth devices, failing if none are detected.

CPU Features

cpu_offlining: Check that individual cores of the CPU can be offlined. This is run on both clients and servers but mainly intended for servers. Each core in turn is offlined and /proc/interrupt queried to determine if the offlining was succesful.

cpu_topology: Check the topology stated in /proc/cpuinfo is reflected in /sys. <RATIONALE>

fwts_cpu_scaling_test: Check that CPU frequency can be set. Uses the Firmware Test Suite to do this.

Disk

disk_read_performance: Check that disk read performance is at an acceptable level after applying the SRU. Uses hdparm to time buffered disk reads and compares it to an acceptable baseline.

Graphics

compiz-check: Check that Compiz can be run on the system after the SRU is applied. This uses a lengthy script to check different criteria for compiz support (driver loaded, features supported by GFX card).

graphics-2d-output: Check that the XOrg server version can be detected after the SRU is applied. Uses xdpyinfo to get the version.

screenshot: Takes a screenshot of the currently running system using the import command from imagemagick. These screenshots are analysed in a batch after all systems are tested to try and find any that display graphical issues.

xorg_process: Check that the X server process is running after the SRU is applied. This is a simple pgrep.

xorg-failsafe: Checks that the X server is not running in failsafe mode after the SRU has been applied. his is based on whether /var/log/Xorg.failsafe.log exists.

Memory

check: Checks that all reported memory can be addressed.

Networking

internet_test: Checks that the systems wired connection is working properly. Pings the default gateway to verify it's working correctly.

network_test: Checks that the systems network devices can be detected. Uses lspci to find the devices.

network-http: Checks that a web page can be retrieved using the systems wired connection after the SRU is applied. Uses wget to retrieve the homepage of cdimage.ubuntu.com.

ping: Identical to internet_test except it pings a specified system rather than the gateway.

Power Management

fwts_wake_alarm: Runs the wake-alarm test from the Firmware Test Suite to check that the BIOS wakealarm feature is working. This test is to help diagnose failures of the suspend test (below).

sleep_state_test: Checks that the system can go into the S3 power state (suspend). The test also tries to wake the system automatically by applying a wake alarm. If this fails than the lab engineer may need to wake the system by pressing the power button. If this is successful the test is not considered to have failed.

rtc: Checks that the systems real-time clock can be detected after the SRU is applied. A pre-requisite to running S3 testing.

Updating

apt-get-gets-updates: A simple test to ensure the system can still access the update system in case a bad SRU severely breaks it. This is implied by the networking tests but it was thought that making this explicit would be good.

USB

detect: Checks that the system can query the USB devices that are connected to it. Uses lsusb to display available devices.

Glossary

hcitool: A program which can be used to perform operations with Bluetooth radios connected to the system, including querying available devices, scanning and creating connections

Kernel/kernel-sru-workflow/CertificationTestSuite (last edited 2012-07-31 14:50:54 by 94-192-232-102)