CertificationTestSuite

Hardware Certification Testing of proposed kernels (SRU testing)

As part of the certification testing process, the Hardware Certification team runs a suite of automated tests on each system that has been certified for a stock Ubuntu release. This test suite has some requirements based on the fact that it must be run on a wide range of hardware in a short period of time (a few days).

  1. All tests must be automated. It is not feasible for an engineer to supervise even one test on over one hundred systems every few weeks.

  2. The test suite runtime must be bounded to no more than a few hours at most (at present less than 20 minutes). If problems occur then this time is precious.

There are currently 30 tests in the entire suite (3 more non-test Checkbox jobs are included in the tests displayed on the results page, and additonally the disk_read_performance test is run once for each drive, so the total can be more on any given system). The focus is on testing each hardware component adequately but not (at the moment) in depth, primarily due to the constraints listed above.

Bluetooth

bluetooth-detect: Check that Bluetooth radios can still be detected after the SRU is applied. Runs hcitool to show detected Bluetooth devices, failing if none are detected.

CPU Features

cpu_offlining: Check that individual cores of the CPU can be offlined. This is run on both clients and servers but mainly intended for servers. Each core in turn is offlined and /proc/interrupt queried to determine if the offlining was succesful.

cpu_topology: Check the topology stated in /proc/cpuinfo is reflected in /sys.

cpu_before/after_suspend: Checks the number of CPUs online after suspend and compares it with the number of CPUs online before suspend. The test fails if there is any variance.

cpu_scaling_test: Check that CPU frequency can be set. Uses the Firmware Test Suite to do this.

Disk

disk_read_performance: Check that disk read performance is at an acceptable level after applying the SRU. Uses hdparm to time buffered disk reads and checks that the figure measured in greater than 40MB per second on IDE disks and 15MB per second in everything else.

Graphics

compiz-check: Check that Compiz can be run on the system after the SRU is applied. This uses a lengthy script to check different criteria for compiz support (driver loaded, features supported by GFX card).

graphics-2d-output: Check that the Xorg server version can be detected after the SRU is applied. Uses xdpyinfo to get the version.

screenshot: Takes a screenshot of the currently running system using the import command from imagemagick. These screenshots are analysed in a batch after all systems are tested to try and find any that display graphical issues.

xorg_process: Check that the X server process is running after the SRU is applied. This uses pgrep to search the list of currently running process for the Xorg process.

xorg-failsafe: Checks that the X server is not running in failsafe mode after the SRU has been applied. his is based on whether /var/log/Xorg.failsafe.log exists.

monitor_connection: A test which uses the iperf tool to monitor the bandwidth achieved on a wireless link. If it doesn't meet a certain threshold the test will be failed.

screenshot: An update to the existing test to use an external source to capture the screen (e.g. a webcam or IP KVM).

screenshot_after_suspend: As above, but after the sleep state has been entered and resumed from.

screenshot_fullscreen_video: This time while displaying a video file fullscreen.

gpu_lockup: Perform some intensive operations and then check for GPU lockup.

Memory

info: Compares the reported memory total to the amount of memory installed as reported by DMI. If there is a significant mismatch then the test is reported as failed.

check: Checks that all reported memory can be addressed.

memory_before/after_suspend: Checks the amount of RAM available after suspend and compares it with the amount of RAM available before suspend. The test fails if there is any variance.

Networking

internet_test: Checks that the systems wired connection is working properly. Pings the default gateway to verify it's working correctly.

network_test: Checks that the systems network devices can be detected. Uses lspci to find the devices.

network-http: Checks that a web page can be retrieved using the systems wired connection after the SRU is applied. Uses wget to retrieve the homepage of cdimage.ubuntu.com.

ping: Identical to internet_test except it pings a specified system rather than the gateway.

Power Management

fwts_wake_alarm: Runs the wake-alarm test from the Firmware Test Suite to check that the BIOS wakealarm feature is working. This test is to help diagnose failures of the suspend test (below).

sleep_state_test: Checks that the system can go into the S3 power state (suspend). The test also tries to wake the system automatically by applying a wake alarm. If this fails than the lab engineer may need to wake the system by pressing the power button. If this is successful the test is not considered to have failed.

rtc: Checks that the systems real-time clock can be detected after the SRU is applied. A pre-requisite to running S3 testing.

Updating

apt-get-gets-updates: A simple test to ensure the system can still access the update system in case a bad SRU severely breaks it. This is implied by the networking tests but it was thought that making this explicit would be good.

USB

detect: Checks that the system can query the USB devices that are connected to it. Uses lsusb to display available devices.

storage_test: Tries to write a file to an installed USB device and verify that the operation was succesful.

Wireless

wireless_scanning: Uses iwlist to scan for available wireless access points in the systems vicinity. It fails if none are returned or iwlist returns an error.

wireless_connection_wpa_bg: Establishes a connection to a wireless access point configured with the 802.11b/g protocols and using WPA security. It then checks that the connection is working properly.

wireless_connection_open_bg: Establishes a connection to a wireless access point configured with the 802.11b/g protocols and using no security. It then checks that the connection is working properly.

wireless_connection_wpa_n: Establishes a connection to a wireless access point configured with the 802.11n protocol and using WPA security. It then checks that the connection is working properly.

wireless_connection_open_n: Establishes a connection to a wireless access point configured with the 802.11n protocol and using no security. It then checks that the connection is working properly.

wireless_connection_after_suspend: Run with the same four combinations of b/g/n protocols and WPA/no security described above, but after the system has been suspended.

Glossary

fwts: A test tool written for the purpose of testing functions of the systems BIOS.

hcitool: A program which can be used to perform operations with Bluetooth radios connected to the system, including querying available devices, scanning and creating connections

hdparm: A command line interface to various kernel interfaces supported by the libata subsystem. Includes a test mode which times the speed of reading through the buffer cache to the disk without any prior caching of data.

iperf: A performance testing tool for networking which uses a client/server setup to measure bandwidth on a particular link.

lspci: A utility for displaying information about PCI buses on a system and the devices connected to them. These usually include the systems graphics cards and wireless/ethernet cards.

lsusb: A utility for displaying information about the USB subsystem and the devices connected to it.

In the Pipeline

We are constantly trying to improve the coverage of the automated SRU testing. At the moment there are no tests in the pipeline. Watch this space for updates.

Feedback

This section contains some feedback on the content of the test suite that the Hardware Certification team plans on following up on in future.

  • The functioning of Bluetooth devices should be tested and not just whether they exist

Kernel/kernel-sru-workflow/CertificationTestSuite (last edited 2012-07-31 14:50:54 by brendan-donegan)