CheckboxExpandTestCoverage

Summary

Add further test coverage to checkbox, including Webcam, Audio and Printing.

Release Note

More system tests, both manual and automatic, have been added. This enables Ubuntu to be tested on a wider variety of hardware and focus development on hardware that does not work properly.

Rationale

In order to provide a high-quality product, we must test as much of it as possible on as much different hardware as possible.

User stories

  • Fred has some unusual hardware that is not supported under the latest beta release of Ubuntu. He would like to be able to test what capabilities are supported and file bugs against those that are not.
  • Certification testing will take advasntage of a richer pool of tests for more in-depth testing.

Design

Tests will be written or imported and added to test suites within Checkbox. These will in turn be distributed over several packages:

  • checkbox - basic tests installed by default on all systems (main)
  • checkbox-extras - extended test coverage for in-depth testing by the Ubuntu community (universe)
  • checkbox-compatibility - used by hardware partners to validate systems (PPA)
  • checkbox-certification - run in the certification labs (PPA)

In released versions of Ubuntu, Checkbox will default to a basic set of tests as only the base checkbox package is installed. Any user can install checkbox-extras for further test coverage. Hardware partners will also be provided with checkbox-compatibility.

The checkbox-compatibility and -certification packages will not depend on -extras. In fact they will be designed to conflict and there will be significant overlap in coverage. A test may first appear in the universe package where it is tested before graduating to the commercial packages while other cases may start private and later be released in -extras.

Test Categories

The following broad categories should have tests developed:

  • audio
  • video
  • networking
  • input devices
  • hibernate
  • suspend
  • other
    • miscellaneous hardware and software

Implementation

Additional tests

The tests below will be added to Checkbox as tests for Karmic. Tests marked as "manual" have at least some manual requirements, even if the test is partially automatic (e.g. confirmation of results or connection of an external device). Tests in the checkbox package will be shipped in the default Checkbox tests, while those in the checkbox-extras package will not be installed by default, but can be added by interested parties.

Test

Description

Manual/Automatic

Package

Implementation Details

Script Exists

Audio[1]

audio-loopback

Test basic audio playback/recording

automatic[2]

checkbox-extras

Implemented by Checkbox team; see Extend Audio Testing specification

audio-extended

Test additional audio hardware and jacks

manual

checkbox-extras

Implemented by OEM team and in the Extend Audio Testing specification

Video

compiz-check

Test for compiz functionality

automatic

checkbox

Implemented by OEM team

x-check

Test X server and modes

automatic(?) Info <!>

checkbox-extras

Info <!> Bryce?

display-external

Test external display (VGA/DVI)

manual

checkbox-extras

Implemented by OEM team

display-hdmi

Test HDMI video/audio output

manual

checkbox-extras

Implemented by OEM team

dvd-playback

Test DVD playback

manual

checkbox-extras

Implemented by OEM team

Webcam

camera-video

Test webcam video capture/playback

manual

checkbox

Implemented by OEM team

camera-still

Test webcam still image capture

manual

checkbox

Implemented by OEM team

camera

Camera detection

automatic

checkbox

Implemented by OEM team

Networking

wireless-bluetooth

Test wireless networking via Bluetooth

manual

checkbox-extras

Implemented by OEM team

wireless-wifi

Detect wifi hardware and test connectivity

automatic

checkbox

Implemented by OEM team

wireless-wifi-quality

Analyse wifi signal/noise quality

automatic

checkbox-extras

Implemented by OEM team

modem

Detect modem

automatic

checkbox

Implemented by OEM team

Suspend/Resume

suspend-lid

Test suspend/resume via lid switch

manual

checkbox

Implemented by OEM team

suspend

Test suspend/resume via software

automatic

checkbox

Implemented by OEM team

hibernate

Test hibernate/resume.

automatic

checkbox

Implemented by OEM team

suspend-stress

Stress-test suspend/resume

automatic/manual[3]

checkbox-extras

Existing kernel team script will be integrated by Checkbox team

fs-performance

Test filesystem performance

automatic

checkbox-extras

Implemented by OEM team

Input Devices

fingerprint

Test fingerprint reader

manual

checkbox-extras

Implemented by OEM team

keyboard-hotkeys

Test media keys/hotkeys

manual

checkbox

Implemented by OEM team

Other

ldtp

Linux Desktop Testing Project tests

automatic(?) Info <!>

checkbox-extras

Info <!>

security

Kernel security scripts

automatic

checkbox-extras

Info <!>

firewire

Test Firewire support

manual

checkbox-extras

Implemented by OEM team

cardreader

Test media (SD, CF, etc.) card readers

manual

checkbox

Implemented by OEM team

[1]: Basic audio playback and recording are already implemented as simple Checkbox tests and not represented in this chart. All audio tests here are specifically in-depth tests and extend what is already present.

[2]: This test is conducted automatically but requires hardware setup ahead of time. The default audio output will be directly connected to the default audio input for testing.

[3]: Portions of the test can be run automatically without requiring user input, which will be the default for automated tests. Extended tests require manual intervention to perform actions such as opening/closing the lid and removing the power cord. This full set of tests will be the default when Checkbox is run interactively.

Additional registry information

In addition to the expanded test coverage outlined above, Checkbox will retrieve the DSTD information from the system BIOS and return it as part of the hardware information. This information can be retrieved via a script that the kernel team has written. This script will be added to the Checkbox registry plugin by the Checkbox team.

Import ecurity team tests

Info <!> Which?, How?

Integration of OEM Tests

The OEM team has a number of tests already written or being written that will be integrated as a part of this process. These tests will be added to the checkbox or checkbox-extras packages as outlined above.

Automatic Tests

Wherever possible, tests should be automated. Automated tests that do not require an inordinately long time to run or are experimental/potentially damaging to the user's system can be included in the default checkbox package.

Manual Tests

Many extended tests will be written as manual tests. Since these user interaction they will not be added to the standard package but instead to -extras and PPAs.

Test/Demo Plan

Additional tests will be initially added to the development version of Checkbox and tested by members of the QA team and by interested community members who install the development release of Ubuntu. Feedback from these tests will be used in part to determine which suites of tests should be enabled by default and which should be in the extended selections.

Unresolved issues

Certain tests may be possible to automate but will be initially created as manual tests. Future specifications will deal specifically with automation of extended testing. Tests that were once 'extended' manual tests that have been automated may become part of the 'standard' suite of default Checkbox tests, as they will have little or no user impact.

BoF agenda and discussion

Make sure all useful bits from here have made it into the spec:

What can be added to Checkbox?

  • ability to grab DSTD (from BIOS) and disassemble it and check the code
    • apparently there are scripts to do this
    • it can tell us if the BIOS is fundamentally broken
      • it sounds like this would be better to include as information in the hardware submission
  • OEM team has a directive to add cases to checkbox
    • already added compiz compatibility, suspend/resume stress test, camera tests
  • audio test -- linking input and output, generating and testing for waveforms
  • webcam
  • printer
  • modem
  • 3g
  • hotkeys/media keys
  • video out
  • cardreaders
  • usb/firewire
  • tv tuners
  • gps
  • hibernate
  • suspend
  • power drain during suspend (long-term suspend test)


CategorySpec

QATeam/Specs/CheckboxExpandTestCoverage (last edited 2009-06-17 14:03:36 by fader)