ExtendedAudioTesting

Summary

Extend both manual and automated audio test cases to test a range of use cases and hardware configurations on a regular basis.

Release Note

Extended audio testing is available. Certain tests are conduced automatically by the System Testing application, but extended tests are available for users who want to help ensure that a wide variety of audio hardware works correctly with Ubuntu.

Rationale

Functioning audio is crucial to the Ubuntu desktop experience. Users should be able to expect sound to work properly from install onward. Testing audio in a variety of configurations across a range of hardware will help us to ensure that our sound support meets user expectations.

User stories

  • George wants to assist with the Ubuntu development effort but has no development or testing experience. He runs Checkbox and performs some simple audio tests. He is pleased that his efforts will help indicate which hardware needs further development efforts in Ubuntu.
  • Harriet has a wide array of audio hardware. Some of this hardware is not properly supported by Ubuntu. She is able to run Checkbox in an extended mode and report bugs on nonfunctioning hardware.
  • Ronald runs multiple automated tests on a wide variety of systems on a daily basis. He wants to be able to tell when a system is having audio problems and file appropriate bugs. He can run Checkbox in an automated fashion across these machines and see which ones have simple audio issues.

Assumptions

  • Automated testing:
    • Hardware can be appropriately configured to perform loopback tests.
    • External sound sources are available for testing internal microphones.
  • Manual testing:
    • A wide array of hardware exists and can be tested by the user community.

Design

Automated Testing

Tested systems will have hardware configuration changes made to enable automated testing. This will consist of simply connecting the default audio output to the default audio input with a patch cable. This will allow automated tests to produce a sound on the audio output and listen for it on the input. Automated Checkbox tests will be created relying on this configuration to test audio.

Manual Testing

A series of manual Checkbox tests will be created to test audio and added to checkbox-extras. The test dependencies will be designed as to reduce redundant tests.

Testing will start at the top of the audio stack -- if the most complex audio test cases pass, then it can be assumed that lower-level parts of the audio stack are functioning properly. Users will also be asked what hardware they have available and will be prompted to test only the hardware that they indicate they have. This is indicated below by the "Skip if Successful" column; a test can be skipped if the test in this column has previously passed successfully.

All test descriptions and instructions should be written in clear user-focused language and should make it clear that the user may skip any test for which they lack the hardware or interest. (E.g. if the user does not own USB audio hardware, they should skip any USB audio tests.)

Implementation

Automated Testing

A Checkbox test or suite will be written to generate a sound on the audio output and listen for it on the input. This MUST generate a pass/fail value as follows:

  • Silence is detected, as indicated by a recorded audio file of sufficiently small size: FAIL

  • Audio is detected, as indicated by a recorded audio file of sufficiently large size: PASS

  • No audio hardware is detected in the list of hardware:
    • This MAY generate a SKIP value in normal system audio testing, as there is currently no way to indicate expected hardware existence on a per-machine basis.

    • This SHOULD generate a FAIL value on hardware known to have audio hardware installed. (E.g. OEM hardware.) Ideally Checkbox should be able to be made aware of expected hardware detection, but this currently can be covered via Extended Manual Testing as described below.

The size threshold of the file indicating silence or recorded audio will be determined during implementation.

For the purposes of this specification, it will be sufficient to determine if audio or silence is detected by the Checkbox test. In the future, more advanced analysis of received audio may be performed to test for the correctness of the generated/received audio.

Manual Testing

Default Tests

The current Checkbox test that simply produces a sound and asks if the user heard it will remain in place. This will remain the default audio test presented to users in a standard Checkbox test.

Extended Tests

An additional suite of Checkbox tests will be written to test audio and added to the checkbox-extras package.

These tests MUST present the user with a list of detected audio hardware and ask the user to confirm their existence. The tests SHOULD ask the user if any audio hardware exists in the system that was not automatically detected. The tests SHOULD also prompt the user to select (via check boxes or similar mechanism) what hardware they possess that cannot be automatically detected, such as microphones, headphones, non-connected USB audio devices, etc.

For each detected piece of hardware, the suite MUST execute a series of tests. These tests SHOULD start at the 'top' of the audio stack; i.e. tests should start with the most complex case. If the initial case succeeds, it can be assumed that all lower level tests would also succeed, and thus they SHOULD not be presented to the user.

As an example, in order to test the internal microphone and speakers of a system, Checkbox might ask the user to speak into the microphone and then play back the recorded audio. If the user indicates that he or she heard the proper recorded audio, it can be assumed that both the internal microphone and speakers are working properly and no more tests for these devices need to be executed. If, however, the user indicates that the recorded audio was improper or not heard, Checkbox might begin by testing the speakers and then testing various aspects of the microphone.

Finally, for each selected piece of hardware that cannot be automatically detected, Checkbox SHOULD perform a series of tests. These tests MAY vary depending on the hardware and SHOULD strive to test as much of the capabilities of the hardware as possible. These tests also SHOULD start at the top of the relevant parts of audio stack as described above.

The following tests will be implemented:

Test

Skip if Successful

Description

record-internal

record a sound through the internal or default microphone using gstreamer and play it back through the default speakers

record-external

record a sound through an external microphone using gstreamer and play it back through the built-in speakers

record-usb

record a sound through an external USB sound capture device using gstreamer and play it back through the default audio output device

playback-internal

record-internal

play a sound through the built-in speakers using gstreamer

playback-external

record-external

ask the user to plug in speakers or headphones (if available) and play back a sound through them using gstreamer

playback-usb

play a sound through an attached USB sound output device

record-pulseaudio

record-internal OR record-external

record a sound through the internal or default microphone using pulseaudio and play it back through the default speakers

playback-pulseaudio

playback-external OR playback-internal

play a sound through the default audio output using pulseaudio

record-alsa

record-external OR record-internal OR record-pulseaudio

record through the default audio input device using alsa

playback-alsa

playback-external OR playback-internal OR playback-pulseaudio

play a sound through the default audio output device using alsa

audio-device-detection

present the user with a list of all detected audio devices

record-alternate-inputs

for each input on the detected sound devices, ask the user to connect a microphone; then record from it and play back the sound on the default audio output

playback-alternate-outputs

for each output on the detected sound devices, ask the user to connect speakers or headphones; play a sound through the connected output device

record-bluetooth

prompt the user to connect a bluetooth recording device; capture a sound a play it back on the default audio output

playback-bluetooth

prompt the user to connect a bluetooth playback device; play a sound on this device

Test/Demo Plan

Testing will be performed by the Checkbox team executing the Checkbox test scripts during development. After this the automated tests will be rolled out to the certification testing environment. As this is a controlled environment with tests run and analysed daily, this will be a sufficient test for including these tests in the next release of Checkbox.

Future work

The proposed automated audio testing is a large leap forward from its current state but still fairly basic. Future specifications should develop more in-depth testing procedures.

In the future additional hardware configuration changes will be considered. One possibility is producing a detectable sound in the environment containing the tested systems which can be picked up by internal microphones.

Unresolved issues

  • Note: The automated loopback test is designed to FAIL if 'silence is recorded' (a sufficiently small file). This will happen if recording (mic) fails but if playback fails with the mic working there is likely to be enough real-world noise to produce a false PASS result. Further research is required to prevent this issue and may require signal analysis.


CategorySpec

QATeam/Specs/ExtendedAudioTesting (last edited 2009-06-12 13:24:13 by cpc4-oxfd8-0-0-cust39)