Ubuntu Wiki

Goal

To be able to test the audio expierience for any particular Ubuntu release.

Reference Platform

It is key that we be able to reproduce our tests as we make changes to the system software so that we can tell if we are making improvements or introducing regressions. To that end, the first place to start is with the platform we will be testing on.

For the Lucid development cycle we will be using the Dell Mini 10v as the reference platform. This is the reference platform for Lucid. This is the platform we are using for all our boot-speed comparisons as well as power measurements. Another good reason to use this platform for reference is it's rather underpowered processor, the 133Mhz Atom.

Test Scenarios (Use Cases)

The idea here is to pick several scenarios that normal users are likely to perform and examine the audio experience in each scenario.

• Reproduceable load on the system.
• How do we measure the "audio experience".
• What are the metrics we are looking at to determine if an audio experiece is good or bad?
• Using internal and external speakers and microphones. Bluetooth and USB devices.
• Do we see large delay's in audio between when a speaker speaks and the speakers voice is heard.
• We've seen issues with USB headsets and delay due to pulseaudio's glitch free mode. In this case we "fixed" the problem by adding "tsched=0" to the end of "load-module module-udev-detect" in the file: /etc/pulse/default.pa. After making the change the system should be rebooted.

Note: In the following test cases we talk of using 'stress' to put a load onto the system. After some experimenting with stress we will come up specific 'stress' uses to get consistent system loads while testing.

Use Case: Skype Call Test (Non-stressed)

1. Start up Skype
2. Make test calls to verify there are no issues with the audio output or with the microphone.
3. Try the test with both internal and external speakers/headphones/microphones
4. Try the test with both usb and bluetooth headphones and microphones

Use Case: Skype Call Test (Stressed)

1. Start up Skype
2. Startup 'stress' the workload simulator for linux.
3. Make test calls to verify there are no issues with the audio output or with the microphone.
4. Try the test with both internal and external speakers/headphones/microphones
5. Try the test with both usb and bluetooth headphones and microphones
6. Repeat this test while using different command line parameters for stress, examine the effects of changing CPU-bound processes, I/O-bound processes and memory allocator processes.

Use Case: VLC playback of a video stream / movie.

1. Start up VLC player
2. Begin playing a movie.
3. Listen for sound glitches, pauses, pops/cracks that should not be there.
4. Try the test with both internal and external speakers/headphones
5. Try the test with both usb and bluetooth headphones

Use Case: Game Sound Test (Non-stressed)

1. Start up torcs racing game in a window.
2. Play the game a bit to see if there are any audio issues before trying to "stress" the system.
3. Try the test with both internal and external speakers/headphones
4. Try the test with both usb and bluetooth headphones

Use Case: Game Sound Test (Stressed)

1. Start up torcs racing game in a window.
2. Startup 'stress' the workload simulator for linux.
3. Play the game while noting any sound issues.
4. Try the test with both internal and external speakers/headphones
5. Try the test with both usb and bluetooth headphones
6. Repeat this test while using different command line parameters for stress, examine the effects of changing CPU-bound processes, I/O-bound processes and memory allocator processes.

Use Case: Music Player Test (Non-stressed)

1. Start up rhythmbox music player.
2. Play a music file that is well known (i.e. one that glitches can be noticed if they occur). Recomend against anything with rapidly changing chords.
3. Listen for sound glitches, pauses, pops/cracks that should not be there.
4. Startup 'stress' the workload simulator for linux. Using different command line parameters for stress, examine the effects of changing CPU-bound processes, I/O-bound processes and memory allocator processes.
5. Try the test with both internal and external speakers/headphones
6. Try the test with both usb and bluetooth headphones

Tools

• stress http://weather.ou.edu/~apw/projects/stress

Notes

• From cking:
  • step 0: classify the kind of issues, which ones are priorities
  • step 1: how to gather data on these
  • step 3: how to keep it tested to check for regressions
• Perform a survey of the last 'n' months of bugs to determine areas that should be examined and tested.