Ubuntu Wiki

Preparations

What you need

• A spare computer or hard drive for test installs. (A virtual machine can be used but we would prefer to see more tests performed on a variety of physical hardware.)

• A moderate level of Linux experience so you are able to find the relevant log files and produce useful bug reports: ReportingBugs

• Some available time before a milestone release to perform tests and file reports.

Setting up

• If you do not already have one, set up a Launchpad account so you can report bugs.

• Login to the test tracker at iso.qa.ubuntu.com which lets you report your test results. The ISO tracker lets you login with your Ubuntu SSO account, but make sure you have logged in at least once to Launchpad using your ubuntu SSO account or else you may encounter issues.

• Select the ISO images you expect to test and look over the relevant test cases, making sure you have the test setup you need.

Subscribing

You can sign up for individual test cases on any ISO. This helps everyone to see which tests are being covered and sends you a notification email when a new build is available, provided you have opted in to email notification inside user settings.

To do this, enter the daily milestone for the iso you wish you subscribe to. On the testcases display page (provided you are logged in), you will see a subscribe button along the bottom. Use the checkboxes to check all of the testcases you are interested in and then press subscribe to subscribe to new builds for that iso. Please note that it will NOT alert you about new daily builds, only builds for milestones other than daily.

Please note that this feature is still in development and there is no visual indication after you press the button that you have been subscribed. You can verify your subscription worked by checking the report here: http://iso.qa.ubuntu.com/qatracker/reports/subscriptions and hovering over the testcase you subscribed to. Your launchpad name should appear in the list

Downloading images

Daily ISO images are found at cdimage.ubuntu.com. We advise that you download the images you expect to test ahead of time and use zsync or rsync to update them, as this will reduce the turn around time on testing. Zsync only downloads the parts of the image that have changed and typically takes about 10% of the full download time with reasonably fresh images. See instructions on using zsync. If your using a virtual machine to test, TestDrive makes this process extremely easy for you by using zsync to get your desired image, and then one of virtualbox, kvm, or qemu to run the virtual machine for you.

The images currently being tested are listed on iso.qa.ubuntu.com with a direct download link and the appropriate rsync command for your convenience (click for the download page). Updates to these builds will be announced on #ubuntu-devel, #ubuntu-testing and (optionally) by email.

Testing

In doing specific testing of the CD/DVD images it is important to focus on those aspects that are typically not used by those who simply run the latest unstable version on their system through daily updates (Such testing is of course also extremely valuable). Key points in image testing include image integrity (md5sums), Live CD and installer functionality.

Each ISO image has a set of test cases associated with it. Click on the tracker to see details of the test case and see Testing/InstallMethods for details on each install method.

To subscribe to a specific ISO and tests, select the flavour from applet menu then the architecture (32 bit, 64 bit) from the list. Then select the tests you wish to subscribe to and then hit subscribe. You will receive an email announcing there are tests available, provided you also opt-in to email notification on your profile.

Reporting results

The test results are tracked on the iso testing tracker website at iso.qa.ubuntu.com. Once you are registered, you can log in to report your test results. The website is divided into several sections, and results are reported based on the following criteria:

• Flavor of Ubuntu (e.g., Ubuntu, Kubuntu, Xubuntu, Edubuntu, etc.)
• Type of CD image (e.g., live or alternate install cd)
• Method of installation (e.g., manual partitioning, partition resizing, etc.)

Links to the different flavors of Ubuntu are present in left-hand menu.

Starting a test

ISO test cases normally last for at least half an hour. To avoid duplicating efforts, it is good to let others know that you have started a test case.

To mark a test as "Started" select the result "Started" and submit the result. It will show up in the list with a "clock" icon.

Adding test reports

When you have conducted a test you can report the results in the ISO tracker with a general comment and bug numbers if applicable.

You can mark a test as Passed or Failed. In the ISO test tracker we mark as failures those experiences that prevented us to do what we wanted to achieve in that test case. I.e. If we want to install, and the partition manager fails, that's a failure. If we do install (or can access to the Live environment, as in this case), the test didn't fail as such. We would mark that as success, but will link the non-critical bugs that we find.

When reporting a broken image it is important that you check the that md5sum corresponds to the ISO being tested to confirm that you do indeed have the correct ISO and that it is not corrupted. To check the md5sum of your image, open a terminal and execute:

md5sum imagename.iso

Where "imagename.iso" is the name of the iso image that you have downloaded and want to verify. The reference value can be found in a file named MD5SUMS in the ISO image download directory. If the md5sum is correct you may also want to run the CD self test on the install media you used.

Bugs

As with all testing it is important to file bugs. If you find a bug, please, search if it has already been reported, and if it hasn't, report it yourself. You should also refer to the bug filing/debugging guide for that specific package (if available) to make sure you are aware of known issues and have attached the relevant log files.

To mark a test as Failed it is mandatory to link the test report with at least one bug in Launchpad. The tracker needs you to type the Launchpad.net bug number. For example: 123456789