Procedures

Differences between revisions 27 and 28
Revision 27 as of 2012-05-29 20:04:06
Size: 6280
Editor: cblmdm72-240-218-227
Comment:
Revision 28 as of 2012-05-29 21:03:10
Size: 6861
Editor: cblmdm72-240-218-227
Comment:
Deletions are marked like this. Additions are marked like this.
Line 16: Line 16:
 * If you do not already have one, set up a [[https://launchpad.net|Launchpad]] account so you can report bugs. If you have signed up to edit the Ubuntu wiki you will already have a Launchpad account.
 * Register with the test tracker at [[http://iso.qa.ubuntu.com/|iso.qa.ubuntu.com]] which lets you report your test results. Even if you have registered for Brainstorm you will need to register for the tracker also.
 * If you do not already have one, set up a [[https://launchpad.net|Launchpad]] account so you can report bugs.
 * Login to the test tracker at [[http://iso.qa.ubuntu.com/|iso.qa.ubuntu.com]] which lets you report your test results. The ISO tracker lets you login with your Ubuntu SSO account, but make sure you have logged in at least once to [[https://launchpad.net|Launchpad]] using your ubuntu SSO account or else you may encounter issues.
Line 28: Line 28:
Daily ISO images are found at [[http://cdimage.ubuntu.com|cdimage.ubuntu.com]]. We advise that you download the images you expect to test ahead of time and use zsync or rsync to update them, as this will reduce the turn around time on testing. Zsync only downloads the parts of the image that have changed and typically takes about 10% of the full download time with reasonably fresh images. See [[https://help.ubuntu.com/community/ZsyncCdImage|instructions on using zsync]]. Daily ISO images are found at [[http://cdimage.ubuntu.com|cdimage.ubuntu.com]]. We advise that you download the images you expect to test ahead of time and use zsync or rsync to update them, as this will reduce the turn around time on testing. Zsync only downloads the parts of the image that have changed and typically takes about 10% of the full download time with reasonably fresh images. See [[https://help.ubuntu.com/community/ZsyncCdImage|instructions on using zsync]]. If your using a virtual machine to test, [[https://wiki.ubuntu.com/UsingDevelopmentReleases|TestDrive]] makes this process extremely easy for you by using zsync to get your desired image, and then one of virtualbox, kvm, or qemu to run the virtual machine for you.
Line 81: Line 81:
As with all testing it is important to [[https://help.ubuntu.com/community/ReportingBugs|file bugs]]. If you find a bug, please, search if it has already been reported, and if it hasn't, report it yourself. You should also refer to the [[https://wiki.ubuntu.com/DebuggingProcedures|bug filing/debugging guide]] for that specific package (if available) to make sure you are aware of known issues and have attached the relevant log files. As with all testing it is important to [[https://help.ubuntu.com/community/ReportingBugs|file bugs]]. If you find a bug, please, search if it has already been reported, and if it hasn't, report it yourself. You should also refer to the [[https://wiki.ubuntu.com/DebuggingProcedures|bug filing/debugging guide]] for that specific package (if available) to make sure you are aware of known issues and have attached the relevant log files. When filing the bug, us the If you find a bug, file it with the '''qa-SERIESNAME-iso''' tag. This can be accomplished readily by using the
{{{
 --tag=qa-SERIESNAME-iso
}}}
tag of ubuntu-bug. For example, for the Quantal Quetzal release, '''ubuntu-bug ubiquity --tag=qa-quantal-iso'''.

Preparations

What you need

  • A spare computer or hard drive for test installs. (A virtual machine can be used but we would prefer to see more tests performed on a variety of physical hardware.)
  • A moderate level of Linux experience so you are able to find the relevant log files and produce useful bug reports: ReportingBugs

  • Some available time before a milestone release to perform tests and file reports.

Setting up

isotesting-logo.png

  • If you do not already have one, set up a Launchpad account so you can report bugs.

  • Login to the test tracker at iso.qa.ubuntu.com which lets you report your test results. The ISO tracker lets you login with your Ubuntu SSO account, but make sure you have logged in at least once to Launchpad using your ubuntu SSO account or else you may encounter issues.

  • Select the ISO images you expect to test and look over the relevant test cases, making sure you have the test setup you need.

Subscribing

  • You can sign up for individual test cases on any ISO. This helps everyone to see which tests are being covered and sends you a notification email when a new build is available, provided you have opted in to email notification inside user settings.

iso-email.png

Downloading images

Daily ISO images are found at cdimage.ubuntu.com. We advise that you download the images you expect to test ahead of time and use zsync or rsync to update them, as this will reduce the turn around time on testing. Zsync only downloads the parts of the image that have changed and typically takes about 10% of the full download time with reasonably fresh images. See instructions on using zsync. If your using a virtual machine to test, TestDrive makes this process extremely easy for you by using zsync to get your desired image, and then one of virtualbox, kvm, or qemu to run the virtual machine for you.

iso-images.png iso-download.png

The images currently being tested are listed on iso.qa.ubuntu.com with a direct download link and the appropriate rsync command for your convenience (click info.png for the download page). Updates to these builds will be announced on #ubuntu-devel, #ubuntu-testing and (optionally) by email.

Testing

In doing specific testing of the CD/DVD images it is important to focus on those aspects that are typically not used by those who simply run the latest unstable version on their system through daily updates (Such testing is of course also extremely valuable). Key points in image testing include image integrity (md5sums), Live CD and installer functionality.

iso-report.png

Each ISO image has a set of test cases associated with it. Click test.png on the tracker to see details of the test case and see Testing/InstallMethods for details on each install method.

To subscribe to a specific ISO and tests, select the flavour from applet menu then the architecture (32 bit, 64 bit) from the list. Then select the tests you wish to subscribe to and then hit subscribe. You will receive an email announcing there are tests available, provided you also opt-in to email notification on your profile.

Reporting results

The test results are tracked on the iso testing tracker website at iso.qa.ubuntu.com. Once you are registered, you can log in to report your test results. The website is divided into several sections, and results are reported based on the following criteria:

  • Flavor of Ubuntu (e.g., Ubuntu, Kubuntu, Xubuntu, Edubuntu, etc.)
  • Type of CD image (e.g., live or alternate install cd)
  • Method of installation (e.g., manual partitioning, partition resizing, etc.)

Links to the different flavors of Ubuntu are present in left-hand menu.

iso-menu.png

Starting a test

ISO test cases normally last for at least half an hour. To avoid duplicating efforts, it is good to let others know that you have started a test case.

To mark a test as "Started" select the result "Started" and submit the result. It will show up in the list with a "clock" icon.

started.png

Adding test reports

When you have conducted a test you can report the results in the ISO tracker with a general comment and bug numbers if applicable.

testreport.png

You can mark a test as Passed or Failed. In the ISO test tracker we mark as failures those experiences that prevented us to do what we wanted to achieve in that test case. I.e. If we want to install, and the partition manager fails, that's a failure. If we do install (or can access to the Live environment, as in this case), the test didn't fail as such. We would mark that as success, but will link the non-critical bugs that we find.

When reporting a broken image it is important that you check the that md5sum corresponds to the ISO being tested to confirm that you do indeed have the correct ISO and that it is not corrupted. To check the md5sum of your image, open a terminal and execute:

md5sum imagename.iso

Where "imagename.iso" is the name of the iso image that you have downloaded and want to verify. The reference value can be found in a file named MD5SUMS in the ISO image download directory. If the md5sum is correct you may also want to run the CD self test on the install media you used.

Bugs

As with all testing it is important to file bugs. If you find a bug, please, search if it has already been reported, and if it hasn't, report it yourself. You should also refer to the bug filing/debugging guide for that specific package (if available) to make sure you are aware of known issues and have attached the relevant log files. When filing the bug, us the If you find a bug, file it with the qa-SERIESNAME-iso tag. This can be accomplished readily by using the

 --tag=qa-SERIESNAME-iso

tag of ubuntu-bug. For example, for the Quantal Quetzal release, ubuntu-bug ubiquity --tag=qa-quantal-iso.

To mark a test as Failed it is mandatory to link the test report with at least one bug in Launchpad. The tracker needs you to type the Launchpad.net bug number. For example: 123456789

Testing/ISO/Procedures (last edited 2013-06-04 22:06:26 by CPE-58-168-89-180)