Your first ISO test
This page will walk through testing and submitting a result for an iso. A single testcase will be executed against a ubuntu desktop iso, but the information can be applied to any iso and any testcase. Additionally, the test run will utilize a virtual machine. Real hardware can be utilized and is desirable for some forms of testing, but a VM is utilized for simplicity in this case.
Video Screencast of Tutorial
A video version of this tutorial can be seen on youtube.
- sudo apt-get install virtualbox
- sudo apt-get install testdrive
- Launchpad/Ubuntu SSO account
Make sure you understand what the QATracker is.
Logging in to the iso tracker
Open a browser and navigate to http://iso.qa.ubuntu.com/
Click the login link located on the lefthand side of the page
- Enter your ubuntu sso account information
Selecting the milestone
Once logged into the package tracker, you should see a list of milestones. During development there is either a ‘daily’ iso, or a set milestone like alpha, or beta, etc. For this test, click the current daily or alpha/beta, etc, milestone for the current development release.
NOTE: Only milestones with a status of 'Testing' can have test results submitted. If the status is set to released, no new results can be contributed.
Finding the testcase
Note the list of products displayed that correspond to the milestone selected. In this example, testing will occur against the i386 build of the ubuntu desktop iso. Find the ‘Ubuntu Desktop i386’ iso by searching the page, then click the link.
Next, a page listing the testcases for the selected iso (product) is shown.
At the top are two links; one containing the download links for the iso, and the other giving information on how to report a bug found while testing the iso. Take a moment to review the bug reporting instructions page. TestDrive will download the iso by itself, so it is not necessary to download the iso manually using the given download link.
Click the link for the “Install (entire disk)" testcase. This is a simple testcase that teaches the basics of installing using ubiquity. For more information on the testcases, see this link and/or read through the testcase in question.
Next, testdrive will be used to setup the virtual machine and download the iso to begin the test.
Obtaining the iso
Open the dash and startup TestDrive.
Select the ubuntu i386 iso from the list of iso's shown inside testdrive and press the ‘sync’ button. Testdrive will download the iso showing it's progress underneath the iso. Once finished, click the launch button and a virtual machine containing the downloaded iso as the primary boot device will start. Reference this wiki page for more information on testdrive itself. It's important to remember to sync the iso before each launch. This ensures the most up to date version of the iso is being tested and the results won't be skewed by using an older version.
Executing the testcase
Go through each step of the testcase and follow the instructions. Perform the action, and ensure the expected result occurs. If something different occurs, note it as a bug, but continue the testcase.
Preparing to submit a result
While performing the test, if an expected result does not occur, note it as a bug. Follow the bug-reporting instructions for the product you are testing, and report it once the test has ended. A bug can be attached to a 'passed' or 'failed' result, and testcases can have multiple bugs.
If a bug was encountered, check the “Bugs” section above the submit form to see if the bug is a duplicate of one already reported against the testcase.
The list contains previously encountered bugs found during testing this testcase. Roll the mouse over the bug icon to see the information, or click to be taken to the report on launchpad. If the bug is found to be a duplicate, note the number of the bug and use it during your report. If the bug was not a duplicate and a new bug needs to be reported, follow the instructions found under the 'Link to bug reporting instructions' for the iso.
Determining the result
A testcase can have unexpected results, but still pass. Generally a passed result can be issued whenever each step in the testcase is able to be executed (even if the results don't match the expected result) and the final result matches the intent of the testcase. If in doubt of whether or not the bug in question represents a testcase failure, report the bug and file the testcase as 'failed' with a comment regarding the confusion. Ask for help on the mailing list for confirmation from someone else on the team.
Submit a result
An example of a failure result submission. Note that a bug number is required to submit a 'failed' result.
An example of a passed submission.
Congratulations, the result has been submitted! After submitting the result should appear in the list below the testcase.
If editing is needed, select the 'pencil' icon in the row of your submission to edit the testcase or delete it.
Further testcases can now executed and reported against. Explore other testcases and submit results. If unable to understand or execute a testcase, seek help on the mailing list. Happy Testing!