Walkthrough

Differences between revisions 1 and 24 (spanning 23 versions)
Revision 1 as of 2012-07-25 21:45:24
Size: 1567
Editor: nskaggs
Comment:
Revision 24 as of 2015-02-02 20:45:35
Size: 2996
Editor: nskaggs
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= WARNING -- WIP -- NOT YET COMPLETE -- WARNING = <<Include(QATeam/Header)>>
||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;"><<TableOfContents>>||
Line 4: Line 5:
This page will walk you through testing and submitting a result for an iso. We will execute a single testcase against a ubuntu desktop iso, but the information can be applied to any iso and any testcase. Additionally, we will run our test using a virtual machine. Real hardware can be utilized and is desirable for some forms of testing. This page will walk through submitting a result for an iso test. While this runs through using a single image test the information can be applied to any iso and any testcase.
Line 6: Line 7:
= Requirements =
* Virtualbox
 * sudo apt-get install virtualbox-ose
* TestDrive
 * sudo apt-get install testdrive
* Launchpad account
 * [[https://help.launchpad.net/YourAccount/NewAccount/|Sign up if needed]]
= Prerequisites =
'''Make sure you understand what the [[https://wiki.ubuntu.com/Testing/QATracker|QATracker]] is -- read that page first!'''

'''Read the [[QATeam/DevelopmentSetup|Development Setup]] page'''
Line 16: Line 14:
 * Click the login link located on the lefthand side of the page  * {{attachment:login.png}} Click the login link located on the lefthand side of the page
Line 19: Line 17:
= Obtaining an iso =
In this example, we will be testing the i386 build of the ubuntu desktop iso. To obtain the download link for the iso, let’s learn how to navigate the iso tracker. On the main page is a list of the current milestones. During development there is either a ‘daily’ iso, or a set milestone like alpha, or beta. For our purposes, click the current daily or milestone iso for the current development release. For example, I will click the [[http://iso.qa.ubuntu.com/qatracker/milestones/219/builds|quantal daily]] link.
= Selecting the milestone =
Once logged into the package tracker, you should see a list of milestones. During development there is either a ‘daily’ iso, or a set milestone like alpha, or beta, etc. For this test, click the current daily or alpha/beta, etc, milestone for the current development release.
Line 22: Line 20:
Now, I am presented with a list of products for the milestone I chose. For our example, let’s test the ‘Ubuntu Desktop i386’ iso. Search the page for the link. {{attachment:milestones.png}}

'''NOTE: Only milestones with a status of 'Testing' can have test results submitted. If the status is set to released, no new results can be contributed'''.

= Finding the testcase =
Note the list of products displayed that correspond to the milestone selected. Find the product for the corresponding image you wish to test. For example, the ubuntu desktop amd64 iso. Find the ‘Ubuntu Desktop amd64’ iso by searching the page, then click the link.

{{attachment:products.png}}

Next, a page listing the testcases for the selected iso (product) is shown.

{{attachment:testcases.png}}

{{attachment:links.png}} At the top are two links; one containing the download links for the iso, and the other giving information on how to report a bug found while testing the iso. Take a moment to review the bug reporting instructions page.

Feel free to run any testcase you wish. If this is your first image test, it's recommended to try the "Install (entire disk)" testcase. This is a simple testcase that teaches the basics of installing using ubiquity. For more information on the testcases, see [[https://wiki.ubuntu.com/Testing/InstallMethods|this link]] and/or read through the testcase in question.

= Executing the testcase =
Once you boot from the image, start reading the testcase. The testcases follow a simple [[https://wiki.ubuntu.com/Testing/TestCaseFormat|format]] - perform an action, and then expect a result. A step in this [[https://wiki.ubuntu.com/Testing/TestCaseFormat|format]] is shown below.

{{attachment:testcase.png}}

Go through each step of the testcase and follow the instructions. Perform the action, and ensure the expected result occurs. If something different occurs, note it as a bug, but continue the testcase.

<<Include(Testing/Walkthrough/SubmissionTemplate)>>

Your first ISO test

This page will walk through submitting a result for an iso test. While this runs through using a single image test the information can be applied to any iso and any testcase.

Prerequisites

Make sure you understand what the QATracker is -- read that page first!

Read the Development Setup page

Logging in to the iso tracker

  • Open a browser and navigate to http://iso.qa.ubuntu.com/

  • login.png Click the login link located on the lefthand side of the page

  • Enter your ubuntu sso account information

Selecting the milestone

Once logged into the package tracker, you should see a list of milestones. During development there is either a ‘daily’ iso, or a set milestone like alpha, or beta, etc. For this test, click the current daily or alpha/beta, etc, milestone for the current development release.

milestones.png

NOTE: Only milestones with a status of 'Testing' can have test results submitted. If the status is set to released, no new results can be contributed.

Finding the testcase

Note the list of products displayed that correspond to the milestone selected. Find the product for the corresponding image you wish to test. For example, the ubuntu desktop amd64 iso. Find the ‘Ubuntu Desktop amd64’ iso by searching the page, then click the link.

products.png

Next, a page listing the testcases for the selected iso (product) is shown.

testcases.png

links.png At the top are two links; one containing the download links for the iso, and the other giving information on how to report a bug found while testing the iso. Take a moment to review the bug reporting instructions page.

Feel free to run any testcase you wish. If this is your first image test, it's recommended to try the "Install (entire disk)" testcase. This is a simple testcase that teaches the basics of installing using ubiquity. For more information on the testcases, see this link and/or read through the testcase in question.

Executing the testcase

Once you boot from the image, start reading the testcase. The testcases follow a simple format - perform an action, and then expect a result. A step in this format is shown below.

testcase.png

Go through each step of the testcase and follow the instructions. Perform the action, and ensure the expected result occurs. If something different occurs, note it as a bug, but continue the testcase.

Preparing to submit a result

Evaluating Bugs

While performing the test, if an expected result does not occur, note it as a bug. Follow the bug-reporting instructions for the product you are testing, and report it once the test has ended. A bug can be attached to a 'passed' or 'failed' result, and testcases can have multiple bugs.

If a bug was encountered, check the “Bugs” section above the submit form to see if the bug is a duplicate of one already reported against the testcase.

previousbugs.png

The list contains previously encountered bugs found during testing this testcase. Roll the mouse over the bug icon to see the information, or click to be taken to the report on launchpad. If the bug is found to be a duplicate, note the number of the bug and use it during your report. If the bug was not a duplicate and a new bug needs to be reported, follow the instructions found under the 'Link to bug reporting instructions' for the iso.

Determining the result

A testcase can have unexpected results, but still pass. Generally a passed result can be issued whenever each step in the testcase is able to be executed (even if the results don't match the expected result) and the final result matches the intent of the testcase. If in doubt of whether or not the bug in question represents a testcase failure, report the bug and file the testcase as 'failed' with a comment regarding the confusion. Ask for help on the mailing list for confirmation from someone else on the team.

Submit a result

An example of a failure result submission. Note that a bug number is required to submit a 'failed' result.

failedresult.png

An example of a passed submission.

passedresult.png

Finishing Up

Congratulations, the result has been submitted! After submitting the result should appear in the list below the testcase.

reporterlist.png

If editing is needed, select the 'pencil' icon in the row of your submission to edit the testcase or delete it.

Next Steps

Further testcases can now executed and reported against. Explore other testcases and submit results. If unable to understand or execute a testcase, seek help on the mailing list. Happy Testing!

Testing/ISO/Walkthrough (last edited 2015-02-02 20:45:35 by nskaggs)