Walkthrough

Differences between revisions 10 and 11
Revision 10 as of 2012-08-01 19:42:18
Size: 5244
Editor: nskaggs
Comment:
Revision 11 as of 2012-08-02 16:16:52
Size: 6236
Editor: nskaggs
Comment:
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
This page will walk you through testing and submitting a result for an iso. We will execute a single testcase against a ubuntu desktop iso, but the information can be applied to any iso and any testcase. Additionally, we will run our test using a virtual machine. Real hardware can be utilized and is desirable for some forms of testing. This page will walk through testing and submitting a result for an iso. A single testcase will be executed against a ubuntu desktop iso, but the information can be applied to any iso and any testcase. Additionally, the test run will utilize a virtual machine. Real hardware can be utilized and is desirable for some forms of testing, but a VM is utilized for simplicity in this case.
Line 5: Line 5:
If you wish, a video version of this tutorial [[https://www.youtube.com/watch?v=noDAsZjvmSA|can be seen on youtube]]. A video version of this tutorial [[https://www.youtube.com/watch?v=noDAsZjvmSA|can be seen on youtube]].
Line 21: Line 21:
In this example, we will be testing the i386 build of the ubuntu desktop iso. To obtain the download link for the iso, let’s learn how to navigate the [[http://iso.qa.ubuntu.com/|iso tracker]]. On the main page is a list of the current milestones. During development there is either a ‘daily’ iso, or a set milestone like alpha, or beta. For our purposes, click the current daily or milestone for the current development release. In this example, testing will occur against the i386 build of the ubuntu desktop iso. The download link for the iso can be obtained by navigating the [[http://iso.qa.ubuntu.com/|iso tracker]]. On the main page is a list of the current milestones. During development there is either a ‘daily’ iso, or a set milestone like alpha, or beta. For this test, click the current daily or milestone for the current development release.
Line 25: Line 25:
Now, I am presented with a list of products for the milestone I chose. For our example, let’s test the ‘Ubuntu Desktop i386’ iso. Search the page for the link and click it. Note the list of products display that correspond to the milestone selected. Fin the ‘Ubuntu Desktop i386’ iso by searching the page, and click the link.
Line 29: Line 29:
Now we’re presented with a page that lists the testcases for the iso we selected. The page lists the testcases for the iso (product) selected.
Line 33: Line 33:
{{attachment:links.png}} At the top are two links; one containing the download links for the iso, and the other giving you information on how to report a bug you may find while testing the iso. Take a moment to review the bug reporting instructions page. We’ll be using [[https://wiki.ubuntu.com/UsingDevelopmentReleases|TestDrive]] to download the iso for us, so we don’t have to worry about grabbing the download link for the iso. {{attachment:links.png}} At the top are two links; one containing the download links for the iso, and the other giving information on how to report a bug found while testing the iso. Take a moment to review the bug reporting instructions page. [[https://wiki.ubuntu.com/UsingDevelopmentReleases|TestDrive]] will download the iso by itself, so it is not necessary to download the iso manually using the download link.
Line 35: Line 35:
So, let’s select a simple testcase to start. Click the link for the “Install (entire disk) testcase. If your curious as to what some of the other testcases mean, see [[https://wiki.ubuntu.com/Testing/InstallMethods|this link]]. Click the link for the “Install (entire disk)" testcase. This is a simple testcase that teaches the basics of installing using ubuiquity. For more information on the testcases, see [[https://wiki.ubuntu.com/Testing/InstallMethods|this link]] and/or read through the testcase in question.
Line 37: Line 37:
Great, we’re all ready to begin. Now let’s download our iso and setup a virtual machine using testdrive and virtualbox. Next, testdrive will be used to setup the virtual machine and download the iso to begin the test.
Line 42: Line 42:
Select the ubuntu i386 iso we want to test and press the ‘sync’ button. It will download the iso. Once finished, click the launch button and the VM will start. See the [[https://wiki.ubuntu.com/UsingDevelopmentReleases|wiki page]] for more information on testdrive itself. It's important to remember to sync your iso before attempting to test -- if you don't you won't be testing the most up to date version of the iso and your results won't match. Select the ubuntu i386 iso from the list of iso's shown inside testdrive and press the ‘sync’ button. Testdrive will download the iso showing it's progress underneath the iso. Once finished, click the launch button and a virtual machine containing the downloaded iso as the primary boot device will start. Reference this [[https://wiki.ubuntu.com/UsingDevelopmentReleases|wiki page]] for more information on testdrive itself. It's important to remember to sync the iso before each launch. This ensures the most up to date version of the iso is being tested and the results won't be skewed by using an older version.
Line 45: Line 45:
With the VM launched, start reading the testcase. Most of the testcases should follow a simple format - perform an action, and then expect a result. A step in this format is shown below. With the VM launched, start reading the testcase. Most of the testcases follow a simple format - perform an action, and then expect a result. A step in this format is shown below.
Line 49: Line 49:
Go through each step of the testcase and follow the instructions. Go through each step of the testcase and follow the instructions. Perform the action, and ensure the expected result occurs. If something different occurs, note it as a bug, but continue the testcase.
Line 52: Line 52:
Ok, the test is all finished. Did everything go ok? If so, you can fill in the result at the bottom with a “pass”. Don’t worry about the hardware profile field for now. If you encountered a bug, it’s a good idea to take a look at the “Bugs” section above the submit form. == Evaluating Bugs ==
While performing the test, if an expected result does not occur, note it as a bug. Follow the bug-reporting instructions for the iso you are testing, and report it once the test has ended. A bug can be attached to a 'passed' or 'failed' result, and testcases can have multiple bugs.

If a bug was encountered, check the “Bugs” section above the submit form to see if the bug is a duplicate of one already reported against the testcase.
Line 56: Line 59:
It will contain previously encountered bugs, of which you may have found a duplicate. Roll your mouse over the bug icon to see the information, or click to be taken to the report on launchpad.
If your bug is a duplicate, simply note the number and report as usual. If it was not a duplicate and you need to report a new bug, if possible report the bug using ubuntu-bug the instructions you read earlier.
In our case, we want to use ‘ubuntu-bug ubiquity’ from our VM installation. Using ‘ubuntu-bug’ will guide us through the bug filing process and attach important logging information to help fix the bug. Once you’ve finished reporting the bug in launchpad, make sure to note the bug number. You’ll need it to place in the bugs field when you submit your report. Now if you encountered a bug but were still able to complete the installation, still file the bug and report it when you submit your ‘passed’ result.
The list contains previously encountered bugs found during testing this testcase. Roll the mouse over the bug icon to see the information, or click to be taken to the report on launchpad.
If the bug is found to be a duplicate, note the number of the bug and use it during your report. {{attachment:links.png}} If the bug was not a duplicate and a new bug needs to be reported, follow the instructions found under the 'Link to bug reporting instructions' for the iso.

= Determining the result =
A testcase can have unexpected results, but still pass. Generally a passed result can be issued whenever each step in the testcase is able to be executed (even if the results don't match the expected result) and the final result matches the intent of the testcase. If in doubt of whether or not the bug in question represents a testcase failure, report the bug and file the testcase as 'failed' with a comment regarding the confusion. Ask for help on the [[https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa|mailing list]] for confirmation from someone else on the team.
Line 62: Line 67:
An example of a failure result submission. Note that you are required to submit a bug if the test failed. An example of a failure result submission. Note that a bug number is required to submit a 'failed' result.
Line 65: Line 70:
Line 72: Line 76:
Congratulations, you’ve just submitted your first result. After submitting you should see your result in the list below the testcase. Congratulations, you’ve just submitted your first result! After submitting the result should appear in the list below the testcase.
Line 75: Line 79:

If editing is needed, select the 'pencil' icon in the row of your submission to edit the testcase or delete it.

= Next Steps =
Further testcases can now executed and reported against. Explore other testcases and submit results. If unable to understand or execute a testcase, seek help on the [[https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa|mailing list]]. Happy Testing!

Your first ISO test

This page will walk through testing and submitting a result for an iso. A single testcase will be executed against a ubuntu desktop iso, but the information can be applied to any iso and any testcase. Additionally, the test run will utilize a virtual machine. Real hardware can be utilized and is desirable for some forms of testing, but a VM is utilized for simplicity in this case.

Video Screencast of Tutorial

A video version of this tutorial can be seen on youtube.

Requirements

  • Virtualbox
    • sudo apt-get install virtualbox-ose
  • TestDrive

    • sudo apt-get install testdrive
  • Launchpad/Ubuntu SSO account

Logging in to the iso tracker

  • Open a browser and navigate to http://iso.qa.ubuntu.com/

  • login.png Click the login link located on the lefthand side of the page

  • Enter your ubuntu sso account information

Finding the testcase

In this example, testing will occur against the i386 build of the ubuntu desktop iso. The download link for the iso can be obtained by navigating the iso tracker. On the main page is a list of the current milestones. During development there is either a ‘daily’ iso, or a set milestone like alpha, or beta. For this test, click the current daily or milestone for the current development release.

milestones.png

Note the list of products display that correspond to the milestone selected. Fin the ‘Ubuntu Desktop i386’ iso by searching the page, and click the link.

products.png

The page lists the testcases for the iso (product) selected.

testcases.png

links.png At the top are two links; one containing the download links for the iso, and the other giving information on how to report a bug found while testing the iso. Take a moment to review the bug reporting instructions page. TestDrive will download the iso by itself, so it is not necessary to download the iso manually using the download link.

Click the link for the “Install (entire disk)" testcase. This is a simple testcase that teaches the basics of installing using ubuiquity. For more information on the testcases, see this link and/or read through the testcase in question.

Next, testdrive will be used to setup the virtual machine and download the iso to begin the test.

Obtaining the iso

Open the dash and startup TestDrive.

Select the ubuntu i386 iso from the list of iso's shown inside testdrive and press the ‘sync’ button. Testdrive will download the iso showing it's progress underneath the iso. Once finished, click the launch button and a virtual machine containing the downloaded iso as the primary boot device will start. Reference this wiki page for more information on testdrive itself. It's important to remember to sync the iso before each launch. This ensures the most up to date version of the iso is being tested and the results won't be skewed by using an older version.

Executing the testcase

With the VM launched, start reading the testcase. Most of the testcases follow a simple format - perform an action, and then expect a result. A step in this format is shown below.

testcase.png

Go through each step of the testcase and follow the instructions. Perform the action, and ensure the expected result occurs. If something different occurs, note it as a bug, but continue the testcase.

Preparing to submit a result

Evaluating Bugs

While performing the test, if an expected result does not occur, note it as a bug. Follow the bug-reporting instructions for the iso you are testing, and report it once the test has ended. A bug can be attached to a 'passed' or 'failed' result, and testcases can have multiple bugs.

If a bug was encountered, check the “Bugs” section above the submit form to see if the bug is a duplicate of one already reported against the testcase.

previousbugs.png

The list contains previously encountered bugs found during testing this testcase. Roll the mouse over the bug icon to see the information, or click to be taken to the report on launchpad. If the bug is found to be a duplicate, note the number of the bug and use it during your report. links.png If the bug was not a duplicate and a new bug needs to be reported, follow the instructions found under the 'Link to bug reporting instructions' for the iso.

Determining the result

A testcase can have unexpected results, but still pass. Generally a passed result can be issued whenever each step in the testcase is able to be executed (even if the results don't match the expected result) and the final result matches the intent of the testcase. If in doubt of whether or not the bug in question represents a testcase failure, report the bug and file the testcase as 'failed' with a comment regarding the confusion. Ask for help on the mailing list for confirmation from someone else on the team.

Submit a result

An example of a failure result submission. Note that a bug number is required to submit a 'failed' result.

failedresult.png

An example of a passed submission.

passedresult.png

Finishing Up

Congratulations, you’ve just submitted your first result! After submitting the result should appear in the list below the testcase.

reporterlist.png

If editing is needed, select the 'pencil' icon in the row of your submission to edit the testcase or delete it.

Next Steps

Further testcases can now executed and reported against. Explore other testcases and submit results. If unable to understand or execute a testcase, seek help on the mailing list. Happy Testing!

Testing/ISO/Walkthrough (last edited 2015-02-02 20:45:35 by nskaggs)