Ubuntu Open Week - ISO Testing Tracker - Ara Pulido - Tue, Nov 3, 2009

• utc

(02:04:19 PM) ara: Hello all, thanks for coming!
(02:04:28 PM) ara: My name is Ara and I am part of the Ubuntu QA team
(02:04:42 PM) ara: In this session, part of the Ubuntu Open Week, I am going to talk about the ISO tracker, one of our tools to manage testing
(02:05:05 PM) ara: Do you guys know the ISO tracker and its function in our testing process?
(02:05:50 PM) ara: In the QA team (http://qa.ubuntu.com/) we have two main activities to perform (among many others): Testing and Bug triaging.
(02:06:24 PM) ara: You might already know that Ubuntu has a 6-month release cycle, which means that every 6 months we have a new Ubuntu release, ready to install. Less known is the fact that, before the final release, up to 8 development releases are launched: Alpha 1 to 6, Beta and Release Candidate.
(02:06:51 PM) ara: (although we are going to have only 3 alphas in the cycle that started today)
(02:07:13 PM) ara: Not only for the final release, but for all these releases, we need to test that installation is working OK for the ISO images. The management of the testing of these images is done in the tracker. Everyone can help, and I will show you how.
(02:07:48 PM) ara: The ISO tracker is hosted at http://iso.qa.ubuntu.com/. Go there and have a quick look.
(02:08:03 PM) ara: I'll give you some seconds...
(02:09:11 PM) ara: Milestones normally happen on Thursdays. During that week testers can start looking at the tracker to check if candidates images are appearing.
(02:09:37 PM) ara: OK, first question
(02:09:41 PM) ara: <erUSUL> QUESTION: you only test clean installs with desktop/alternate or there is testing with upgrades using the alternate cd ??
(02:10:15 PM) ara: erUSUL, we also test upgrades, not only clean installations, we will get back to that later
(02:10:43 PM) ara: continuing...
(02:10:43 PM) ara: First thing you will need to do is to create and account in the ISO tracker. Take into account that if you already have an account in the Ubuntu Brainstorm (http://brainstorm.ubuntu.com/) you can use the same account.
(02:11:15 PM) ara: Click on Log in, Create New Account tab.
(02:11:55 PM) ara: Select a username, email, and solve the simple captcha. Your password will be sent to that email account. Let me know when you all have an account.
(02:13:00 PM) ara: While people create their accounts, I will continue explaining a bit how the tracker is structured
(02:14:17 PM) ara: OK, a couple of questions
(02:14:27 PM) ara: <jtniehof> QUESTION: do we have to do anything special to get a brainstorm account to carry over to iso.qa? (I can log into brainstorm, but not iso.qa)
(02:15:08 PM) ara: jtniehof, normally nothing is required. it is weird. I will look into that this week. In the mean time, you can create a new account
(02:15:45 PM) ara: <sebsebseb> QUESTION:  Where is or  when will there be a Lucid Lynx release scheduled?
(02:16:19 PM) ara: sebsebseb, the Lucid schedule is already public at https://wiki.ubuntu.com/LucidReleaseSchedule
(02:16:52 PM) ara: OK, going back to the structure of the tracker...
(02:16:58 PM) ara: Whenever there are images to test, a message tells us which milestone we are testing. In this case, I have created a fake milestone for this session, so we can play around without breaking any previous results :-)
(02:17:26 PM) ara: You can see the message "We are currently testing candidate images for the fake Ubuntu Open Week release"
(02:17:57 PM) ara: At Ubuntu, we produce builds for many different *buntu flavours. In the ISO Tracker we have a category for each of them. You can filter by category using the “Filter” menu in the bottom left of the page. That would allow you to concentrate in the flavour you want to test.
(02:18:36 PM) ara: For this fake milestone I have created fake builds to test for Ubuntu, Ubuntu Netbook Remix, Mythbuntu and Ubuntu Studio
(02:19:03 PM) ara: I haven't included them all to avoid confusion (we have soooo many builds!!!!)
(02:19:48 PM) ara: Once you select a category, you will be able to select an image to download for that category. A category like “Ubuntu” may have different images: i386 Live CD, amd64 alternate, etc., and all of them need to be tested before releasing a milestone.
(02:20:32 PM) ara: If you click now in any of those images, you will have access to a direct link of the ISO to download. Don't try to download it, though. This is a fake milestone, and you won't find much in those links :)
(02:21:37 PM) ara: For each product we have a selection of tests you may run for that image. For every test, there is a link to the test case description in the test case wiki. The test cases are easy to follow when running any of them. All the test cases available need to be covered before the release of a milestone, so everybody is welcome to help :-)
(02:22:12 PM) ara: To have an idea, we had 194 testcases to run for Karmic in each milestone
(02:22:30 PM) ara: So, to sum up the structure, every category contains a set of ISO images and every image contains at least one test case.
(02:22:58 PM) ara: Here you can find a diagram that will help you understand: http://ubuntutesting.files.wordpress.com/2009/09/iso_tracker_diagram.png?w=500&h=353
(02:23:35 PM) ara: Any questions regarding this structure? (or any other question)
(02:24:43 PM) ara: So, let's start pretending that we are in the middle of a testing cycle
(02:25:22 PM) ara: Let's select one of the builds, let's say, Ubuntu Alternate i386
(02:25:26 PM) ara: click on it
(02:25:46 PM) ara: <erUSUL> QUESTION: i asume most people test in VM's or i am mistaken? if so how does that affect the testing?
(02:26:13 PM) ara: erUSUL, testing happens either in real hardware and/or VM. Many people test in VM, but some others test in real HW
(02:26:52 PM) ara: erUSUL, obviously, it is always better in real HW, but HW testing already happen in Canonical certification labs, so it is not that bad to use VMs
(02:27:10 PM) ara: <akgraner> QUESTION: should I download the image, burn a CD and test?  or would it be better to do the testing in a VM?
(02:27:37 PM) ara: akgraner, OK, more or less the same question. Either way is OK. I will get back to this issue later in the talk
(02:27:58 PM) ara: <sebsebseb> QUESTION:  I am thinking what does milestone mean.  I could look it up, but it would be useful to have the answer as part of the session, for other people that aren't sure what it means.
(02:29:09 PM) ara: sebsebseb, good question. A milestone is a announced Ubuntu release that it is not final. We produce official ISOs for those milestones (Alpha1, Alpha2... Beta, RC) and they need to be tested, as we encourage people to install them if they want to help with the development process
(02:29:28 PM) ara: We discourage them, though, for production
(02:29:52 PM) ara: OK, going back to our fake testing
(02:30:11 PM) ara: We have selected Ubuntu alternate i386
(02:30:24 PM) ara: We have a list of 9 test cases that we need to cover
(02:30:39 PM) ara: as you can see right now, nobody have reported any of them
(02:30:48 PM) ara: so we have to work hard to cover them all
(02:31:09 PM) ara: also, we have a link to the ISO on the top of the page
(02:31:47 PM) ara: let's select one of the test cases, let's say Install (auto-resize)
(02:32:46 PM) ara: everybody is in that test case page?
(02:32:57 PM) ara: You can see that nobody has tested it yet
(02:33:08 PM) ara: You have a link to the testcase with instructions to follow
(02:33:25 PM) ara: In this case the link points to http://testcases.qa.ubuntu.com/Install/AlternateResize
(02:33:43 PM) ara: All test cases are described in the test case wiki
(02:34:22 PM) ara: So, once we follow the instructions for that test case, we have to report back to the tracker
(02:35:10 PM) ara: A good report will help the release team decide whether a fix is necessary before announcing the milestone
(02:35:22 PM) ara: So, how do we report?
(02:35:41 PM) ara: First decision: FAILED or PASSED?
(02:35:55 PM) ara: If the installation is unsuccessful, this is, if the bug you encountered is bad enough to prevent you from using the system, mark the test as failed.
(02:36:27 PM) ara: If you encounter no bugs or only bugs that didn't affect directly to your ability to run the system, mark the test as passed
(02:36:31 PM) Idiot is now known as Guest82618
(02:37:31 PM) ara: I.e. An error in the partitioner broke the installation: FAILED
(02:38:02 PM) ara: i.e. You installed correctly, but Firefox was in English, when you installed in Spanish: PASSED (with bugs)
(02:38:59 PM) ara: If you encounter any bugs, no matter if you marked the test as PASSED or FAILED, you should be reporting it in Launchpad. Once reported, add the number of the bug to your report and mark it as "Serious" if you think that the release team should be looking at it as soon as possible.
(02:39:37 PM) ara: And the comments:
(02:39:55 PM) ara: And this is related to the VM/HW question before
(02:40:08 PM) ara: Comments are always welcome. You can explain briefly the configuration of the test machine, if it was run in a physical machine or a virtual machine.
(02:40:28 PM) ara: But  bugs should be added to Launchpad
(02:40:45 PM) ara: I will add now a report for that test case
(02:42:02 PM) ara: If you refresh that page (http://iso.qa.ubuntu.com/qatracker/result/3424/19)
(02:42:09 PM) ara: you can see now my report
(02:42:19 PM) ara: You can add yours as well
(02:42:30 PM) ara: (this is no real data, so you won't do any harm)
(02:42:48 PM) ara: Couple of questions
(02:43:12 PM) ara: QUESTION:  Why is  there  a session about  the milestones/development versions,  (alpha, beta, rc,  which is what I thought it meant),  when  most Ubuntu users are told not to run the development version, because of how they shoudn't be relyed on fully, for proper computer usage.
(02:43:32 PM) ara: sebsebseb, wow, that's more a meta question :-)
(02:43:39 PM) ara: sebsebseb, but I'll try to answer it
(02:44:24 PM) ara: sebsebseb, as I told before, there is a lot of test cases that need testing. Some people want to help with the Ubuntu development and they don't know how to start. This could be a good starting point :-)
(02:45:06 PM) ara: sebsebseb, also, ISO testing is not meant to replace your ubuntu installation. It is better to use spare HW, a spare partition or a virtual machine
(02:45:48 PM) ara: QUESTION: when you said "run the system" is "install the system" you do not test beyond the end of the install; do you?
(02:46:53 PM) ara: erUSUL, you can do some smoke testing. Open your browser, browse the web, use open office, etc, use compiz, etc. Do the things you would normally do.
(02:47:40 PM) ara: OK, continuing...
(02:48:11 PM) ara: (btw, I've seen some of you already "reported" in the tracker. Nice :-)
(02:48:24 PM) ara: As I told before, every single test case needs to be run at least once before releasing a milestone. When the date of the release is near, it is very important to focus on those tests that haven't been run yet.
(02:48:51 PM) ara: These following two links give you the clue on what to test
(02:48:59 PM) ara: http://iso.qa.ubuntu.com/qatracker/build/all/untested
(02:48:59 PM) ara: http://iso.qa.ubuntu.com/qatracker/build/all/inprogress
(02:49:40 PM) ara: for our fake milestone we already have one in progress, \o/
(02:50:15 PM) ara: Another feature of the ISO tracker: Subscriptions
(02:50:43 PM) ara: Sometimes you can only want to help with a flavour or kind of build. If you are, let's say, a Mythbuntu user and only care about this flavour of Ubuntu, you can subscribe to it, and get email announces every time a new build of your subscriptions is ready to test. To subscribe to a type of build, just click on the build, select the testcases you would like to test (normally you would select all from a build) and click on subs
(02:50:43 PM) ara: cribe. Finally, to subscribe to the emails, go to your profile page, on the top right corner, and select E-mail notification.
(02:51:58 PM) ara: ok, a good question
(02:51:59 PM) ara: <kennethvenken> QUESTION: test cases are run prior to releasing a milestone. So when are they tested? Do the tests for alpha 6 start right after the alpha 5 release or a few days before alpha 6?
(02:53:12 PM) ara: OK, normally testing start happening Monday of a week of a release (that happen on Thursdays). But it can vary, so subscriptions are very important if you want to be notified about it
(02:53:30 PM) ara: So, when it is going to be the next ISO testing week?
(02:53:49 PM) ara: As we told before, you can check the Lucid Lynx release schedule at https://wiki.ubuntu.com/LucidReleaseSchedule
(02:54:04 PM) ara: As you can see, the first milestone, Alpha 1, is scheduled to be released December 3rd (not surprisingly, a Thursday). That means that the week of November 30th is the perfect week to help testing the ISOs.
(02:54:54 PM) ara: So, if you want to help testing, join us on #ubuntu-testing that week and start helping with the Ubuntu development process!
(02:55:18 PM) ara: (we are daily on that channel to solve and chat about Ubuntu testing)
(02:55:51 PM) ara: <sebsebseb> QUESTION: Why is there only going to be three alphas, instead of six?
(02:56:43 PM) ara: sebsebseb, I guess that was a decision from the release team, and I don't have the answer. You can ask it in #ubuntu-release and the release team will surely help you
(02:58:04 PM) ara: OK, if there are no more questions I think we can wrap up
(02:58:38 PM) ara: I hope to see you all at #ubuntu-testing for Lucid Alpha 1 ISO testing!