Revision 2 as of 2009-11-03 19:32:05

Clear message

Ubuntu Open Week - ISO Testing Tracker - Ara Pulido - Tue, Nov 3, 2009

  • utc

(02:04:19 PM) ara: Hello all, thanks for coming!
(02:04:28 PM) ara: My name is Ara and I am part of the Ubuntu QA team
(02:04:42 PM) ara: In this session, part of the Ubuntu Open Week, I am going to talk about the ISO tracker, one of our tools to manage testing
(02:05:05 PM) ara: Do you guys know the ISO tracker and its function in our testing process?
(02:05:50 PM) ara: In the QA team (http://qa.ubuntu.com/) we have two main activities to perform (among many others): Testing and Bug triaging.
(02:06:24 PM) ara: You might already know that Ubuntu has a 6-month release cycle, which means that every 6 months we have a new Ubuntu release, ready to install. Less known is the fact that, before the final release, up to 8 development releases are launched: Alpha 1 to 6, Beta and Release Candidate.
(02:06:51 PM) ara: (although we are going to have only 3 alphas in the cycle that started today)
(02:07:13 PM) ara: Not only for the final release, but for all these releases, we need to test that installation is working OK for the ISO images. The management of the testing of these images is done in the tracker. Everyone can help, and I will show you how.
(02:07:48 PM) ara: The ISO tracker is hosted at http://iso.qa.ubuntu.com/. Go there and have a quick look.
(02:08:03 PM) ara: I'll give you some seconds...
(02:09:11 PM) ara: Milestones normally happen on Thursdays. During that week testers can start looking at the tracker to check if candidates images are appearing.
(02:09:37 PM) ara: OK, first question
(02:09:41 PM) ara: <erUSUL> QUESTION: you only test clean installs with desktop/alternate or there is testing with upgrades using the alternate cd ??
(02:10:15 PM) ara: erUSUL, we also test upgrades, not only clean installations, we will get back to that later
(02:10:43 PM) ara: continuing...
(02:10:43 PM) ara: First thing you will need to do is to create and account in the ISO tracker. Take into account that if you already have an account in the Ubuntu Brainstorm (http://brainstorm.ubuntu.com/) you can use the same account.
(02:11:15 PM) ara: Click on Log in, Create New Account tab.
(02:11:55 PM) ara: Select a username, email, and solve the simple captcha. Your password will be sent to that email account. Let me know when you all have an account.
(02:13:00 PM) ara: While people create their accounts, I will continue explaining a bit how the tracker is structured
(02:14:17 PM) ara: OK, a couple of questions
(02:14:27 PM) ara: <jtniehof> QUESTION: do we have to do anything special to get a brainstorm account to carry over to iso.qa? (I can log into brainstorm, but not iso.qa)
(02:15:08 PM) ara: jtniehof, normally nothing is required. it is weird. I will look into that this week. In the mean time, you can create a new account
(02:15:45 PM) ara: <sebsebseb> QUESTION:  Where is or  when will there be a Lucid Lynx release scheduled?
(02:16:19 PM) ara: sebsebseb, the Lucid schedule is already public at https://wiki.ubuntu.com/LucidReleaseSchedule
(02:16:52 PM) ara: OK, going back to the structure of the tracker...
(02:16:58 PM) ara: Whenever there are images to test, a message tells us which milestone we are testing. In this case, I have created a fake milestone for this session, so we can play around without breaking any previous results :-)
(02:17:26 PM) ara: You can see the message "We are currently testing candidate images for the fake Ubuntu Open Week release"
(02:17:57 PM) ara: At Ubuntu, we produce builds for many different *buntu flavours. In the ISO Tracker we have a category for each of them. You can filter by category using the “Filter” menu in the bottom left of the page. That would allow you to concentrate in the flavour you want to test.
(02:18:36 PM) ara: For this fake milestone I have created fake builds to test for Ubuntu, Ubuntu Netbook Remix, Mythbuntu and Ubuntu Studio
(02:19:03 PM) ara: I haven't included them all to avoid confusion (we have soooo many builds!!!!)
(02:19:48 PM) ara: Once you select a category, you will be able to select an image to download for that category. A category like “Ubuntu” may have different images: i386 Live CD, amd64 alternate, etc., and all of them need to be tested before releasing a milestone.
(02:20:32 PM) ara: If you click now in any of those images, you will have access to a direct link of the ISO to download. Don't try to download it, though. This is a fake milestone, and you won't find much in those links :)
(02:21:37 PM) ara: For each product we have a selection of tests you may run for that image. For every test, there is a link to the test case description in the test case wiki. The test cases are easy to follow when running any of them. All the test cases available need to be covered before the release of a milestone, so everybody is welcome to help :-)
(02:22:12 PM) ara: To have an idea, we had 194 testcases to run for Karmic in each milestone
(02:22:30 PM) ara: So, to sum up the structure, every category contains a set of ISO images and every image contains at least one test case.
(02:22:58 PM) ara: Here you can find a diagram that will help you understand: http://ubuntutesting.files.wordpress.com/2009/09/iso_tracker_diagram.png?w=500&h=353
(02:23:35 PM) ara: Any questions regarding this structure? (or any other question)
(02:24:43 PM) ara: So, let's start pretending that we are in the middle of a testing cycle
(02:25:22 PM) ara: Let's select one of the builds, let's say, Ubuntu Alternate i386
(02:25:26 PM) ara: click on it
(02:25:46 PM) ara: <erUSUL> QUESTION: i asume most people test in VM's or i am mistaken? if so how does that affect the testing?
(02:26:13 PM) ara: erUSUL, testing happens either in real hardware and/or VM. Many people test in VM, but some others test in real HW
(02:26:52 PM) ara: erUSUL, obviously, it is always better in real HW, but HW testing already happen in Canonical certification labs, so it is not that bad to use VMs
(02:27:10 PM) ara: <akgraner> QUESTION: should I download the image, burn a CD and test?  or would it be better to do the testing in a VM?
(02:27:37 PM) ara: akgraner, OK, more or less the same question. Either way is OK. I will get back to this issue later in the talk
(02:27:58 PM) ara: <sebsebseb> QUESTION:  I am thinking what does milestone mean.  I could look it up, but it would be useful to have the answer as part of the session, for other people that aren't sure what it means.
(02:29:09 PM) ara: sebsebseb, good question. A milestone is a announced Ubuntu release that it is not final. We produce official ISOs for those milestones (Alpha1, Alpha2... Beta, RC) and they need to be tested, as we encourage people to install them if they want to help with the development process
(02:29:28 PM) ara: We discourage them, though, for production
(02:29:52 PM) ara: OK, going back to our fake testing
(02:30:11 PM) ara: We have selected Ubuntu alternate i386
(02:30:24 PM) ara: We have a list of 9 test cases that we need to cover
(02:30:39 PM) ara: as you can see right now, nobody have reported any of them
(02:30:48 PM) ara: so we have to work hard to cover them all
(02:31:09 PM) ara: also, we have a link to the ISO on the top of the page
(02:31:47 PM) ara: let's select one of the test cases, let's say Install (auto-resize)