Ubuntu Wiki

Ubuntu Open Week - Ubuntu Iso Testing - Henrik Omma & David Morley - Fri, May 2, 2008

=== jcastro changed the topic of #ubuntu-classroom to: Ubuntu Open Week | Information and Logs: https://wiki.ubuntu.com/UbuntuOpenWeek | How to ask questions: https://wiki.ubuntu.com/UbuntuOpenWeek/Rules | Ask questions in #ubuntu-classroom-chat, prefaced with "QUESTION:" |See https://wiki.ubuntu.com/UbuntuOpenWeek/JoiningIn to filter out channel noise | "Ubuntu ISO testing" Henrik Omma & David Morley

[18:59] <jcastro> heno and davmor2 will be discussing ISO testing, which is a great way to 
help getting Ubuntu out the door!

[19:00] <heno> hello :)

[19:00] <davmor2> hello

[19:01] <heno> Some of you may have perticipated in ISO testing before the Hardy release

[19:03] <davmor2> The basis of the testing team is to get the Ubuntu family out of the door 
with as few major bugs as possible.

[19:04] <davmor2> The main reason we are here today is that the more tester we have the more 
bugs get spotted.

[19:05] <davmor2> The good thing is if you can install ubuntu on your machine at home you can 
test it before releases.

[19:05] <heno> thanks, I was locked out for a second :)

[19:05] <davmor2> There are two main ways to do this one Virtual.  Two on hardware.

[19:06] <heno> We need to both validate the actual CD images, but also search for bugs 
generally

[19:06] <heno> even though the devel version is used regularly by people, structured testing always uncovers bugs

[19:07] <heno> esp. in features that are not used every day like the installer

[19:08] <heno> To actually contribute productively in the testing crunch you actually need to 
prepare a bit ahead of time

[19:09] <heno> It helps to have read the test cases and pre-download the ISO images

[19:09] <heno> you then update them with rsync

[19:10] <heno> the images are often re-spun 2-3 times after the archive freezes as bugs are 
fixed

[19:10] <heno> Most of what you'd need to know is covered here 
https://wiki.ubuntu.com/Testing/ISO

[19:10] <heno> and we hang out in #ubuntu-testing to coordinate

[19:12] <heno> we test based on the cases here https://wiki.ubuntu.com/Testing/Cases and 
report results here http://iso.qa.ubuntu.com/

[19:12] <heno> Ok, that's a brief introduction, let's open up for questions

[19:13] <ompaul> <hardywireless> QUESTION: i have done some iso testing, and it was great to 
do, but , having found a bug, i submitted it, and it was worked on, but i could not submit 
more then one bug report...( as i tested the next daily build the following day )

[19:13] <heno> next

[19:14] <heno> hardywireless: I don't understand why you couldn't submit more than one bug 
report, was this in Launchpad?

[19:15] <heno> you can only submit one test report for a given test case on a given build, 
but you can attach as many bug numbers as you need to

[19:15] <ompaul> <hardywireless> QUESTION: iso testing site

[19:16] <heno> if the image had not changed then the original report would still be up and 
you can edit it to add more bug links

[19:17] <heno> you click the document view icon next you the report

[19:18] <heno> at least I think you can update your own cases

[19:18] <davmor2> hardywireless: The issue here I think is that when a new image is released 
that becomes the default for tracker.  You can however add comments to the newer version like 
bug xxxxxx is now fixed etc.

[19:18] <heno> as site admin I can update all cases

[19:19] <heno> davmor2: can you confirm that you can edit your own test case reports?

[19:19] <ompaul> <hardywireless> QUESTION: i also noticed that the 
http://iso.qa.ubuntu.com/qatracker/ is not always as up to date as 
http://cdimage.ubuntu.com/daily/current/

[19:20] <davmor2> heno: Yes I edit them regularly to add bug reports.  I tend to put in a 
note and then write the reports after and add them all at once

[19:20] <heno> that's true. we need to automate the sync there

[19:20] <ompaul> <BonesolTeraDyne> QUESTION: What do you suggest as a minimum amount of time 
a tester should spend on bug hunting\ testing a build?

[19:21] <heno> sometimes the golden image candidate will be held back to an earlier build on 
purpose though

[19:22] <davmor2> hardywireless: Also the tracker only kicks in for candidate images and not 
all daily builds

[19:22] <heno> BonesolTeraDyne: as long as it takes to go through the test case

[19:22] <heno> testing more is of course a bonus

[19:23] <ompaul> Question: If I could put together a package that would allow automated 
installation for testing purposes, is there any possibility that it could be included in the 
development versions of Ubuntu (assuming it gets sponsored for main, etc)

[19:23] <heno> in reality we sometimes do quick sanity checks on certain install types or 
features

[19:24] <heno> ompaul: how automated? not asking for a password? :)

[19:24] <ompaul> heno, I have to ask the person who asked me brb

[19:24] <davmor2> That would be cool.  However sometimes the person doing the install turns 
up a bug by simply not knowing what to do next :)

[19:25] <ompaul> hardywireless> comment: more of a compliment, as i went to irc 
ubuntu-testing, the bug i reported was fixed in a few day's just before the release of hardy, 
thats really great...

[19:25] <ompaul> heno, yes

[19:25] <heno> We are doing some automation of install testing of images

[19:25] <heno> and we also want to automate upgrade testing with a ton of packages installed

[19:26] <heno> there has also been talk of adding desktop automation test scripts to the live 
CD - self-testing-desktop

[19:27] <heno> but I'm not sure I see the value in just installing a bunch of packages 
automatically

[19:27] <heno> there are already tools like piuparts, lintian and auto-package-test

[19:27] <ompaul> heno, they would match jane's or john's use case

[19:28] <heno> that install and test packages, but that's in a controlled environment like a 
chroot or kvm

[19:28] <heno> ompaul: if it's done in a responsible way, then yes

[19:29] <ompaul> heno, vis the source of the question: <highvoltage> for my question, I was 
thinking more in terms of iso testing

[19:29] <heno> I don't want a package installable from synaptic that when run installs 1000 
more packages though :)

[19:29] <ompaul> <highvoltage> so that you could put a bunch of isos (like 20 of them or so) 
in a directory, and have it all installed in a vm automatically

[19:30] <heno> highvoltage: from Hardy we can do preseeding of the Ubiquity installer as 
well, which would be one way to automate it

[19:31] <heno> another way would be to set up a way to poke into a kvm or vbox instance to 
drive it

[19:31] <ompaul> highvoltage> it wouldn't install the packages automatically you would be 
able to pass a kernel parameter to the boot cd that would automate the /entire/ install 
process by sending keystrokes to the installer

[19:31] <heno> highvoltage: we would love to do that for Intrepid and help would be great :)

[19:32] <heno> highvoltage: yep, I think feeding input to the installer would be even better 
than pre-seeding, more realistic

[19:32] <davmor2> highvoltage: it certainly would help take the pressure off installing in vm 
:)

[19:33] <heno> highvoltage: let's speak more about the details afterward

[19:33] <heno> next?

[19:35] <ompaul> <hardywireless> QUESTION: can there be a way to blacklist drivers , when 
booting from livecd , for example if it hangs on a such a driver ...

[19:36] <heno> this is not my area, but you can turn of certain things like acpi with kernel 
parameters

[19:36] <heno> not sure there is an option for every driver

[19:37] <davmor2> I think one of the key things to remember about the Iso Testing Team is 
that it is ideal for newish users trying to find a way to contribute to Ubuntu.  There is 
nothing that technical to testing.  If you can install and run a system you can test.

[19:38] <heno> hardywireless: ompaul helpfully refers us to 
https://help.ubuntu.com/community/BootOptions

[19:39] <davmor2> next

[19:39] <ompaul> hardywireless, that page contains a list for how to launch both an installed 
system and iso with boot options, generally setting module=no is good but cases may break 
that mode so I would say check before you jump

[19:41] <ompaul> <hardywireless> QUESTION: how many people are doing iso testing right now?

[19:42] <davmor2> Hmmm tricky question.  Does that include all the users who install or 
upgrade before the release is out?

[19:42] <heno> hardywireless: there are typically 5-6 people who do a large number of tests and then ~30 who do one test image each or so

[19:43] <heno> there is an overview at http://iso.qa.ubuntu.com/qatracker/subscriptions

[19:43] <heno> next?

[19:44] <ompaul> <hardywireless> QUESTION: whats the next version we can test? :) , and how 
did you become a tester?

[19:45] <heno> the next sensible version to test is Intrepid Alpha 1

[19:45] <heno> there will also be a hardy update release to test

[19:46] <heno> that reminds me: please run hardy wit -proposed turned on so we can test those 
updates

[19:46] <davmor2> I started contributing back to Ubuntu by joining Bug Squad.  However with 
the apport bug reports becoming more frequent it became to complex for me.  So I looked for 
something else and at the time Heno asked for testers.

[19:47] <heno> (I'm employed by Canonical and started dabbling in QA about a year ago)

[19:48] <heno> I did bug triage first and the coordinated testing for 7.04

[19:49] <heno> more questions?

[19:49] <ompaul> <chipitsine> QUESTION: I noticed ext2 cannot be mounted by LABEL even man 
page states so in 7.05 <chipitsine> in 7.04 actually

[19:50] <heno> hm, not quite related to ISO testing as such

[19:51] <ompaul> <ted_> QUESTION: Should more emphasis be placed on testing ubiquity because 
it can't be updated on the CD, whereas all the other apps can be updated post-install?

[19:51] <heno> chipitsine: please file general bugs in launchpad

[19:52] <heno> ted_: it's clearly a major testing target, yes, but from experience we also 
find important bugs in other things during testing too

[19:52] <heno> but ubiquity really depends on this testing

[19:52] <davmor2> ted_: The same can be said for Debian-Installer on the alternative cds too.

[19:52] <heno> esp. as bugs can be hardware-specific

[19:53] <heno> that's one reason I'm afraid of automating install testing too much

[19:54] <heno> human testing will often find things you haven't scripted for

[19:54] <heno> next?

[19:54] <heno> We'll take one more I think

[19:54] <davmor2> especially if the automation is setup by some one with the knowledge to do 
the install correctly.

[19:55] <heno> please continue to ask in #ubuntu-testing though

[19:55] <heno> it's often good to do silly things like navigating back and forth in the 
installer too

[19:56] <ompaul> <ted_> QUESTION: Is there any specific effort to test X configuration on 
different graphics cards / monitors, since that is often an area of difficulty for many 
users?

[19:56] <heno> to tease out bugs

[19:57] <heno> ted_: not since the laptop testing program some years ago. clearly something 
we should revisit

[19:57] <davmor2> ted_: I test on really hardware by choice.  This has a selection of 
different hardware in it.  The only thing I'm short of is Ati gfx chipset.

[19:57] <heno> we are planning to add hardware profiles to the test tracker so we can see 
what HW has been covered

[19:58] <ompaul> comment: (mine) the objective is to be a good human and try to create the 
innovative exception to the expected behaviour - would you both agree

[19:58] <ompaul> i.e. try to break it!

[19:58] <davmor2> Hell yes.  Then report the break :)

[19:59] <davmor2> Especially if it can be reproduced time and time again

[19:59] <ompaul> okay guys thanks for a great session

[20:00] <ompaul> davmor2, heno your mailing list and channel ?

[20:00] <davmor2> channel is #ubuntu-testing

[20:01] <davmor2> list is ubuntu-qa@lists.ubuntu.com