Ubuntu Wiki

Ubuntu Open Week - Automated testing - Lars Wirzenius & Henrik Omma - Tue , Oct 23, 2007

17:06 < liw> I'm here
17:06 < liw> I'm Lars :)
17:06 -!- mode/#ubuntu-classroom [+v liw] by PriceChild, ChanServ
17:07 -!- mode/#ubuntu-classroom [-v dholbach] by PriceChild
17:08 -!- mode/#ubuntu-classroom [+v heno] by ChanServ
17:08 <+liw> shall we start?
17:08 -!- mode/#ubuntu-classroom [+o LjL] by ChanServ
17:08 <+heno> welcome to a session on automated testing
17:08 -!- LjL changed the topic of #ubuntu-classroom to: Ubuntu Open Week info: Information and Logs: https://wiki.ubuntu.com/UbuntuOpenWeek | Ubuntu classroom transcripts: https://wiki.ubuntu.com/ClassroomTranscripts | Please ask questions in #ubuntu-classroom-chat not here | Current: Automated testing - Lars Wirzenius & Henrik Omma
17:09 -!- mode/#ubuntu-classroom [-o LjL] by LjL
17:09 <+heno> liw: please go ahead
17:09 <+liw> all right, I've not done one of these before, so please tell me if I'm doing something wrong
17:09 < hydrogen> saying that is doing something wrong ;p
17:10  * hydrogen be's quiet
17:10 <+liw> We intend to automate some of the quality assurance work for Hardy. That's because testing everything manually multiple times is very tedious and uses up a lot of energy that could be used for other things. It also slows down development and bug fixing, since the feedback that something works or doesn't work can take weeks or months
17:10 <+liw> There are several parts for this:
17:10 <+liw> * installation testing of ISO images, using pre-seeding
17:10 <+liw> * upgrade tests of installed systems, using piuparts and vlosuts and other tools
17:11 <+liw> * tests if installed systems, using autopkgtest, and also of GUI programs probably using Accerciser and Macaroon
17:11 <+liw> More stuff may come, later, once we've gotten this to work.
17:11 <+liw> Some of that has been going on already, but not terribly well organized
17:11 <+liw> Once the initial infrastructure starts working, we hope to involve the entire Ubuntu development community in defining and implementing tests.
17:12 <+liw> that's my introduction -- it's a lot of things quickly, and a lot of details are missing, but I'm now open for questions
17:12 <+heno> with this we hope to find problems early so that bugs can be worked on as they appear
17:13 <+liw> heno, if you have things to say, now's the time
17:13 <+heno> I'll just give the link to the testing pages https://wiki.ubuntu.com/Testing
17:14 <+liw> yes, definitely find bugs as early as possible, and possibly before they're uploaded, even
17:14 <+liw> I have some links, too, that might be useful: https://wiki.ubuntu.com/QATeam
17:14 <+liw> http://alioth.debian.org/projects/piuparts/
17:14 <+liw> http://people.redhat.com/zcerza/dogtail/
17:14 <+heno> liw: <ttread_3> QUESTION: Can you explain what 'pre-seeding' is?
17:14 <+liw> http://live.gnome.org/Accerciser
17:14 <+liw> http://monotonous.org/2007/10/18/making-freedom-accessible/
17:15 <+liw> pre-seeding is the procedure where the installer (the text-mode one or the gui one, they use the same backend) is given the answers a user would normally give from the keyboard/mouse so that they're read from a file instead
17:15 <+liw> this way the entire installation process can be automated
17:16 <+heno> by using that we can test all the installer backend stuff quite easily on a range of hardware
17:16 <+liw> I don't seem to have a handy link to explain that in detail right now
17:17 <+heno> but we don't test the clicking of buttons in the installer gui
17:17 <+liw> heno, right, and also test the same things repeatedly -- the same tests every time there's a new ISO
17:17 <+heno> that will come later using the ATK framework and tools like dogtail
17:17 <+heno> liw: <hydrogen> QUESTION: could you elaborate a bit on how you plan to do this feasibly?  especially wrt hardware testing
17:18 <+liw> the gui installer will basically be installed in two halves: the backend and actual installation stuff with pre-seeding, and the user interface with dogtail/accerciser/macaroon
17:18 <+liw> ah yes, hardware testing
17:19 <+heno> (as you can tell we are still evaluating which tools to use)
17:19 <+liw> I plan on doing the bulk of the ISO testing in emulators, so not with real hardware
17:19 <+liw> that of course doens't test anything that depends on actual hardware
17:19 <+liw> so for actual hardware testing we need people to still do installation testing on real hardware
17:20 <+liw> but I hope to make that partially automated as well
17:20 < hydrogen> do you have any ideas (or are you working on any) to increase the range of testers?
17:20 <@PriceChild> hydrogen, questions in #ubuntu-classroom-chat please
17:20 <+liw> I'll answer hydrogen's question now, but please relay questions via -chat in the future
17:21 <+liw> I don't have concrete plans for increasing the range of testers, except that I want to make testing more fun
17:21 <+liw> I did quite a number of test installations of gutsy beta and rc and final isos, and it was quite boring, so I want to make it a more pleasant experience
17:22 <+heno> We will also make test reporting easier with the QA website https://qa.stgraber.org/
17:22 <+liw> and by fun I mean less tedious -- I don't intend to add nethack to the ISOs for testers to play during the test
17:23 <+heno> we can semi-automate some of the testing and reporting; sun a single command to test openoffice and upload the results, say
17:23 <+heno> run even
17:23 <+liw> less tedious, more automated --- and perhaps more communal: get more people interact with each other during the test phases, perhaps
17:23 <+liw> next question, please
17:24 < LjL> <begert__> QUESTION: are there plans for releasing a "test kit" that users can run so that results can be gathered on a wide variety of system configurations?
17:24 <+liw> yes, something like that will be needed
17:24 <+heno> We'd like to do that, but don't have any detailed plans
17:25 <@PriceChild> <scorpioxy> QUESTION: how do you plan on testing problems with X and the graphics drivers. Especially problems such as black screen on bootup, wrong refresh rates and so on
17:25 <+heno> it would be good to combine it with the hw-test program
17:25 <+liw> something I'd like to explore is a "tester live cd", which allows you to test lots of things (automatically), without requiring you to touch the hard disk
17:25 <@PriceChild> gah
17:25 < begert__> thank you
17:26 <+liw> I'm done with begert's question, so unless heno has something to say, I'll continue with scorpioxy's
17:26 <+heno> On the last question: that will have to be done wit real HW
17:26 <+heno> though we can add some automation
17:26 <+liw> testing problems with X and graphics drivers (and wireless etc) does need to happen on real hardware... what heno said
17:27 <+liw> and that's one part especially where Ubuntu needs lots of volunteers, and where we need to make it especially easy and fun to participate
17:27 <+liw> so that people with a free evening and a computer can do something real and useful, preferably without having to re-install their system afterwards
17:28 <+liw> next question
17:28 <@PriceChild> <seer-as-shubhu> QUESTION: once you do testing on emulators why is there need for hardware testing?
17:28 <+heno> we also need to keep track of what hardware is being used for the testing
17:28 <+heno> answered by the previous question :)
17:28 <+liw> emulators don't emulate all hardware, and what they do emulate, they don't emulate perfectly, so testing on real hardware is still reqwuired
17:28 <+liw> next question
17:28 <@PriceChild> <ttread_3> QUESTION: Continuing on the pre-seeding, is it in the form of a special ISO that boots to the live CD and then directly to installation with programmed responses?  Or some other way external to the ISO?
17:29 <+liw> that's one of the details I don't know about in Ubuntu -- in Debian it's part of the standard installer
17:29 <+liw> you give a special boot command line argument (but the Ubuntu gui installer can ask for a pre-seed url)
17:29 <+liw> if we do make a tester live cd, we can make that always ask for a pre-seed, of course
17:30 <+liw> but basically the pre-seed file is (or can be) external to the iso
17:30 <+liw> next question
17:30 <@PriceChild> <dorto> QUESTION: Which is the most basic automated test method that could be used to test Ubuntu Hardy Tribe/milestone releases?
17:30 <+heno> using a custom ISO would defeat the purpose of testing candidate ISOs, so we won't be doing that
17:31 <+liw> heno, right, but a custom ISO might be useful for things like hardware testing, to verify that bugs in drivers have indeed been fixed; but this is all up in the air for now
17:31 <+heno> right, up for discussion at UDS
17:31 <+liw> the most basic automated test method would be to use pre-seeding to drive an installation onto a computer, or one of the upgrade testing tools to test upgrades from gutsy to hardy
17:32 <+liw> that much will certainly be doable fairly quickly after UDS and other meetings are over
17:32 <+liw> things like automated testing of gui programs using the ATK layer are going to require a bit more time and effort to set up
17:32 <+liw> next question
17:33 <@PriceChild> <radoe> QUESTION: pre-seeding is just one part to test if the installer just works, any plans for a automated test if it did its job right?
17:33 <+liw> part of the testing is validating that the result is correct, that is a very good point
17:34 <+liw> I haven't yet thought much about how we'll do that on real hardware, in emulators it will be done using the test harness
17:35 < Myrtti> liw: _o/
17:35 <+liw> next question
17:36 <@PriceChild> <giftnudel> QUESTION: Will there be an easy way for users to write their own tests? Like a easy-to-use test framework?
17:36 <+liw> that is the goal
17:36 <+liw> in fact I would like to see all tests written by users :)
17:36 < giftnudel> good!
17:37 <+heno> accerciser and macaroon are tools that help you record tests
17:37 <+heno> they may need some adjusting after that though
17:37 <+liw> for the gui testing, for example, we have a vision of a "recording tool", where the user can make a new test by pressing a "start recording" button, then do some things, and then press "end recording", and put the resulting test script onto a website or something
17:37 <+heno> (it's all python scripts)
17:38 < giftnudel> that answers the question, thanks.
17:38 <+liw> in similar ways for, say, server testing: someone will need to write tests (perhaps simple shell or python scripts) to test that a web server works before and after, that a database still returns the right data after an upgrade, and so on
17:38 <+liw> next question then, please
17:39 < LjL> <seer-as-shubhu> QUESTION:liw said "testing problems with X and graphics drivers (and wireless etc) does need to happen on real hardware"  but isn't hardware is required for testing ? then how do you catgorize which needs hardware and which doesnt?
17:39 <+liw> basically for each test you have to make a decision about whether it needs real hardware or whether it can be run in an emulator
17:39 -!- mode/#ubuntu-classroom [+o LjL] by ChanServ
17:40 <+liw> that needs to be an informed decision by a human, I'm afraid, perhaps aided by some automated heuristics
17:41 <@LjL> <Rudd-X> QUESTION: in case of catastrophic test failures on the (admittedly good) scenario of a live CD with testing, how will you detect a test that failed midway?  May I suggest that you use the swap partitions on the hard disk if one is available?
17:41 <+liw> I'd like to see as much as possible be testable in emulators, since emulator testing can be automated, and automation is not just fun and pleasant, it also gives faster turnaround time and thus speeds up development :)
17:41 <+liw> I don't yet know the answer to that question, sorry
17:41 <+liw> the suggestion of using swap partitions is a good one
17:42 <+liw> next question
17:42 <@LjL> <LjL> QUESTION: Do you have any plans to automate testing of successful installation and removal of main/universe packages?  I understand that testing for all possible 'postinst' failure scenarios (say) is likely NP-complete, but perhaps you have any tricks in mind to test in the most common scenarios?
17:43 <+liw> the piuparts tool tests installation, upgrading, and removal of packages
17:43 <+liw> if I have time, I intend to test every package in Ubuntu with piuparts
17:43 <+heno> *cough* which liw wrote *cough*
17:43 <+liw> both on bare-bones systems, and in a few carefully chosen representative systems
17:44 < savvas> did they say anything about graphics card / sound cards testing?
17:44 <+liw> next question
17:44 <@LjL> liw: there is none at the moment
17:45 <@LjL> <radoe> QUESTION: Wouldn't it be nice to have the upcoming test framework as an independent (yet optional) part of final releases? So one can write tests and verify large rollouts quickly. Not sure if this is the right thing for GUI tasks, but all "scriptable" tests would be nice.
17:45 <+liw> that would definitely be nice
17:45 <+liw> no, I take that back
17:45 <+liw> it would be AWESOME and IT WOULD ROCK THE WORLD
17:45 <+liw> but the testing stuff takes space, and the CD is pretty full already
17:46 <+heno> yep, it's clearly the way to go
17:46 <+heno> most of the framework is on the CD already
17:46 <+heno> including ATK and python
17:46 <+heno> just need to add some scripts
17:46 <+liw> right, so hopefully we can fit all the testing stuff by only removing one background image
17:47 <+liw> or something like that O:-)
17:47 <@LjL> <Rudd-X> QUESTION: what's the projected/expected/target size of the testing payload?
17:47 <+heno> which is in itself highly controversial, just ask kwwii :)
17:47 <+liw> I have as yet no idea how the testing payload will be
17:48 <+liw> for example, one Accercises/dogtail script is pretty small, but if have/get good coverage, there's going to be lots and lots of them
17:48 <+liw> and some of them may need helper files, such as data for the test
17:48 <+heno> and we may need sample test data like PDFs and mp3 files
17:49 <+heno> though they could be downloaded during the test
17:49 <+liw> obviously we should use background images as test data...
17:49 <+liw> next question?
17:50 <@LjL> <chrisle> QUESTION: Is there a tool that checks for correct desktop file while building packages (for gnome and kde menus)?
17:50 <+liw> I don't know of such a tool -- what would it check for?
17:51 < chrisle> to create one in the correct place
17:51 < Rudd-X> desktop file correctness?
17:52 <+liw> chrisle, I think the lintian and linda tools might do that, and if not, writing new tests for them to do that seems like a reasonable thing to do
17:52 <+liw> lintian and linda are "static checkers", meaning they check the package without running any files, which is fast and secure
17:52 <+liw> next question
17:52 <@LjL> <desertc> QUESTION: Is there much collaboration between other Linux distributions, as you develop your automated tests?
17:53 < chrisle> yes because a lot debs are only rebuild because of missing desktop files
17:53 <+liw> I don't think there's much collaboration right now, but it is our intention to contribute any tests specific to a program to upstream
17:53 <+liw> and I'd like to see Debian share any installation and other tests that we make for Ubuntu, to the extent they're suitable for Debian
17:54 <+liw> definitely most of the testing infrastructure will be useful for Debian, and perhaps other Linux distros
17:54 <+heno> I have spoken with Redhat and Novell people about this, but no concrete work has been done yet
17:54 <+heno> Sun and Mozilla are also doing good work on automated testing
17:55 <+liw> aye, as distros and individual software projects grow in size, everyone is seeing the benefits of automated testing
17:55  * liw thinks automated testing is luverly
17:55 <+liw> next question?
17:55 <@LjL> <savvas> QUESTION: What are the plans for the next (hardy) long term support release? what are you going to test mostly of hardware?
17:56 <+liw> we don't have very concrete plans yet, I think, those will be formulated during UDS
17:56 <+liw> that is, in the next couple of weeks
17:56 <@LjL> <radoe> QUESTION: blueprint planning a testing framework has yet to be started?
17:57 <+liw> I think that's correct, it's still waiting for UDS to be started; heno, is that right?
17:57 <+heno> right
17:58 <+heno> e.g. https://blueprints.launchpad.net/ubuntu/+spec/desktop-automated-tests
17:58 <@LjL> <ian_brasil> Question: how much are you integrating autotest into the Ubuntu environment and do you have any plans to write a web front end to this if so?
17:59 < rrittenhouse> yea
17:59 <+liw> all test results should be reported on public web sites, and for things that are clearly bugs, reports should be filed on launchpad, and the qa.stgraber.org site will be extended to co-ordinate testing
18:00 <+liw> exactly how depends on what turns out to be useful, I expect
18:00 <+liw> next question
18:00 <@LjL> <nealmcb> QUESTION: Interoperation with Active Directory infrastructures is important for our users.  But many testers don't have access to an AD site.  Also, users may want to try out an AD howto on a known-good configuration if they are having problems with their local AD.  Are there any AD test sites with public ids and passwords that testers and/or users can test against?
18:01 <+liw> I don't know anything about Active Directory, so I must answer "I don't know", sorry
18:01 <+liw> if such things are technically feasible, and useful, then perhaps they can be arranged
18:01 <+liw> next question
18:01 <@LjL> <ttread_3> QUESTION: Is automated testing being developed on all the desktops - Kubuntu and Xubuntu included?
18:01 <+heno> there is a server team spec to improve compatability with Windows
18:02 <+liw> the ATK framework is specific to GTK, and if Xubuntu includes it, then the testing frameworks should work on that, too
18:02 <+heno> Kubuntu is tricky because it lacks a working access framwork like ATK
18:03 <+liw> it might not be possible to support Kubuntu with this, directly, but perhaps KDE will get a similar framework later, and then we'll extend the testing framework to support KDE as well
18:03 <+heno> most gtk apps should work (most of Xubuntu)
18:03 <@LjL> <desertc> QUESTION: Are hardware vendors included in the "big picture"?  Vendors like System76 and, of course, Dell provide an enormous hardware base with standard configurations.  Are there special ways they can get involved with the process of automated testing of future releases, since they have so much at stake with their customers?
18:03 <+heno> as indeed promised for KDE4
18:04 <+heno> we have a hw certification program for such partners
18:04 <+liw> heno, we do? oh dear, I was hoping I could convince them to send me one of each of their hardware models :)
18:04 <+heno> where we would perform testing for them and commit to fixing problems
18:05 <+liw> that sounds very good to me
18:05 <+heno> liw: you can still try :)
18:05 <+liw> I think we're running out of time in about 100 seconds
18:05 <@LjL> <radoe> QUESTION: how will the development of all this be coordinated?
18:05 <+heno> on #ubuntu-testing in weekly QA team meetings
18:06 <+heno> we're also setting up a mailing list
18:06 <+liw> definitely in public, with full opportunity for anyone interested in joining
18:06 <+heno> indeed
18:07 <+heno> ok, thanks everyone!
18:07 <+heno> great questions!
18:07 <@LjL> thank you heno and liw
18:07 -!- LjL changed the topic of #ubuntu-classroom to: Ubuntu Open Week info: Information and Logs: https://wiki.ubuntu.com/UbuntuOpenWeek | Ubuntu classroom transcripts: https://wiki.ubuntu.com/ClassroomTranscripts | Please ask questions in #ubuntu-classroom-chat not here | Next session: Launchpad Q&A - Kiko
18:07 <+liw> thanks indeed -- it's nice to know I'm not the only one interested in testing the heck out of Ubuntu before the next release
18:07 < wnorrix> i am
18:07 < wnorrix> :)