AutoServerTests

Revision 6 as of 2010-01-27 02:12:35

Clear message

Dev Week -- Automated server testing -- soren -- Tue, Jan 26

UTC

(03:00:21 PM) soren: Hi, everyone.
(03:00:21 PM) soren: Thanks for coming to my session on Automated Server Testing.
(03:00:33 PM) soren: So..
(03:00:41 PM) soren: In the server team, we've traditionally had a problem with collecting test results.
(03:01:02 PM) soren: (question in #ubuntu-classroom-chat, by the way. please put "QUESTION" so that I will spot them)
(03:01:14 PM) soren: This is because our target audience and most of our users are using Ubuntu on servers that are being used to service real users.
(03:01:30 PM) soren: Real users, as you are probably aware, depend on their servers to work.
(03:01:46 PM) soren: They need mail server to be up and delivering mail so that they can get their daily dosage of spam..
(03:02:06 PM) soren: They need their file server to be around so they can get access to their music and various pirated software..
(03:02:19 PM) soren: They need their proxy server to work so that they can log onto facebook..
(03:02:32 PM) soren: They need the LDAP server to work so that they can look up the phone number for the pizza guy..
(03:02:37 PM) soren: And other important things.
(03:02:41 PM) soren: You get the idea.
(03:02:48 PM) soren: If something should fail, it means pain and suffering for the poor sysadmin.
(03:02:57 PM) soren: Hence, sysadmins are very hesitant to upgrade anything before it's been through lots and lots of QA.
(03:03:08 PM) soren: However, unless /some/ of them /do/ upgrade, there's not going to be much QA work done.
(03:03:26 PM) soren: This places us in a rather unfortunate situation, where a significant portion of our bug reports don't come in until after release.
(03:03:48 PM) soren: Anyone involved in Ubuntu development will know that this is a hassle, since fixing things after release is much more tedious than before release, since we have much less freedom to make changes.
(03:04:03 PM) soren: This is very difficult to change, and I haven't come up with a golden solution.
(03:04:30 PM) soren: However, the sooner we catch problems, the more time we have to work on fun stuff since we'll be putting out less fires in the end.
(03:04:51 PM) soren: See, while we're cursed with a user base that doesn't start testing our product until it's essentially too late..
(03:04:54 PM) tedg left the room.
(03:05:09 PM) soren: ..we areblessed with a type of software that traditionally comes with a good test suite.
(03:05:26 PM) soren: MySQL for instance, comes with an extensive test suite.
(03:05:39 PM) soren: This test suite runs every time we upload a new version of mysql to Ubuntu.
(03:06:00 PM) soren: If the test suite fails, the build fails, and the uploader gets an e-mail.
(03:06:07 PM) soren: ...and it's all very obvious that something needs fixing.
(03:06:09 PM) soren: This is great.
(03:06:18 PM) soren: Well..
(03:06:21 PM) soren: Sort of.
(03:06:32 PM) soren: The thing is, every package in Ubuntu has dependencies of some sort.
(03:06:43 PM) soren: For instance, almost everything depends on libc
(03:07:00 PM) soren: This means that a change in libc will inevitably affect MySQL somehow.
(03:07:28 PM) soren: Luckily, if this causes problems, it is (hopefully) caught by MySQL's test suite.
(03:07:44 PM) soren: Less luckily, this test suite, as I just mentioned..
(03:07:49 PM) soren: is run when MySQL is uploaded..
(03:07:53 PM) soren: not when libc is uploaded.
(03:08:13 PM) soren: So we may not notice a problem until the next time someone uploads MySQL. This could be weeks or even months!
(03:08:38 PM) soren: And trying to narrow down the change that broke something is hard with all the stuff doing on in Ubuntu development over the course of months.
(03:08:42 PM) soren: So..
(03:09:03 PM) soren: to address this, we've set up and automated system that rebuilds MySQL ( and a bunch of other stuff) every night in a PPA.
(03:09:30 PM) soren: That way, if we trust the test suite, we can relax  and know that MySQL still works, despite any changes in its dependency chain.
(03:09:46 PM) soren: We do the same for libvirt, php5, postgresql, etc.
(03:10:09 PM) soren: Basically, anything that has a test suite that runs at build time and that causes the build to fail if it doesn't pass, should be added.
(03:10:23 PM) soren: This at least makes me sleep better :)
(03:11:01 PM) soren: So, the automated testing stuff in Lucid consists of two parts.
(03:11:10 PM) soren: The above is the first part, which is pretty nice.
(03:11:15 PM) soren: The second part is awesome:
(03:11:16 PM) soren: :)
(03:11:25 PM) soren: It's an automated ISO testing system.
(03:11:40 PM) soren: ISO testing is the thankless and tedious job of installing Ubuntu from an ISO over and over again..
(03:12:00 PM) soren: ..with small adjustmets each time to make sure things haven't changed unexpectedly.
(03:12:36 PM) soren: QUESTION: ~Shkodrani> why not run the test suite only when a packege on which, for instance MySQL relays on?
(03:13:03 PM) soren: The cost of checking whether something in MySQL's dependency chain has changed is rather high. At the very least, it's tedious.
(03:13:18 PM) soren: ..and just doing the rebuild is cheap and simple to get up and running.
(03:13:29 PM) soren: It's all run by a 10 line shell script or thereabouts.
(03:13:48 PM) soren: Ok, ISO testing..
(03:14:05 PM) soren: Every time we come close to an alpha, beta or any other kind of release..
(03:14:14 PM) soren: ..we all spend a lot of itme going through this install process.
(03:14:26 PM) soren: Well, we /should/ anyway. I positively suck at getting it done, but there you go.
(03:14:45 PM) soren: My fellow server team member, Mathias Gug, has had a preseed based setup running for a while now.
(03:15:01 PM) soren: Basically, preseeding is a way to answer all of the installer's questions up front.
(03:15:11 PM) soren: So, he takes all the answers..
(03:15:18 PM) soren: passes them to the install using clever hacks..
(03:15:31 PM) soren: ..and the install zips through the instlalation without bothering Mathias with questions.
(03:15:45 PM) soren: In the end, he can log into the installed system and run the las tparts of the test cases.
(03:16:07 PM) soren: This has served us well, and has probably saved us several man days (or weeks?) of testing tie over the last few years.
(03:16:18 PM) soren: However, it doesn't actually test the same things as the ISO test cases describe.
(03:16:33 PM) soren: The ISO test cases speak of the interaction between the user and the installer..
(03:16:46 PM) soren: However, the point of preseeding is to /avoid/ interaction, and to skip it entirely.
(03:16:54 PM) soren: Don't get me wrong..
(03:17:00 PM) soren: Preseed testing is super valuable.
(03:17:23 PM) soren: Installing that way is a supported install method, so having this well tested is wicked cool and really important.
(03:17:52 PM) soren: ...but I wanted to test the interactivity as well.
(03:18:07 PM) soren: So, being the virtualisation geek that I am..
(03:18:10 PM) soren: I decided to use the KVM autotest framework to do the ISO testing.
(03:18:18 PM) soren: Now, KVM autotest was designed to test KVM.
(03:19:02 PM) soren: KVM developers use it to install a bunch of different operating systems and test things to make sure they didn't change anything in KVM that broke functionality in one of the guest operating systems.
(03:19:16 PM) soren: What we want to do, though, is somewhat the opposite.
(03:19:45 PM) soren: We assume that KVM works and instead want to test the operating system.
(03:20:14 PM) soren: So, the KVM autotest framework works by runing a virtual machine..
(03:20:28 PM) soren: grabs a screenshot every second..
(03:20:47 PM) soren: ..and when the screenshot looks a particular way (e.g. when a particular dialog comes up),
(03:21:07 PM) soren: it can respond with a series of key presses or mouse events.
(03:21:22 PM) soren: This way, we can emulate a complete, interactive install session.
(03:21:27 PM) soren: Awesome stuff.
(03:21:53 PM) soren: I've started documenting this, but haven't gotten all that far, since I kept changing things faster than I could update the docs :)
(03:22:10 PM) soren: The documentation lives at  https://wiki.ubuntu.com/AutomatedISOTesting
(03:22:21 PM) soren: If you all open that page..
(03:22:38 PM) soren: ..and scroll down to the "step files" section..
(03:23:01 PM) soren: you can see a sample step from a "step file".
(03:23:17 PM) soren: A step file is a description of a test case.
(03:23:40 PM) soren: Now, looking at the sample, you can see a "step 9.45" and a "screendump" line.
(03:23:57 PM) soren: They're pretty much just meta-data for the creator or editor of the step file
(03:24:04 PM) soren: so don't worry about those.
(03:24:11 PM) soren: The important lines are the "barrier_2" and "key" ones.
(03:24:19 PM) soren: The barrier_2 line tells the testing system to wait..
(03:24:41 PM) soren: ..until the rectangle of size 117x34 of the screen, starting at 79x303..
(03:24:52 PM) soren: should have md5sum de7e18c10594ab288855a570dee7f159 within the next 47 seconds.
(03:25:04 PM) soren: If this doesn't happen, the test will fail, and a report will be generated.
(03:25:15 PM) soren: If it does pass, it goes on to the next step: "key ret"
(03:25:35 PM) soren: As you can probably guess, "key ret" sends a keypress to the guest, namely Return.
(03:26:07 PM) soren: The result of those two lines is: Wait for the language prompt right after boot to show up, and once it does, press return to accept the default "English".
(03:26:40 PM) soren: Now, pretty soon, it became obvious that there was going to be a lot of duplication involved here.
(03:26:56 PM) soren: ...all the installs would have to wait for that prompt and respond to it in the same way.
(03:27:20 PM) soren: Even worse: If that prompt were to change, /every/ step file would need to be updated.
(03:27:54 PM) soren: Even worse again: In the beginning there was no concept of "updating" step files. You had to start all over.
(03:28:08 PM) soren: Starting over makes plain old ISO testing feel like a fun time.
(03:28:08 PM) soren: It's not.
(03:28:09 PM) soren: Just so you know.
(03:28:21 PM) soren: I love people for doing it, but it's really not that much fun. :)
(03:28:45 PM) soren: Ok, so to address the mass duplication of steps and stuff, I added a step file generator.
(03:29:17 PM) soren: The step file generator generates a step file (you probably guessed this much) based on the task to be installed and the partitioning scheme to be used.
(03:30:05 PM) soren: This means that I can tell the test frame work: Hey, please test an install of the LAMP task, with LVM partitioning and do it on amd64.
(03:30:05 PM) soren: And it does so.
(03:30:12 PM) soren: See, this is all running in a virtual machines.
(03:30:16 PM) soren: Virtual machines are cool.
(03:30:21 PM) soren: So cool, in fact...
(03:30:33 PM) soren: That you can use them to make installer videos.
(03:30:55 PM) soren: So, to see what happens during a test run, you can attach a recorder thingie and turn the result into an avi.
(03:31:14 PM) soren: Now, like any decent TV chef, I've cheated and done this all in advance.
(03:31:38 PM) soren: Now, unlike most decent TV chef's, what I did in advance failed.
(03:31:58 PM) soren: And even more unlike TV chef's, I'm going to show it to you anyway, because it's useful.
(03:32:06 PM) soren: Without further ado:
(03:32:26 PM) soren: heh..
(03:32:29 PM) soren: wait for it..
(03:32:30 PM) soren: http://people.canonical.com/~soren/lamplvminstall.avi
(03:32:32 PM) soren: There we go.
(03:32:40 PM) soren: wget http://people.canonical.com/~soren/lamplvminstall.avi ; mplayer lamplvminstall.avi
(03:32:48 PM) soren: This test case failed.
(03:33:02 PM) soren: Somewhat surprisingly.
(03:33:11 PM) soren: If you fast forward all the way to the end..
(03:33:44 PM) soren: (watch the rest as well, it's fun to watch the test system typing the username "John W. Doe III" and the password and whatnot)
(03:34:01 PM) soren: ..at the end, you'll see if breaks off before the install actually finishes.
(03:34:06 PM) soren: Like... seconds before it would have finished.
(03:34:27 PM) soren: Honestly, I did not mean for this to happen, but it's a good learning experience :)
(03:34:32 PM) soren: Ok, if we all look at..
(03:34:41 PM) ***soren digs through launchpad, bear with me.
(03:34:57 PM) soren: http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/
(03:35:05 PM) soren: Those are the input files for the step file generator.
(03:35:47 PM) soren: Yes, they are poorly named, but please appreciate that just days ago, they were all named "foo", "bar", "wibble", "wobble", etc. so this is a massive improvement.
(03:36:12 PM) soren: QUESTION: That method could be used for UI testing in a *lot* of different GUI apps, not just ISO installations. Any plans to  document/release it more generally?
(03:36:16 PM) soren: (from rmunn)
(03:36:18 PM) soren: Yes1
(03:36:19 PM) soren: !
(03:36:35 PM) soren: I meant to get that done for today, but the real world imposed and made a mockery of my plans.
(03:36:46 PM) soren: This can totally be used to do GUI installs as well.
(03:37:06 PM) soren: Looking at http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/ again..
(03:37:19 PM) soren: Specifically, 060-finish_install_and_reboot.steps
(03:37:27 PM) soren: http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/annotate/head:/client/tests/kvm/generator_data/lucid/060-finish_install_and_reboot.steps
(03:38:03 PM) soren: This is the step that failed.
(03:38:22 PM) soren: For some reason (that I have yet to figure out, I only spotted this failure an hour ago) this times out.
(03:38:42 PM) soren: It says 579, but perhaps those a special kind of seconds that are not as long as most people's seconds.
(03:39:11 PM) soren: The point is this: I only have to change the timeout in this one place, and all the test cases will be updated.
(03:39:27 PM) soren: < ~rmunn> QUESTION: I see a lot of keystrokes used to select various dialog widgets. Can the KVM testing system simulate mouse clicks and/or mouse
(03:39:31 PM) soren:                    movements (e.g., for testing mouseover stuff) as well?
(03:39:36 PM) soren: cut'n'paste for the lose :(
(03:39:38 PM) soren: Well..
(03:39:39 PM) soren: Yes.
(03:39:42 PM) soren: Sort of :)
(03:40:03 PM) soren: The autotest framework supports it, I've added support for it to the frontend, but kvm has an.. um.. issue :)
(03:40:30 PM) soren: It used to emulate a mouse, so it would move the cursor relative to the current position.
(03:40:43 PM) soren: However, these days, GNOME and such give you...
(03:40:48 PM) soren: mouse acceleration!
(03:40:49 PM) soren: Yay!
(03:40:52 PM) soren: No. Not yay.
(03:41:26 PM) soren: Mouse acceleration is the enemy when you're actually warping the mouse from one place to another, because it thinks you just moved your mouse /really/ fast, and then moves it even further than you wanted it to.
(03:41:31 PM) soren: This took me /forever/ to realise.
(03:41:37 PM) soren: So, I've made it pretend to use a tablet.
(03:41:45 PM) soren: Tablets offer absolute positioning, so this helped a lot.
(03:42:27 PM) soren: However, the command to tell kvm to click on something internally translated into "mouse_event(click_button1, 0, 0, 0)", where 0,0,0 are the coordinates.
(03:42:38 PM) soren: Now, if you're in relative positioning mode (using a regular mouse), this is good.
(03:42:42 PM) soren: You want to click right where you are.
(03:42:54 PM) soren: ..if you're using a tablet, it means you can only click in the top left corner.
(03:42:57 PM) soren: No fun.
(03:43:24 PM) soren: I wrote a patch for that, but I'm not sure it's in upstream KVM yet, but it'll be in Lucid half an hour after I start working on those GUI test cases :)
(03:44:08 PM) soren: So, yes, GUI testing is totally an optoin.
(03:44:11 PM) soren: option, too.
(03:44:43 PM) soren: Another problem I had with this is that it was designed to test a variable kvm against a static set of OS's.
(03:45:04 PM) soren: The OS's should look and act the same regardless of what changed in KVM. That is the whole point of these tests: To make sure they don't change.
(03:45:19 PM) soren: However, we change the OS all the time. That's what we do :)
(03:46:07 PM) soren: ..but since the designers of this test system never meant for it to be used this way, they didn't add an option to edit these step files very conveniently.
(03:46:22 PM) soren: To fix this, I've added an option to the test system to fall back to the stepmaker (the GUI used to create step files) if a test fails.
(03:46:48 PM) soren: This is great if you're running tests on your laptop or a machine you have direct access to rather than a machine running in a dusty corner of a data center.
(03:47:27 PM) soren: It really comes in useful when the screens change (wording changes, extra/fewer dialogs, change of theme (in the GUI)).
(03:47:35 PM) soren: Having to start over is, as I mentioned, no fun at all.
(03:48:26 PM) soren: Please shoot any questions you may have. I haven't really prepared much more than this.
(03:48:38 PM) soren: Still, questions belong in #ubuntu-classroom-chat
(03:50:07 PM) soren: If there are no more questions, I'll sing for the rest of the time slot.
(03:50:34 PM) soren: 21:50:25 < ~Omahn23> soren: QUESTION As an end user/sysadmin, is there anything I can do to help in testing with this new framework?
(03:51:07 PM) soren: Well, seeing as these things run in virtual machines, running them in more places is not going to make much difference, so /running/ the tests is probably not something we need help with
(03:51:11 PM) soren: However!
(03:51:20 PM) soren: The more test cases we can include, the better.
(03:51:25 PM) soren: The more, the merrier.
(03:51:49 PM) soren: I'd love to have more test cases to include in our daily runs of this system.
(03:52:15 PM) soren: 21:52:01 < hggdh> QUESTION: so we can automate pseudo-interactive testing. How to deal with the tests that require meat between the keyboard and the chair?
(03:52:18 PM) soren: Examples?
(03:52:31 PM) soren: 21:50:12 < Ramonster> soren: Any thoughts on testing servers while they actually perform one of the roles you talked about at the start ?
(03:52:44 PM) soren: Ramonster: You mean functional testing of e.g. a LAMP server?
(03:53:01 PM) soren: 21:52:23 < mscahill> QUESTION: you briefly mentioned PPA testing. what packages are included in this testing?
(03:53:04 PM) ***soren looks that up.
(03:53:26 PM) soren: 21:53:08 < Ramonster> soren: Yeah
(03:53:29 PM) soren: alright.
(03:53:43 PM) soren: Um, yes, but it's not part of this work I've been doing.
(03:53:49 PM) soren: We're not very strong in that area at all.
(03:53:53 PM) soren: ...and that's a shame.
(03:54:13 PM) soren: PKGS="libvirt postgresql-8.3 postgresql-8.4 mysql-dfsg-5.0 mysql-dfsg-5.1 openldap php5 python2.6 atlas"
(03:54:20 PM) soren: Is the list of packages built daily.
(03:55:10 PM) soren: Well, the security team has a bunch of tests they run whenever they change anything. They often can't rely on anyone else to test anything (since they don't go through -proposed), so they need to be really thorough.
(03:55:34 PM) soren: I'm working on getting those run every day as well. They should provide some amount of functional testing.
(03:55:46 PM) soren: 21:53:12 < yltsrc> QUESTION: is writing test cases required for bugfixing?
(03:55:58 PM) soren: Not per se.
(03:56:40 PM) soren: Most test cases need updating once a bug is fixed, and most things I can think of would be covered by this, so new test cases (for this system, I mean) wouldn't be a requirement for bug fixes.
(03:57:18 PM) soren: 21:54:55 < mscahill> QUESTION: are there plans to allow automated testing for package maintainers with their own PPA?
(03:57:28 PM) soren: Sure, anyone is free to run that script and do their own testing.
(03:57:34 PM) soren: Hm... I may not have published it anywhere.
(03:57:37 PM) ***soren fixes that.
(03:57:47 PM) soren: Well, /me makes a note to fix that
(03:58:48 PM) soren: I do have a few ideas for doing functional testing of upgrades of various sort, but most of those ideas are only a few hours old, so they're not even half baked yet :)
(03:59:16 PM) soren: Did I miss any questions?
(03:59:43 PM) soren: 21:59:33 < Ramonster> soren: That's the problem atm everyone is walking around with these half-baked ideas :)
(03:59:50 PM) soren: Whoops, didn't mean to post that here :)
(04:00:40 PM) soren: Thanks for showing up, everyone.
(04:00:45 PM) soren: that's it!