AutoServerTests

Differences between revisions 3 and 4
Revision 3 as of 2010-01-26 20:21:10
Size: 7997
Editor: pool-71-123-25-78
Comment:
Revision 4 as of 2010-01-26 20:31:49
Size: 11429
Editor: pool-71-123-25-78
Comment:
Deletions are marked like this. Additions are marked like this.
Line 88: Line 88:
(03:21:07 PM) soren: it can respond with a series of key presses or mouse events.
(03:21:22 PM) soren: This way, we can emulate a complete, interactive install session.
(03:21:27 PM) soren: Awesome stuff.
(03:21:53 PM) soren: I've started documenting this, but haven't gotten all that far, since I kept changing things faster than I could update the docs :)
(03:22:10 PM) soren: The documentation lives at https://wiki.ubuntu.com/AutomatedISOTesting
(03:22:21 PM) soren: If you all open that page..
(03:22:38 PM) soren: ..and scroll down to the "step files" section..
(03:23:01 PM) soren: you can see a sample step from a "step file".
(03:23:17 PM) soren: A step file is a description of a test case.
(03:23:40 PM) soren: Now, looking at the sample, you can see a "step 9.45" and a "screendump" line.
(03:23:57 PM) soren: They're pretty much just meta-data for the creator or editor of the step file
(03:24:04 PM) soren: so don't worry about those.
(03:24:11 PM) soren: The important lines are the "barrier_2" and "key" ones.
(03:24:19 PM) soren: The barrier_2 line tells the testing system to wait..
(03:24:41 PM) soren: ..until the rectangle of size 117x34 of the screen, starting at 79x303..
(03:24:52 PM) soren: should have md5sum de7e18c10594ab288855a570dee7f159 within the next 47 seconds.
(03:25:04 PM) soren: If this doesn't happen, the test will fail, and a report will be generated.
(03:25:15 PM) soren: If it does pass, it goes on to the next step: "key ret"
(03:25:35 PM) soren: As you can probably guess, "key ret" sends a keypress to the guest, namely Return.
(03:26:07 PM) soren: The result of those two lines is: Wait for the language prompt right after boot to show up, and once it does, press return to accept the default "English".
(03:26:40 PM) soren: Now, pretty soon, it became obvious that there was going to be a lot of duplication involved here.
(03:26:56 PM) soren: ...all the installs would have to wait for that prompt and respond to it in the same way.
(03:27:20 PM) soren: Even worse: If that prompt were to change, /every/ step file would need to be updated.
(03:27:54 PM) soren: Even worse again: In the beginning there was no concept of "updating" step files. You had to start all over.
(03:28:08 PM) soren: Starting over makes plain old ISO testing feel like a fun time.
(03:28:08 PM) soren: It's not.
(03:28:09 PM) soren: Just so you know.
(03:28:21 PM) soren: I love people for doing it, but it's really not that much fun. :)
(03:28:45 PM) soren: Ok, so to address the mass duplication of steps and stuff, I added a step file generator.
(03:29:17 PM) soren: The step file generator generates a step file (you probably guessed this much) based on the task to be installed and the partitioning scheme to be used.
(03:30:05 PM) soren: This means that I can tell the test frame work: Hey, please test an install of the LAMP task, with LVM partitioning and do it on amd64.
(03:30:05 PM) soren: And it does so.
(03:30:12 PM) soren: See, this is all running in a virtual machines.
(03:30:16 PM) soren: Virtual machines are cool.
(03:30:21 PM) soren: So cool, in fact...
(03:30:33 PM) soren: That you can use them to make installer videos.
(03:30:55 PM) soren: So, to see what happens during a test run, you can attach a recorder thingie and turn the result into an avi.
(03:31:14 PM) soren: Now, like any decent TV chef, I've cheated and done this all in advance.

Dev Week -- Automated server testing -- soren -- Tue, Jan 26

UTC

(03:00:21 PM) soren: Hi, everyone.
(03:00:21 PM) soren: Thanks for coming to my session on Automated Server Testing.
(03:00:33 PM) soren: So..
(03:00:41 PM) soren: In the server team, we've traditionally had a problem with collecting test results.
(03:01:02 PM) soren: (question in #ubuntu-classroom-chat, by the way. please put "QUESTION" so that I will spot them)
(03:01:14 PM) soren: This is because our target audience and most of our users are using Ubuntu on servers that are being used to service real users.
(03:01:30 PM) soren: Real users, as you are probably aware, depend on their servers to work.
(03:01:46 PM) soren: They need mail server to be up and delivering mail so that they can get their daily dosage of spam..
(03:02:06 PM) soren: They need their file server to be around so they can get access to their music and various pirated software..
(03:02:19 PM) soren: They need their proxy server to work so that they can log onto facebook..
(03:02:32 PM) soren: They need the LDAP server to work so that they can look up the phone number for the pizza guy..
(03:02:37 PM) soren: And other important things.
(03:02:41 PM) soren: You get the idea.
(03:02:48 PM) soren: If something should fail, it means pain and suffering for the poor sysadmin.
(03:02:57 PM) soren: Hence, sysadmins are very hesitant to upgrade anything before it's been through lots and lots of QA.
(03:03:08 PM) soren: However, unless /some/ of them /do/ upgrade, there's not going to be much QA work done.
(03:03:26 PM) soren: This places us in a rather unfortunate situation, where a significant portion of our bug reports don't come in until after release.
(03:03:48 PM) soren: Anyone involved in Ubuntu development will know that this is a hassle, since fixing things after release is much more tedious than before release, since we have much less freedom to make changes.
(03:04:03 PM) soren: This is very difficult to change, and I haven't come up with a golden solution.
(03:04:30 PM) soren: However, the sooner we catch problems, the more time we have to work on fun stuff since we'll be putting out less fires in the end.
(03:04:51 PM) soren: See, while we're cursed with a user base that doesn't start testing our product until it's essentially too late..
(03:04:54 PM) tedg left the room.
(03:05:09 PM) soren: ..we areblessed with a type of software that traditionally comes with a good test suite.
(03:05:26 PM) soren: MySQL for instance, comes with an extensive test suite.
(03:05:39 PM) soren: This test suite runs every time we upload a new version of mysql to Ubuntu.
(03:06:00 PM) soren: If the test suite fails, the build fails, and the uploader gets an e-mail.
(03:06:07 PM) soren: ...and it's all very obvious that something needs fixing.
(03:06:09 PM) soren: This is great.
(03:06:18 PM) soren: Well..
(03:06:21 PM) soren: Sort of.
(03:06:32 PM) soren: The thing is, every package in Ubuntu has dependencies of some sort.
(03:06:43 PM) soren: For instance, almost everything depends on libc
(03:07:00 PM) soren: This means that a change in libc will inevitably affect MySQL somehow.
(03:07:28 PM) soren: Luckily, if this causes problems, it is (hopefully) caught by MySQL's test suite.
(03:07:44 PM) soren: Less luckily, this test suite, as I just mentioned..
(03:07:49 PM) soren: is run when MySQL is uploaded..
(03:07:53 PM) soren: not when libc is uploaded.
(03:08:13 PM) soren: So we may not notice a problem until the next time someone uploads MySQL. This could be weeks or even months!
(03:08:38 PM) soren: And trying to narrow down the change that broke something is hard with all the stuff doing on in Ubuntu development over the course of months.
(03:08:42 PM) soren: So..
(03:09:03 PM) soren: to address this, we've set up and automated system that rebuilds MySQL ( and a bunch of other stuff) every night in a PPA.
(03:09:30 PM) soren: That way, if we trust the test suite, we can relax  and know that MySQL still works, despite any changes in its dependency chain.
(03:09:46 PM) soren: We do the same for libvirt, php5, postgresql, etc.
(03:10:09 PM) soren: Basically, anything that has a test suite that runs at build time and that causes the build to fail if it doesn't pass, should be added.
(03:10:23 PM) soren: This at least makes me sleep better :)
(03:11:01 PM) soren: So, the automated testing stuff in Lucid consists of two parts.
(03:11:10 PM) soren: The above is the first part, which is pretty nice.
(03:11:15 PM) soren: The second part is awesome:
(03:11:16 PM) soren: :)
(03:11:25 PM) soren: It's an automated ISO testing system.
(03:11:40 PM) soren: ISO testing is the thankless and tedious job of installing Ubuntu from an ISO over and over again..
(03:12:00 PM) soren: ..with small adjustmets each time to make sure things haven't changed unexpectedly.
(03:12:36 PM) soren: QUESTION: ~Shkodrani> why not run the test suite only when a packege on which, for instance MySQL relays on?
(03:13:03 PM) soren: The cost of checking whether something in MySQL's dependency chain has changed is rather high. At the very least, it's tedious.
(03:13:18 PM) soren: ..and just doing the rebuild is cheap and simple to get up and running.
(03:13:29 PM) soren: It's all run by a 10 line shell script or thereabouts.
(03:13:48 PM) soren: Ok, ISO testing..
(03:14:05 PM) soren: Every time we come close to an alpha, beta or any other kind of release..
(03:14:14 PM) soren: ..we all spend a lot of itme going through this install process.
(03:14:26 PM) soren: Well, we /should/ anyway. I positively suck at getting it done, but there you go.
(03:14:45 PM) soren: My fellow server team member, Mathias Gug, has had a preseed based setup running for a while now.
(03:15:01 PM) soren: Basically, preseeding is a way to answer all of the installer's questions up front.
(03:15:11 PM) soren: So, he takes all the answers..
(03:15:18 PM) soren: passes them to the install using clever hacks..
(03:15:31 PM) soren: ..and the install zips through the instlalation without bothering Mathias with questions.
(03:15:45 PM) soren: In the end, he can log into the installed system and run the las tparts of the test cases.
(03:16:07 PM) soren: This has served us well, and has probably saved us several man days (or weeks?) of testing tie over the last few years.
(03:16:18 PM) soren: However, it doesn't actually test the same things as the ISO test cases describe.
(03:16:33 PM) soren: The ISO test cases speak of the interaction between the user and the installer..
(03:16:46 PM) soren: However, the point of preseeding is to /avoid/ interaction, and to skip it entirely.
(03:16:54 PM) soren: Don't get me wrong..
(03:17:00 PM) soren: Preseed testing is super valuable.
(03:17:23 PM) soren: Installing that way is a supported install method, so having this well tested is wicked cool and really important.
(03:17:52 PM) soren: ...but I wanted to test the interactivity as well.
(03:18:07 PM) soren: So, being the virtualisation geek that I am..
(03:18:10 PM) soren: I decided to use the KVM autotest framework to do the ISO testing.
(03:18:18 PM) soren: Now, KVM autotest was designed to test KVM.
(03:19:02 PM) soren: KVM developers use it to install a bunch of different operating systems and test things to make sure they didn't change anything in KVM that broke functionality in one of the guest operating systems.
(03:19:16 PM) soren: What we want to do, though, is somewhat the opposite.
(03:19:45 PM) soren: We assume that KVM works and instead want to test the operating system.
(03:20:14 PM) soren: So, the KVM autotest framework works by runing a virtual machine..
(03:20:28 PM) soren: grabs a screenshot every second..
(03:20:47 PM) soren: ..and when the screenshot looks a particular way (e.g. when a particular dialog comes up),
(03:21:07 PM) soren: it can respond with a series of key presses or mouse events.
(03:21:22 PM) soren: This way, we can emulate a complete, interactive install session.
(03:21:27 PM) soren: Awesome stuff.
(03:21:53 PM) soren: I've started documenting this, but haven't gotten all that far, since I kept changing things faster than I could update the docs :)
(03:22:10 PM) soren: The documentation lives at  https://wiki.ubuntu.com/AutomatedISOTesting
(03:22:21 PM) soren: If you all open that page..
(03:22:38 PM) soren: ..and scroll down to the "step files" section..
(03:23:01 PM) soren: you can see a sample step from a "step file".
(03:23:17 PM) soren: A step file is a description of a test case.
(03:23:40 PM) soren: Now, looking at the sample, you can see a "step 9.45" and a "screendump" line.
(03:23:57 PM) soren: They're pretty much just meta-data for the creator or editor of the step file
(03:24:04 PM) soren: so don't worry about those.
(03:24:11 PM) soren: The important lines are the "barrier_2" and "key" ones.
(03:24:19 PM) soren: The barrier_2 line tells the testing system to wait..
(03:24:41 PM) soren: ..until the rectangle of size 117x34 of the screen, starting at 79x303..
(03:24:52 PM) soren: should have md5sum de7e18c10594ab288855a570dee7f159 within the next 47 seconds.
(03:25:04 PM) soren: If this doesn't happen, the test will fail, and a report will be generated.
(03:25:15 PM) soren: If it does pass, it goes on to the next step: "key ret"
(03:25:35 PM) soren: As you can probably guess, "key ret" sends a keypress to the guest, namely Return.
(03:26:07 PM) soren: The result of those two lines is: Wait for the language prompt right after boot to show up, and once it does, press return to accept the default "English".
(03:26:40 PM) soren: Now, pretty soon, it became obvious that there was going to be a lot of duplication involved here.
(03:26:56 PM) soren: ...all the installs would have to wait for that prompt and respond to it in the same way.
(03:27:20 PM) soren: Even worse: If that prompt were to change, /every/ step file would need to be updated.
(03:27:54 PM) soren: Even worse again: In the beginning there was no concept of "updating" step files. You had to start all over.
(03:28:08 PM) soren: Starting over makes plain old ISO testing feel like a fun time.
(03:28:08 PM) soren: It's not.
(03:28:09 PM) soren: Just so you know.
(03:28:21 PM) soren: I love people for doing it, but it's really not that much fun. :)
(03:28:45 PM) soren: Ok, so to address the mass duplication of steps and stuff, I added a step file generator.
(03:29:17 PM) soren: The step file generator generates a step file (you probably guessed this much) based on the task to be installed and the partitioning scheme to be used.
(03:30:05 PM) soren: This means that I can tell the test frame work: Hey, please test an install of the LAMP task, with LVM partitioning and do it on amd64.
(03:30:05 PM) soren: And it does so.
(03:30:12 PM) soren: See, this is all running in a virtual machines.
(03:30:16 PM) soren: Virtual machines are cool.
(03:30:21 PM) soren: So cool, in fact...
(03:30:33 PM) soren: That you can use them to make installer videos.
(03:30:55 PM) soren: So, to see what happens during a test run, you can attach a recorder thingie and turn the result into an avi.
(03:31:14 PM) soren: Now, like any decent TV chef, I've cheated and done this all in advance.
...

MeetingLogs/devweek1001/AutoServerTests (last edited 2010-01-29 10:04:29 by i59F765F3)