AutoServerTests
Differences between revisions 3 and 4
7997
Comment:
|
11429
|
Deletions are marked like this. | Additions are marked like this. |
Line 88: | Line 88: |
(03:21:07 PM) soren: it can respond with a series of key presses or mouse events. (03:21:22 PM) soren: This way, we can emulate a complete, interactive install session. (03:21:27 PM) soren: Awesome stuff. (03:21:53 PM) soren: I've started documenting this, but haven't gotten all that far, since I kept changing things faster than I could update the docs :) (03:22:10 PM) soren: The documentation lives at https://wiki.ubuntu.com/AutomatedISOTesting (03:22:21 PM) soren: If you all open that page.. (03:22:38 PM) soren: ..and scroll down to the "step files" section.. (03:23:01 PM) soren: you can see a sample step from a "step file". (03:23:17 PM) soren: A step file is a description of a test case. (03:23:40 PM) soren: Now, looking at the sample, you can see a "step 9.45" and a "screendump" line. (03:23:57 PM) soren: They're pretty much just meta-data for the creator or editor of the step file (03:24:04 PM) soren: so don't worry about those. (03:24:11 PM) soren: The important lines are the "barrier_2" and "key" ones. (03:24:19 PM) soren: The barrier_2 line tells the testing system to wait.. (03:24:41 PM) soren: ..until the rectangle of size 117x34 of the screen, starting at 79x303.. (03:24:52 PM) soren: should have md5sum de7e18c10594ab288855a570dee7f159 within the next 47 seconds. (03:25:04 PM) soren: If this doesn't happen, the test will fail, and a report will be generated. (03:25:15 PM) soren: If it does pass, it goes on to the next step: "key ret" (03:25:35 PM) soren: As you can probably guess, "key ret" sends a keypress to the guest, namely Return. (03:26:07 PM) soren: The result of those two lines is: Wait for the language prompt right after boot to show up, and once it does, press return to accept the default "English". (03:26:40 PM) soren: Now, pretty soon, it became obvious that there was going to be a lot of duplication involved here. (03:26:56 PM) soren: ...all the installs would have to wait for that prompt and respond to it in the same way. (03:27:20 PM) soren: Even worse: If that prompt were to change, /every/ step file would need to be updated. (03:27:54 PM) soren: Even worse again: In the beginning there was no concept of "updating" step files. You had to start all over. (03:28:08 PM) soren: Starting over makes plain old ISO testing feel like a fun time. (03:28:08 PM) soren: It's not. (03:28:09 PM) soren: Just so you know. (03:28:21 PM) soren: I love people for doing it, but it's really not that much fun. :) (03:28:45 PM) soren: Ok, so to address the mass duplication of steps and stuff, I added a step file generator. (03:29:17 PM) soren: The step file generator generates a step file (you probably guessed this much) based on the task to be installed and the partitioning scheme to be used. (03:30:05 PM) soren: This means that I can tell the test frame work: Hey, please test an install of the LAMP task, with LVM partitioning and do it on amd64. (03:30:05 PM) soren: And it does so. (03:30:12 PM) soren: See, this is all running in a virtual machines. (03:30:16 PM) soren: Virtual machines are cool. (03:30:21 PM) soren: So cool, in fact... (03:30:33 PM) soren: That you can use them to make installer videos. (03:30:55 PM) soren: So, to see what happens during a test run, you can attach a recorder thingie and turn the result into an avi. (03:31:14 PM) soren: Now, like any decent TV chef, I've cheated and done this all in advance. |
Dev Week -- Automated server testing -- soren -- Tue, Jan 26
UTC
(03:00:21 PM) soren: Hi, everyone. (03:00:21 PM) soren: Thanks for coming to my session on Automated Server Testing. (03:00:33 PM) soren: So.. (03:00:41 PM) soren: In the server team, we've traditionally had a problem with collecting test results. (03:01:02 PM) soren: (question in #ubuntu-classroom-chat, by the way. please put "QUESTION" so that I will spot them) (03:01:14 PM) soren: This is because our target audience and most of our users are using Ubuntu on servers that are being used to service real users. (03:01:30 PM) soren: Real users, as you are probably aware, depend on their servers to work. (03:01:46 PM) soren: They need mail server to be up and delivering mail so that they can get their daily dosage of spam.. (03:02:06 PM) soren: They need their file server to be around so they can get access to their music and various pirated software.. (03:02:19 PM) soren: They need their proxy server to work so that they can log onto facebook.. (03:02:32 PM) soren: They need the LDAP server to work so that they can look up the phone number for the pizza guy.. (03:02:37 PM) soren: And other important things. (03:02:41 PM) soren: You get the idea. (03:02:48 PM) soren: If something should fail, it means pain and suffering for the poor sysadmin. (03:02:57 PM) soren: Hence, sysadmins are very hesitant to upgrade anything before it's been through lots and lots of QA. (03:03:08 PM) soren: However, unless /some/ of them /do/ upgrade, there's not going to be much QA work done. (03:03:26 PM) soren: This places us in a rather unfortunate situation, where a significant portion of our bug reports don't come in until after release. (03:03:48 PM) soren: Anyone involved in Ubuntu development will know that this is a hassle, since fixing things after release is much more tedious than before release, since we have much less freedom to make changes. (03:04:03 PM) soren: This is very difficult to change, and I haven't come up with a golden solution. (03:04:30 PM) soren: However, the sooner we catch problems, the more time we have to work on fun stuff since we'll be putting out less fires in the end. (03:04:51 PM) soren: See, while we're cursed with a user base that doesn't start testing our product until it's essentially too late.. (03:04:54 PM) tedg left the room. (03:05:09 PM) soren: ..we areblessed with a type of software that traditionally comes with a good test suite. (03:05:26 PM) soren: MySQL for instance, comes with an extensive test suite. (03:05:39 PM) soren: This test suite runs every time we upload a new version of mysql to Ubuntu. (03:06:00 PM) soren: If the test suite fails, the build fails, and the uploader gets an e-mail. (03:06:07 PM) soren: ...and it's all very obvious that something needs fixing. (03:06:09 PM) soren: This is great. (03:06:18 PM) soren: Well.. (03:06:21 PM) soren: Sort of. (03:06:32 PM) soren: The thing is, every package in Ubuntu has dependencies of some sort. (03:06:43 PM) soren: For instance, almost everything depends on libc (03:07:00 PM) soren: This means that a change in libc will inevitably affect MySQL somehow. (03:07:28 PM) soren: Luckily, if this causes problems, it is (hopefully) caught by MySQL's test suite. (03:07:44 PM) soren: Less luckily, this test suite, as I just mentioned.. (03:07:49 PM) soren: is run when MySQL is uploaded.. (03:07:53 PM) soren: not when libc is uploaded. (03:08:13 PM) soren: So we may not notice a problem until the next time someone uploads MySQL. This could be weeks or even months! (03:08:38 PM) soren: And trying to narrow down the change that broke something is hard with all the stuff doing on in Ubuntu development over the course of months. (03:08:42 PM) soren: So.. (03:09:03 PM) soren: to address this, we've set up and automated system that rebuilds MySQL ( and a bunch of other stuff) every night in a PPA. (03:09:30 PM) soren: That way, if we trust the test suite, we can relax and know that MySQL still works, despite any changes in its dependency chain. (03:09:46 PM) soren: We do the same for libvirt, php5, postgresql, etc. (03:10:09 PM) soren: Basically, anything that has a test suite that runs at build time and that causes the build to fail if it doesn't pass, should be added. (03:10:23 PM) soren: This at least makes me sleep better :) (03:11:01 PM) soren: So, the automated testing stuff in Lucid consists of two parts. (03:11:10 PM) soren: The above is the first part, which is pretty nice. (03:11:15 PM) soren: The second part is awesome: (03:11:16 PM) soren: :) (03:11:25 PM) soren: It's an automated ISO testing system. (03:11:40 PM) soren: ISO testing is the thankless and tedious job of installing Ubuntu from an ISO over and over again.. (03:12:00 PM) soren: ..with small adjustmets each time to make sure things haven't changed unexpectedly. (03:12:36 PM) soren: QUESTION: ~Shkodrani> why not run the test suite only when a packege on which, for instance MySQL relays on? (03:13:03 PM) soren: The cost of checking whether something in MySQL's dependency chain has changed is rather high. At the very least, it's tedious. (03:13:18 PM) soren: ..and just doing the rebuild is cheap and simple to get up and running. (03:13:29 PM) soren: It's all run by a 10 line shell script or thereabouts. (03:13:48 PM) soren: Ok, ISO testing.. (03:14:05 PM) soren: Every time we come close to an alpha, beta or any other kind of release.. (03:14:14 PM) soren: ..we all spend a lot of itme going through this install process. (03:14:26 PM) soren: Well, we /should/ anyway. I positively suck at getting it done, but there you go. (03:14:45 PM) soren: My fellow server team member, Mathias Gug, has had a preseed based setup running for a while now. (03:15:01 PM) soren: Basically, preseeding is a way to answer all of the installer's questions up front. (03:15:11 PM) soren: So, he takes all the answers.. (03:15:18 PM) soren: passes them to the install using clever hacks.. (03:15:31 PM) soren: ..and the install zips through the instlalation without bothering Mathias with questions. (03:15:45 PM) soren: In the end, he can log into the installed system and run the las tparts of the test cases. (03:16:07 PM) soren: This has served us well, and has probably saved us several man days (or weeks?) of testing tie over the last few years. (03:16:18 PM) soren: However, it doesn't actually test the same things as the ISO test cases describe. (03:16:33 PM) soren: The ISO test cases speak of the interaction between the user and the installer.. (03:16:46 PM) soren: However, the point of preseeding is to /avoid/ interaction, and to skip it entirely. (03:16:54 PM) soren: Don't get me wrong.. (03:17:00 PM) soren: Preseed testing is super valuable. (03:17:23 PM) soren: Installing that way is a supported install method, so having this well tested is wicked cool and really important. (03:17:52 PM) soren: ...but I wanted to test the interactivity as well. (03:18:07 PM) soren: So, being the virtualisation geek that I am.. (03:18:10 PM) soren: I decided to use the KVM autotest framework to do the ISO testing. (03:18:18 PM) soren: Now, KVM autotest was designed to test KVM. (03:19:02 PM) soren: KVM developers use it to install a bunch of different operating systems and test things to make sure they didn't change anything in KVM that broke functionality in one of the guest operating systems. (03:19:16 PM) soren: What we want to do, though, is somewhat the opposite. (03:19:45 PM) soren: We assume that KVM works and instead want to test the operating system. (03:20:14 PM) soren: So, the KVM autotest framework works by runing a virtual machine.. (03:20:28 PM) soren: grabs a screenshot every second.. (03:20:47 PM) soren: ..and when the screenshot looks a particular way (e.g. when a particular dialog comes up), (03:21:07 PM) soren: it can respond with a series of key presses or mouse events. (03:21:22 PM) soren: This way, we can emulate a complete, interactive install session. (03:21:27 PM) soren: Awesome stuff. (03:21:53 PM) soren: I've started documenting this, but haven't gotten all that far, since I kept changing things faster than I could update the docs :) (03:22:10 PM) soren: The documentation lives at https://wiki.ubuntu.com/AutomatedISOTesting (03:22:21 PM) soren: If you all open that page.. (03:22:38 PM) soren: ..and scroll down to the "step files" section.. (03:23:01 PM) soren: you can see a sample step from a "step file". (03:23:17 PM) soren: A step file is a description of a test case. (03:23:40 PM) soren: Now, looking at the sample, you can see a "step 9.45" and a "screendump" line. (03:23:57 PM) soren: They're pretty much just meta-data for the creator or editor of the step file (03:24:04 PM) soren: so don't worry about those. (03:24:11 PM) soren: The important lines are the "barrier_2" and "key" ones. (03:24:19 PM) soren: The barrier_2 line tells the testing system to wait.. (03:24:41 PM) soren: ..until the rectangle of size 117x34 of the screen, starting at 79x303.. (03:24:52 PM) soren: should have md5sum de7e18c10594ab288855a570dee7f159 within the next 47 seconds. (03:25:04 PM) soren: If this doesn't happen, the test will fail, and a report will be generated. (03:25:15 PM) soren: If it does pass, it goes on to the next step: "key ret" (03:25:35 PM) soren: As you can probably guess, "key ret" sends a keypress to the guest, namely Return. (03:26:07 PM) soren: The result of those two lines is: Wait for the language prompt right after boot to show up, and once it does, press return to accept the default "English". (03:26:40 PM) soren: Now, pretty soon, it became obvious that there was going to be a lot of duplication involved here. (03:26:56 PM) soren: ...all the installs would have to wait for that prompt and respond to it in the same way. (03:27:20 PM) soren: Even worse: If that prompt were to change, /every/ step file would need to be updated. (03:27:54 PM) soren: Even worse again: In the beginning there was no concept of "updating" step files. You had to start all over. (03:28:08 PM) soren: Starting over makes plain old ISO testing feel like a fun time. (03:28:08 PM) soren: It's not. (03:28:09 PM) soren: Just so you know. (03:28:21 PM) soren: I love people for doing it, but it's really not that much fun. :) (03:28:45 PM) soren: Ok, so to address the mass duplication of steps and stuff, I added a step file generator. (03:29:17 PM) soren: The step file generator generates a step file (you probably guessed this much) based on the task to be installed and the partitioning scheme to be used. (03:30:05 PM) soren: This means that I can tell the test frame work: Hey, please test an install of the LAMP task, with LVM partitioning and do it on amd64. (03:30:05 PM) soren: And it does so. (03:30:12 PM) soren: See, this is all running in a virtual machines. (03:30:16 PM) soren: Virtual machines are cool. (03:30:21 PM) soren: So cool, in fact... (03:30:33 PM) soren: That you can use them to make installer videos. (03:30:55 PM) soren: So, to see what happens during a test run, you can attach a recorder thingie and turn the result into an avi. (03:31:14 PM) soren: Now, like any decent TV chef, I've cheated and done this all in advance. ...
MeetingLogs/devweek1001/AutoServerTests (last edited 2010-01-29 10:04:29 by i59F765F3)