Dev Week -- Automated server testing -- soren -- Tue, Jan 26
1 [20:00] * soren clears throat 2 [20:00] <soren> Hi, everyone. 3 [20:00] <soren> Thanks for coming to my session on Automated Server Testing. 4 [20:00] <soren> So.. 5 [20:00] <soren> In the server team, we've traditionally had a problem with collecting test results. 6 [20:01] <soren> (question in #ubuntu-classroom-chat, by the way. please put "QUESTION" so that I will spot them) 7 [20:01] <soren> This is because our target audience and most of our users are using Ubuntu on servers that are being used to service real users. 8 [20:01] <soren> Real users, as you are probably aware, depend on their servers to work. 9 [20:01] <soren> They need mail server to be up and delivering mail so that they can get their daily dosage of spam.. 10 [20:02] <soren> They need their file server to be around so they can get access to their music and various pirated software.. 11 [20:02] <soren> They need their proxy server to work so that they can log onto facebook.. 12 [20:02] <soren> They need the LDAP server to work so that they can look up the phone number for the pizza guy.. 13 [20:02] <soren> And other important things. 14 [20:02] <soren> You get the idea. 15 [20:02] <soren> If something should fail, it means pain and suffering for the poor sysadmin. 16 [20:02] <soren> Hence, sysadmins are very hesitant to upgrade anything before it's been through lots and lots of QA. 17 [20:03] <soren> However, unless /some/ of them /do/ upgrade, there's not going to be much QA work done. 18 [20:03] <soren> This places us in a rather unfortunate situation, where a significant portion of our bug reports don't come in until after release. 19 [20:03] <soren> Anyone involved in Ubuntu development will know that this is a hassle, since fixing things after release is much more tedious than before release, since we have much less freedom to make changes. 20 [20:04] <soren> This is very difficult to change, and I haven't come up with a golden solution. 21 [20:04] <soren> However, the sooner we catch problems, the more time we have to work on fun stuff since we'll be putting out less fires in the end. 22 [20:04] <soren> See, while we're cursed with a user base that doesn't start testing our product until it's essentially too late.. 23 [20:05] <soren> ..we areblessed with a type of software that traditionally comes with a good test suite. 24 [20:05] <soren> MySQL for instance, comes with an extensive test suite. 25 [20:05] <soren> This test suite runs every time we upload a new version of mysql to Ubuntu. 26 [20:06] <soren> If the test suite fails, the build fails, and the uploader gets an e-mail. 27 [20:06] <soren> ...and it's all very obvious that something needs fixing. 28 [20:06] <soren> This is great. 29 [20:06] <soren> Well.. 30 [20:06] <soren> Sort of. 31 [20:06] <soren> The thing is, every package in Ubuntu has dependencies of some sort. 32 [20:06] <soren> For instance, almost everything depends on libc 33 [20:07] <soren> This means that a change in libc will inevitably affect MySQL somehow. 34 [20:07] <soren> Luckily, if this causes problems, it is (hopefully) caught by MySQL's test suite. 35 [20:07] <soren> Less luckily, this test suite, as I just mentioned.. 36 [20:07] <soren> is run when MySQL is uploaded.. 37 [20:07] <soren> not when libc is uploaded. 38 [20:08] <soren> So we may not notice a problem until the next time someone uploads MySQL. This could be weeks or even months! 39 [20:08] <soren> And trying to narrow down the change that broke something is hard with all the stuff doing on in Ubuntu development over the course of months. 40 [20:08] <soren> So.. 41 [20:09] <soren> to address this, we've set up and automated system that rebuilds MySQL ( and a bunch of other stuff) every night in a PPA. 42 [20:09] <soren> That way, if we trust the test suite, we can relax and know that MySQL still works, despite any changes in its dependency chain. 43 [20:09] <soren> We do the same for libvirt, php5, postgresql, etc. 44 [20:10] <soren> Basically, anything that has a test suite that runs at build time and that causes the build to fail if it doesn't pass, should be added. 45 [20:10] <soren> This at least makes me sleep better :) 46 [20:11] <soren> So, the automated testing stuff in Lucid consists of two parts. 47 [20:11] <soren> The above is the first part, which is pretty nice. 48 [20:11] <soren> The second part is awesome: 49 [20:11] <soren> :) 50 [20:11] <soren> It's an automated ISO testing system. 51 [20:11] <soren> ISO testing is the thankless and tedious job of installing Ubuntu from an ISO over and over again.. 52 [20:12] <soren> ..with small adjustmets each time to make sure things haven't changed unexpectedly. 53 [20:12] <soren> QUESTION: ~Shkodrani> why not run the test suite only when a packege on which, for instance MySQL relays on? 54 [20:13] <soren> The cost of checking whether something in MySQL's dependency chain has changed is rather high. At the very least, it's tedious. 55 [20:13] <soren> ..and just doing the rebuild is cheap and simple to get up and running. 56 [20:13] <soren> It's all run by a 10 line shell script or thereabouts. 57 [20:13] <soren> Ok, ISO testing.. 58 [20:14] <soren> Every time we come close to an alpha, beta or any other kind of release.. 59 [20:14] <soren> ..we all spend a lot of itme going through this install process. 60 [20:14] <soren> Well, we /should/ anyway. I positively suck at getting it done, but there you go. 61 [20:14] <soren> My fellow server team member, Mathias Gug, has had a preseed based setup running for a while now. 62 [20:15] <soren> Basically, preseeding is a way to answer all of the installer's questions up front. 63 [20:15] <soren> So, he takes all the answers.. 64 [20:15] <soren> passes them to the install using clever hacks.. 65 [20:15] <soren> ..and the install zips through the instlalation without bothering Mathias with questions. 66 [20:15] <soren> In the end, he can log into the installed system and run the las tparts of the test cases. 67 [20:16] <soren> This has served us well, and has probably saved us several man days (or weeks?) of testing tie over the last few years. 68 [20:16] <soren> However, it doesn't actually test the same things as the ISO test cases describe. 69 [20:16] <soren> The ISO test cases speak of the interaction between the user and the installer.. 70 [20:16] <soren> However, the point of preseeding is to /avoid/ interaction, and to skip it entirely. 71 [20:16] <soren> Don't get me wrong.. 72 [20:17] <soren> Preseed testing is super valuable. 73 [20:17] <soren> Installing that way is a supported install method, so having this well tested is wicked cool and really important. 74 [20:17] <soren> ...but I wanted to test the interactivity as well. 75 [20:18] <soren> So, being the virtualisation geek that I am.. 76 [20:18] <soren> I decided to use the KVM autotest framework to do the ISO testing. 77 [20:18] <soren> Now, KVM autotest was designed to test KVM. 78 [20:19] <soren> KVM developers use it to install a bunch of different operating systems and test things to make sure they didn't change anything in KVM that broke functionality in one of the guest operating systems. 79 [20:19] <soren> What we want to do, though, is somewhat the opposite. 80 [20:19] <soren> We assume that KVM works and instead want to test the operating system. 81 [20:20] <soren> So, the KVM autotest framework works by runing a virtual machine.. 82 [20:20] <soren> grabs a screenshot every second.. 83 [20:20] <soren> ..and when the screenshot looks a particular way (e.g. when a particular dialog comes up), 84 [20:21] <soren> it can respond with a series of key presses or mouse events. 85 [20:21] <soren> This way, we can emulate a complete, interactive install session. 86 [20:21] <soren> Awesome stuff. 87 [20:21] <soren> I've started documenting this, but haven't gotten all that far, since I kept changing things faster than I could update the docs :) 88 [20:22] <soren> The documentation lives at https://wiki.ubuntu.com/AutomatedISOTesting 89 [20:22] <soren> If you all open that page.. 90 [20:22] <soren> ..and scroll down to the "step files" section.. 91 [20:23] <soren> you can see a sample step from a "step file". 92 [20:23] <soren> A step file is a description of a test case. 93 [20:23] <soren> Now, looking at the sample, you can see a "step 9.45" and a "screendump" line. 94 [20:23] <soren> They're pretty much just meta-data for the creator or editor of the step file 95 [20:24] <soren> so don't worry about those. 96 [20:24] <soren> The important lines are the "barrier_2" and "key" ones. 97 [20:24] <soren> The barrier_2 line tells the testing system to wait.. 98 [20:24] <soren> ..until the rectangle of size 117x34 of the screen, starting at 79x303.. 99 [20:24] <soren> should have md5sum de7e18c10594ab288855a570dee7f159 within the next 47 seconds. 100 [20:25] <soren> If this doesn't happen, the test will fail, and a report will be generated. 101 [20:25] <soren> If it does pass, it goes on to the next step: "key ret" 102 [20:25] <soren> As you can probably guess, "key ret" sends a keypress to the guest, namely Return. 103 [20:26] <soren> The result of those two lines is: Wait for the language prompt right after boot to show up, and once it does, press return to accept the default "English". 104 [20:26] <soren> Now, pretty soon, it became obvious that there was going to be a lot of duplication involved here. 105 [20:26] <soren> ...all the installs would have to wait for that prompt and respond to it in the same way. 106 [20:27] <soren> Even worse: If that prompt were to change, /every/ step file would need to be updated. 107 [20:27] <soren> Even worse again: In the beginning there was no concept of "updating" step files. You had to start all over. 108 [20:28] <soren> Starting over makes plain old ISO testing feel like a fun time. 109 [20:28] <soren> It's not. 110 [20:28] <soren> Just so you know. 111 [20:28] <soren> I love people for doing it, but it's really not that much fun. :) 112 [20:28] <soren> Ok, so to address the mass duplication of steps and stuff, I added a step file generator. 113 [20:29] <soren> The step file generator generates a step file (you probably guessed this much) based on the task to be installed and the partitioning scheme to be used. 114 [20:30] <soren> This means that I can tell the test frame work: Hey, please test an install of the LAMP task, with LVM partitioning and do it on amd64. 115 [20:30] <soren> And it does so. 116 [20:30] <soren> See, this is all running in a virtual machines. 117 [20:30] <soren> Virtual machines are cool. 118 [20:30] <soren> So cool, in fact... 119 [20:30] <soren> That you can use them to make installer videos. 120 [20:30] <soren> So, to see what happens during a test run, you can attach a recorder thingie and turn the result into an avi. 121 [20:31] <soren> Now, like any decent TV chef, I've cheated and done this all in advance. 122 [20:31] <soren> Now, unlike most decent TV chef's, what I did in advance failed. 123 [20:31] <soren> And even more unlike TV chef's, I'm going to show it to you anyway, because it's useful. 124 [20:32] <soren> Without further ado: 125 [20:32] <soren> heh.. 126 [20:32] <soren> wait for it.. 127 [20:32] <soren> http://people.canonical.com/~soren/lamplvminstall.avi 128 [20:32] <soren> There we go. 129 [20:32] <soren> wget http://people.canonical.com/~soren/lamplvminstall.avi ; mplayer lamplvminstall.avi 130 [20:32] <soren> This test case failed. 131 [20:33] <soren> Somewhat surprisingly. 132 [20:33] <soren> If you fast forward all the way to the end.. 133 [20:33] <soren> (watch the rest as well, it's fun to watch the test system typing the username "John W. Doe III" and the password and whatnot) 134 [20:34] <soren> ..at the end, you'll see if breaks off before the install actually finishes. 135 [20:34] <soren> Like... seconds before it would have finished. 136 [20:34] <soren> Honestly, I did not mean for this to happen, but it's a good learning experience :) 137 [20:34] <soren> Ok, if we all look at.. 138 [20:34] * soren digs through launchpad, bear with me. 139 [20:34] <soren> http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/ 140 [20:35] <soren> Those are the input files for the step file generator. 141 [20:35] <soren> Yes, they are poorly named, but please appreciate that just days ago, they were all named "foo", "bar", "wibble", "wobble", etc. so this is a massive improvement. 142 [20:36] <soren> QUESTION: That method could be used for UI testing in a *lot* of different GUI apps, not just ISO installations. Any plans to document/release it more generally? 143 [20:36] <soren> (from rmunn) 144 [20:36] <soren> Yes1 145 [20:36] <soren> ! 146 [20:36] <soren> I meant to get that done for today, but the real world imposed and made a mockery of my plans. 147 [20:36] <soren> This can totally be used to do GUI installs as well. 148 [20:37] <soren> Looking at http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/ again.. 149 [20:37] <soren> Specifically, 060-finish_install_and_reboot.steps 150 [20:37] <soren> http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/annotate/head:/client/tests/kvm/generator_data/lucid/060-finish_install_and_reboot.steps 151 [20:38] <soren> This is the step that failed. 152 [20:38] <soren> For some reason (that I have yet to figure out, I only spotted this failure an hour ago) this times out. 153 [20:38] <soren> It says 579, but perhaps those a special kind of seconds that are not as long as most people's seconds. 154 [20:39] <soren> The point is this: I only have to change the timeout in this one place, and all the test cases will be updated. 155 [20:39] <soren> < ~rmunn> QUESTION: I see a lot of keystrokes used to select various dialog widgets. Can the KVM testing system simulate mouse clicks and/or mouse 156 [20:39] <soren> movements (e.g., for testing mouseover stuff) as well? 157 [20:39] <soren> cut'n'paste for the lose :( 158 [20:39] <soren> Well.. 159 [20:39] <soren> Yes. 160 [20:39] <soren> Sort of :) 161 [20:40] <soren> The autotest framework supports it, I've added support for it to the frontend, but kvm has an.. um.. issue :) 162 [20:40] <soren> It used to emulate a mouse, so it would move the cursor relative to the current position. 163 [20:40] <soren> However, these days, GNOME and such give you... 164 [20:40] <soren> mouse acceleration! 165 [20:40] <soren> Yay! 166 [20:40] <soren> No. Not yay. 167 [20:41] <soren> Mouse acceleration is the enemy when you're actually warping the mouse from one place to another, because it thinks you just moved your mouse /really/ fast, and then moves it even further than you wanted it to. 168 [20:41] <soren> This took me /forever/ to realise. 169 [20:41] <soren> So, I've made it pretend to use a tablet. 170 [20:41] <soren> Tablets offer absolute positioning, so this helped a lot. 171 [20:42] <soren> However, the command to tell kvm to click on something internally translated into "mouse_event(click_button1, 0, 0, 0)", where 0,0,0 are the coordinates. 172 [20:42] <soren> Now, if you're in relative positioning mode (using a regular mouse), this is good. 173 [20:42] <soren> You want to click right where you are. 174 [20:42] <soren> ..if you're using a tablet, it means you can only click in the top left corner. 175 [20:42] <soren> No fun. 176 [20:43] <soren> I wrote a patch for that, but I'm not sure it's in upstream KVM yet, but it'll be in Lucid half an hour after I start working on those GUI test cases :) 177 [20:44] <soren> So, yes, GUI testing is totally an optoin. 178 [20:44] <soren> option, too. 179 [20:44] <soren> Another problem I had with this is that it was designed to test a variable kvm against a static set of OS's. 180 [20:45] <soren> The OS's should look and act the same regardless of what changed in KVM. That is the whole point of these tests: To make sure they don't change. 181 [20:45] <soren> However, we change the OS all the time. That's what we do :) 182 [20:46] <soren> ..but since the designers of this test system never meant for it to be used this way, they didn't add an option to edit these step files very conveniently. 183 [20:46] <soren> To fix this, I've added an option to the test system to fall back to the stepmaker (the GUI used to create step files) if a test fails. 184 [20:46] <soren> This is great if you're running tests on your laptop or a machine you have direct access to rather than a machine running in a dusty corner of a data center. 185 [20:47] <soren> It really comes in useful when the screens change (wording changes, extra/fewer dialogs, change of theme (in the GUI)). 186 [20:47] <soren> Having to start over is, as I mentioned, no fun at all. 187 [20:48] <soren> Please shoot any questions you may have. I haven't really prepared much more than this. 188 [20:48] <soren> Still, questions belong in #ubuntu-classroom-chat 189 [20:50] <soren> If there are no more questions, I'll sing for the rest of the time slot. 190 [20:50] <soren> 21:50:25 < ~Omahn23> soren: QUESTION As an end user/sysadmin, is there anything I can do to help in testing with this new framework? 191 [20:51] <soren> Well, seeing as these things run in virtual machines, running them in more places is not going to make much difference, so /running/ the tests is probably not something we need help with 192 [20:51] <soren> However! 193 [20:51] <soren> The more test cases we can include, the better. 194 [20:51] <soren> The more, the merrier. 195 [20:51] <soren> I'd love to have more test cases to include in our daily runs of this system. 196 [20:52] <soren> 21:52:01 < hggdh> QUESTION: so we can automate pseudo-interactive testing. How to deal with the tests that require meat between the keyboard and the chair? 197 [20:52] <soren> Examples? 198 [20:52] <soren> 21:50:12 < Ramonster> soren: Any thoughts on testing servers while they actually perform one of the roles you talked about at the start ? 199 [20:52] <soren> Ramonster: You mean functional testing of e.g. a LAMP server? 200 [20:53] <soren> 21:52:23 < mscahill> QUESTION: you briefly mentioned PPA testing. what packages are included in this testing? 201 [20:53] * soren looks that up. 202 [20:53] <soren> 21:53:08 < Ramonster> soren: Yeah 203 [20:53] <soren> alright. 204 [20:53] <soren> Um, yes, but it's not part of this work I've been doing. 205 [20:53] <soren> We're not very strong in that area at all. 206 [20:53] <soren> ...and that's a shame. 207 [20:54] <soren> PKGS="libvirt postgresql-8.3 postgresql-8.4 mysql-dfsg-5.0 mysql-dfsg-5.1 openldap php5 python2.6 atlas" 208 [20:54] <soren> Is the list of packages built daily. 209 [20:55] <soren> Well, the security team has a bunch of tests they run whenever they change anything. They often can't rely on anyone else to test anything (since they don't go through -proposed), so they need to be really thorough. 210 [20:55] <soren> I'm working on getting those run every day as well. They should provide some amount of functional testing. 211 [20:55] <soren> 21:53:12 < yltsrc> QUESTION: is writing test cases required for bugfixing? 212 [20:55] <soren> Not per se. 213 [20:56] <soren> Most test cases need updating once a bug is fixed, and most things I can think of would be covered by this, so new test cases (for this system, I mean) wouldn't be a requirement for bug fixes. 214 [20:57] <soren> 21:54:55 < mscahill> QUESTION: are there plans to allow automated testing for package maintainers with their own PPA? 215 [20:57] <soren> Sure, anyone is free to run that script and do their own testing. 216 [20:57] <soren> Hm... I may not have published it anywhere. 217 [20:57] * soren fixes that. 218 [20:57] <soren> Well, /me makes a note to fix that 219 [20:58] <soren> I do have a few ideas for doing functional testing of upgrades of various sort, but most of those ideas are only a few hours old, so they're not even half baked yet :) 220 [20:59] <soren> Did I miss any questions? 221 [20:59] <soren> 21:59:33 < Ramonster> soren: That's the problem atm everyone is walking around with these half-baked ideas :) 222 [20:59] <soren> Whoops, didn't mean to post that here :) 223 [21:00] <soren> Thanks for showing up, everyone. 224 [21:00] <soren> that's it!