AutoServerTests
Differences between revisions 6 and 7
21089
Comment:
|
← Revision 7 as of 2010-01-29 10:04:29 ⇥
19973
|
Deletions are marked like this. | Additions are marked like this. |
Line 4: | Line 4: |
{{{ (03:00:21 PM) soren: Hi, everyone. (03:00:21 PM) soren: Thanks for coming to my session on Automated Server Testing. (03:00:33 PM) soren: So.. (03:00:41 PM) soren: In the server team, we've traditionally had a problem with collecting test results. (03:01:02 PM) soren: (question in #ubuntu-classroom-chat, by the way. please put "QUESTION" so that I will spot them) (03:01:14 PM) soren: This is because our target audience and most of our users are using Ubuntu on servers that are being used to service real users. (03:01:30 PM) soren: Real users, as you are probably aware, depend on their servers to work. (03:01:46 PM) soren: They need mail server to be up and delivering mail so that they can get their daily dosage of spam.. (03:02:06 PM) soren: They need their file server to be around so they can get access to their music and various pirated software.. (03:02:19 PM) soren: They need their proxy server to work so that they can log onto facebook.. (03:02:32 PM) soren: They need the LDAP server to work so that they can look up the phone number for the pizza guy.. (03:02:37 PM) soren: And other important things. (03:02:41 PM) soren: You get the idea. (03:02:48 PM) soren: If something should fail, it means pain and suffering for the poor sysadmin. (03:02:57 PM) soren: Hence, sysadmins are very hesitant to upgrade anything before it's been through lots and lots of QA. (03:03:08 PM) soren: However, unless /some/ of them /do/ upgrade, there's not going to be much QA work done. (03:03:26 PM) soren: This places us in a rather unfortunate situation, where a significant portion of our bug reports don't come in until after release. (03:03:48 PM) soren: Anyone involved in Ubuntu development will know that this is a hassle, since fixing things after release is much more tedious than before release, since we have much less freedom to make changes. (03:04:03 PM) soren: This is very difficult to change, and I haven't come up with a golden solution. (03:04:30 PM) soren: However, the sooner we catch problems, the more time we have to work on fun stuff since we'll be putting out less fires in the end. (03:04:51 PM) soren: See, while we're cursed with a user base that doesn't start testing our product until it's essentially too late.. (03:04:54 PM) tedg left the room. (03:05:09 PM) soren: ..we areblessed with a type of software that traditionally comes with a good test suite. (03:05:26 PM) soren: MySQL for instance, comes with an extensive test suite. (03:05:39 PM) soren: This test suite runs every time we upload a new version of mysql to Ubuntu. (03:06:00 PM) soren: If the test suite fails, the build fails, and the uploader gets an e-mail. (03:06:07 PM) soren: ...and it's all very obvious that something needs fixing. (03:06:09 PM) soren: This is great. (03:06:18 PM) soren: Well.. (03:06:21 PM) soren: Sort of. (03:06:32 PM) soren: The thing is, every package in Ubuntu has dependencies of some sort. (03:06:43 PM) soren: For instance, almost everything depends on libc (03:07:00 PM) soren: This means that a change in libc will inevitably affect MySQL somehow. (03:07:28 PM) soren: Luckily, if this causes problems, it is (hopefully) caught by MySQL's test suite. (03:07:44 PM) soren: Less luckily, this test suite, as I just mentioned.. (03:07:49 PM) soren: is run when MySQL is uploaded.. (03:07:53 PM) soren: not when libc is uploaded. (03:08:13 PM) soren: So we may not notice a problem until the next time someone uploads MySQL. This could be weeks or even months! (03:08:38 PM) soren: And trying to narrow down the change that broke something is hard with all the stuff doing on in Ubuntu development over the course of months. (03:08:42 PM) soren: So.. (03:09:03 PM) soren: to address this, we've set up and automated system that rebuilds MySQL ( and a bunch of other stuff) every night in a PPA. (03:09:30 PM) soren: That way, if we trust the test suite, we can relax and know that MySQL still works, despite any changes in its dependency chain. (03:09:46 PM) soren: We do the same for libvirt, php5, postgresql, etc. (03:10:09 PM) soren: Basically, anything that has a test suite that runs at build time and that causes the build to fail if it doesn't pass, should be added. (03:10:23 PM) soren: This at least makes me sleep better :) (03:11:01 PM) soren: So, the automated testing stuff in Lucid consists of two parts. (03:11:10 PM) soren: The above is the first part, which is pretty nice. (03:11:15 PM) soren: The second part is awesome: (03:11:16 PM) soren: :) (03:11:25 PM) soren: It's an automated ISO testing system. (03:11:40 PM) soren: ISO testing is the thankless and tedious job of installing Ubuntu from an ISO over and over again.. (03:12:00 PM) soren: ..with small adjustmets each time to make sure things haven't changed unexpectedly. (03:12:36 PM) soren: QUESTION: ~Shkodrani> why not run the test suite only when a packege on which, for instance MySQL relays on? (03:13:03 PM) soren: The cost of checking whether something in MySQL's dependency chain has changed is rather high. At the very least, it's tedious. (03:13:18 PM) soren: ..and just doing the rebuild is cheap and simple to get up and running. (03:13:29 PM) soren: It's all run by a 10 line shell script or thereabouts. (03:13:48 PM) soren: Ok, ISO testing.. (03:14:05 PM) soren: Every time we come close to an alpha, beta or any other kind of release.. (03:14:14 PM) soren: ..we all spend a lot of itme going through this install process. (03:14:26 PM) soren: Well, we /should/ anyway. I positively suck at getting it done, but there you go. (03:14:45 PM) soren: My fellow server team member, Mathias Gug, has had a preseed based setup running for a while now. (03:15:01 PM) soren: Basically, preseeding is a way to answer all of the installer's questions up front. (03:15:11 PM) soren: So, he takes all the answers.. (03:15:18 PM) soren: passes them to the install using clever hacks.. (03:15:31 PM) soren: ..and the install zips through the instlalation without bothering Mathias with questions. (03:15:45 PM) soren: In the end, he can log into the installed system and run the las tparts of the test cases. (03:16:07 PM) soren: This has served us well, and has probably saved us several man days (or weeks?) of testing tie over the last few years. (03:16:18 PM) soren: However, it doesn't actually test the same things as the ISO test cases describe. (03:16:33 PM) soren: The ISO test cases speak of the interaction between the user and the installer.. (03:16:46 PM) soren: However, the point of preseeding is to /avoid/ interaction, and to skip it entirely. (03:16:54 PM) soren: Don't get me wrong.. (03:17:00 PM) soren: Preseed testing is super valuable. (03:17:23 PM) soren: Installing that way is a supported install method, so having this well tested is wicked cool and really important. (03:17:52 PM) soren: ...but I wanted to test the interactivity as well. (03:18:07 PM) soren: So, being the virtualisation geek that I am.. (03:18:10 PM) soren: I decided to use the KVM autotest framework to do the ISO testing. (03:18:18 PM) soren: Now, KVM autotest was designed to test KVM. (03:19:02 PM) soren: KVM developers use it to install a bunch of different operating systems and test things to make sure they didn't change anything in KVM that broke functionality in one of the guest operating systems. (03:19:16 PM) soren: What we want to do, though, is somewhat the opposite. (03:19:45 PM) soren: We assume that KVM works and instead want to test the operating system. (03:20:14 PM) soren: So, the KVM autotest framework works by runing a virtual machine.. (03:20:28 PM) soren: grabs a screenshot every second.. (03:20:47 PM) soren: ..and when the screenshot looks a particular way (e.g. when a particular dialog comes up), (03:21:07 PM) soren: it can respond with a series of key presses or mouse events. (03:21:22 PM) soren: This way, we can emulate a complete, interactive install session. (03:21:27 PM) soren: Awesome stuff. (03:21:53 PM) soren: I've started documenting this, but haven't gotten all that far, since I kept changing things faster than I could update the docs :) (03:22:10 PM) soren: The documentation lives at https://wiki.ubuntu.com/AutomatedISOTesting (03:22:21 PM) soren: If you all open that page.. (03:22:38 PM) soren: ..and scroll down to the "step files" section.. (03:23:01 PM) soren: you can see a sample step from a "step file". (03:23:17 PM) soren: A step file is a description of a test case. (03:23:40 PM) soren: Now, looking at the sample, you can see a "step 9.45" and a "screendump" line. (03:23:57 PM) soren: They're pretty much just meta-data for the creator or editor of the step file (03:24:04 PM) soren: so don't worry about those. (03:24:11 PM) soren: The important lines are the "barrier_2" and "key" ones. (03:24:19 PM) soren: The barrier_2 line tells the testing system to wait.. (03:24:41 PM) soren: ..until the rectangle of size 117x34 of the screen, starting at 79x303.. (03:24:52 PM) soren: should have md5sum de7e18c10594ab288855a570dee7f159 within the next 47 seconds. (03:25:04 PM) soren: If this doesn't happen, the test will fail, and a report will be generated. (03:25:15 PM) soren: If it does pass, it goes on to the next step: "key ret" (03:25:35 PM) soren: As you can probably guess, "key ret" sends a keypress to the guest, namely Return. (03:26:07 PM) soren: The result of those two lines is: Wait for the language prompt right after boot to show up, and once it does, press return to accept the default "English". (03:26:40 PM) soren: Now, pretty soon, it became obvious that there was going to be a lot of duplication involved here. (03:26:56 PM) soren: ...all the installs would have to wait for that prompt and respond to it in the same way. (03:27:20 PM) soren: Even worse: If that prompt were to change, /every/ step file would need to be updated. (03:27:54 PM) soren: Even worse again: In the beginning there was no concept of "updating" step files. You had to start all over. (03:28:08 PM) soren: Starting over makes plain old ISO testing feel like a fun time. (03:28:08 PM) soren: It's not. (03:28:09 PM) soren: Just so you know. (03:28:21 PM) soren: I love people for doing it, but it's really not that much fun. :) (03:28:45 PM) soren: Ok, so to address the mass duplication of steps and stuff, I added a step file generator. (03:29:17 PM) soren: The step file generator generates a step file (you probably guessed this much) based on the task to be installed and the partitioning scheme to be used. (03:30:05 PM) soren: This means that I can tell the test frame work: Hey, please test an install of the LAMP task, with LVM partitioning and do it on amd64. (03:30:05 PM) soren: And it does so. (03:30:12 PM) soren: See, this is all running in a virtual machines. (03:30:16 PM) soren: Virtual machines are cool. (03:30:21 PM) soren: So cool, in fact... (03:30:33 PM) soren: That you can use them to make installer videos. (03:30:55 PM) soren: So, to see what happens during a test run, you can attach a recorder thingie and turn the result into an avi. (03:31:14 PM) soren: Now, like any decent TV chef, I've cheated and done this all in advance. (03:31:38 PM) soren: Now, unlike most decent TV chef's, what I did in advance failed. (03:31:58 PM) soren: And even more unlike TV chef's, I'm going to show it to you anyway, because it's useful. (03:32:06 PM) soren: Without further ado: (03:32:26 PM) soren: heh.. (03:32:29 PM) soren: wait for it.. (03:32:30 PM) soren: http://people.canonical.com/~soren/lamplvminstall.avi (03:32:32 PM) soren: There we go. (03:32:40 PM) soren: wget http://people.canonical.com/~soren/lamplvminstall.avi ; mplayer lamplvminstall.avi (03:32:48 PM) soren: This test case failed. (03:33:02 PM) soren: Somewhat surprisingly. (03:33:11 PM) soren: If you fast forward all the way to the end.. (03:33:44 PM) soren: (watch the rest as well, it's fun to watch the test system typing the username "John W. Doe III" and the password and whatnot) (03:34:01 PM) soren: ..at the end, you'll see if breaks off before the install actually finishes. (03:34:06 PM) soren: Like... seconds before it would have finished. (03:34:27 PM) soren: Honestly, I did not mean for this to happen, but it's a good learning experience :) (03:34:32 PM) soren: Ok, if we all look at.. (03:34:41 PM) ***soren digs through launchpad, bear with me. (03:34:57 PM) soren: http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/ (03:35:05 PM) soren: Those are the input files for the step file generator. (03:35:47 PM) soren: Yes, they are poorly named, but please appreciate that just days ago, they were all named "foo", "bar", "wibble", "wobble", etc. so this is a massive improvement. (03:36:12 PM) soren: QUESTION: That method could be used for UI testing in a *lot* of different GUI apps, not just ISO installations. Any plans to document/release it more generally? (03:36:16 PM) soren: (from rmunn) (03:36:18 PM) soren: Yes1 (03:36:19 PM) soren: ! (03:36:35 PM) soren: I meant to get that done for today, but the real world imposed and made a mockery of my plans. (03:36:46 PM) soren: This can totally be used to do GUI installs as well. (03:37:06 PM) soren: Looking at http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/ again.. (03:37:19 PM) soren: Specifically, 060-finish_install_and_reboot.steps (03:37:27 PM) soren: http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/annotate/head:/client/tests/kvm/generator_data/lucid/060-finish_install_and_reboot.steps (03:38:03 PM) soren: This is the step that failed. (03:38:22 PM) soren: For some reason (that I have yet to figure out, I only spotted this failure an hour ago) this times out. (03:38:42 PM) soren: It says 579, but perhaps those a special kind of seconds that are not as long as most people's seconds. (03:39:11 PM) soren: The point is this: I only have to change the timeout in this one place, and all the test cases will be updated. (03:39:27 PM) soren: < ~rmunn> QUESTION: I see a lot of keystrokes used to select various dialog widgets. Can the KVM testing system simulate mouse clicks and/or mouse (03:39:31 PM) soren: movements (e.g., for testing mouseover stuff) as well? (03:39:36 PM) soren: cut'n'paste for the lose :( (03:39:38 PM) soren: Well.. (03:39:39 PM) soren: Yes. (03:39:42 PM) soren: Sort of :) (03:40:03 PM) soren: The autotest framework supports it, I've added support for it to the frontend, but kvm has an.. um.. issue :) (03:40:30 PM) soren: It used to emulate a mouse, so it would move the cursor relative to the current position. (03:40:43 PM) soren: However, these days, GNOME and such give you... (03:40:48 PM) soren: mouse acceleration! (03:40:49 PM) soren: Yay! (03:40:52 PM) soren: No. Not yay. (03:41:26 PM) soren: Mouse acceleration is the enemy when you're actually warping the mouse from one place to another, because it thinks you just moved your mouse /really/ fast, and then moves it even further than you wanted it to. (03:41:31 PM) soren: This took me /forever/ to realise. (03:41:37 PM) soren: So, I've made it pretend to use a tablet. (03:41:45 PM) soren: Tablets offer absolute positioning, so this helped a lot. (03:42:27 PM) soren: However, the command to tell kvm to click on something internally translated into "mouse_event(click_button1, 0, 0, 0)", where 0,0,0 are the coordinates. (03:42:38 PM) soren: Now, if you're in relative positioning mode (using a regular mouse), this is good. (03:42:42 PM) soren: You want to click right where you are. (03:42:54 PM) soren: ..if you're using a tablet, it means you can only click in the top left corner. (03:42:57 PM) soren: No fun. (03:43:24 PM) soren: I wrote a patch for that, but I'm not sure it's in upstream KVM yet, but it'll be in Lucid half an hour after I start working on those GUI test cases :) (03:44:08 PM) soren: So, yes, GUI testing is totally an optoin. (03:44:11 PM) soren: option, too. (03:44:43 PM) soren: Another problem I had with this is that it was designed to test a variable kvm against a static set of OS's. (03:45:04 PM) soren: The OS's should look and act the same regardless of what changed in KVM. That is the whole point of these tests: To make sure they don't change. (03:45:19 PM) soren: However, we change the OS all the time. That's what we do :) (03:46:07 PM) soren: ..but since the designers of this test system never meant for it to be used this way, they didn't add an option to edit these step files very conveniently. (03:46:22 PM) soren: To fix this, I've added an option to the test system to fall back to the stepmaker (the GUI used to create step files) if a test fails. (03:46:48 PM) soren: This is great if you're running tests on your laptop or a machine you have direct access to rather than a machine running in a dusty corner of a data center. (03:47:27 PM) soren: It really comes in useful when the screens change (wording changes, extra/fewer dialogs, change of theme (in the GUI)). (03:47:35 PM) soren: Having to start over is, as I mentioned, no fun at all. (03:48:26 PM) soren: Please shoot any questions you may have. I haven't really prepared much more than this. (03:48:38 PM) soren: Still, questions belong in #ubuntu-classroom-chat (03:50:07 PM) soren: If there are no more questions, I'll sing for the rest of the time slot. (03:50:34 PM) soren: 21:50:25 < ~Omahn23> soren: QUESTION As an end user/sysadmin, is there anything I can do to help in testing with this new framework? (03:51:07 PM) soren: Well, seeing as these things run in virtual machines, running them in more places is not going to make much difference, so /running/ the tests is probably not something we need help with (03:51:11 PM) soren: However! (03:51:20 PM) soren: The more test cases we can include, the better. (03:51:25 PM) soren: The more, the merrier. (03:51:49 PM) soren: I'd love to have more test cases to include in our daily runs of this system. (03:52:15 PM) soren: 21:52:01 < hggdh> QUESTION: so we can automate pseudo-interactive testing. How to deal with the tests that require meat between the keyboard and the chair? (03:52:18 PM) soren: Examples? (03:52:31 PM) soren: 21:50:12 < Ramonster> soren: Any thoughts on testing servers while they actually perform one of the roles you talked about at the start ? (03:52:44 PM) soren: Ramonster: You mean functional testing of e.g. a LAMP server? (03:53:01 PM) soren: 21:52:23 < mscahill> QUESTION: you briefly mentioned PPA testing. what packages are included in this testing? (03:53:04 PM) ***soren looks that up. (03:53:26 PM) soren: 21:53:08 < Ramonster> soren: Yeah (03:53:29 PM) soren: alright. (03:53:43 PM) soren: Um, yes, but it's not part of this work I've been doing. (03:53:49 PM) soren: We're not very strong in that area at all. (03:53:53 PM) soren: ...and that's a shame. (03:54:13 PM) soren: PKGS="libvirt postgresql-8.3 postgresql-8.4 mysql-dfsg-5.0 mysql-dfsg-5.1 openldap php5 python2.6 atlas" (03:54:20 PM) soren: Is the list of packages built daily. (03:55:10 PM) soren: Well, the security team has a bunch of tests they run whenever they change anything. They often can't rely on anyone else to test anything (since they don't go through -proposed), so they need to be really thorough. (03:55:34 PM) soren: I'm working on getting those run every day as well. They should provide some amount of functional testing. (03:55:46 PM) soren: 21:53:12 < yltsrc> QUESTION: is writing test cases required for bugfixing? (03:55:58 PM) soren: Not per se. (03:56:40 PM) soren: Most test cases need updating once a bug is fixed, and most things I can think of would be covered by this, so new test cases (for this system, I mean) wouldn't be a requirement for bug fixes. (03:57:18 PM) soren: 21:54:55 < mscahill> QUESTION: are there plans to allow automated testing for package maintainers with their own PPA? (03:57:28 PM) soren: Sure, anyone is free to run that script and do their own testing. (03:57:34 PM) soren: Hm... I may not have published it anywhere. (03:57:37 PM) ***soren fixes that. (03:57:47 PM) soren: Well, /me makes a note to fix that (03:58:48 PM) soren: I do have a few ideas for doing functional testing of upgrades of various sort, but most of those ideas are only a few hours old, so they're not even half baked yet :) (03:59:16 PM) soren: Did I miss any questions? (03:59:43 PM) soren: 21:59:33 < Ramonster> soren: That's the problem atm everyone is walking around with these half-baked ideas :) (03:59:50 PM) soren: Whoops, didn't mean to post that here :) (04:00:40 PM) soren: Thanks for showing up, everyone. (04:00:45 PM) soren: that's it! |
{{{#!IRC [20:00] * soren clears throat [20:00] <soren> Hi, everyone. [20:00] <soren> Thanks for coming to my session on Automated Server Testing. [20:00] <soren> So.. [20:00] <soren> In the server team, we've traditionally had a problem with collecting test results. [20:01] <soren> (question in #ubuntu-classroom-chat, by the way. please put "QUESTION" so that I will spot them) [20:01] <soren> This is because our target audience and most of our users are using Ubuntu on servers that are being used to service real users. [20:01] <soren> Real users, as you are probably aware, depend on their servers to work. [20:01] <soren> They need mail server to be up and delivering mail so that they can get their daily dosage of spam.. [20:02] <soren> They need their file server to be around so they can get access to their music and various pirated software.. [20:02] <soren> They need their proxy server to work so that they can log onto facebook.. [20:02] <soren> They need the LDAP server to work so that they can look up the phone number for the pizza guy.. [20:02] <soren> And other important things. [20:02] <soren> You get the idea. [20:02] <soren> If something should fail, it means pain and suffering for the poor sysadmin. [20:02] <soren> Hence, sysadmins are very hesitant to upgrade anything before it's been through lots and lots of QA. [20:03] <soren> However, unless /some/ of them /do/ upgrade, there's not going to be much QA work done. [20:03] <soren> This places us in a rather unfortunate situation, where a significant portion of our bug reports don't come in until after release. [20:03] <soren> Anyone involved in Ubuntu development will know that this is a hassle, since fixing things after release is much more tedious than before release, since we have much less freedom to make changes. [20:04] <soren> This is very difficult to change, and I haven't come up with a golden solution. [20:04] <soren> However, the sooner we catch problems, the more time we have to work on fun stuff since we'll be putting out less fires in the end. [20:04] <soren> See, while we're cursed with a user base that doesn't start testing our product until it's essentially too late.. [20:05] <soren> ..we areblessed with a type of software that traditionally comes with a good test suite. [20:05] <soren> MySQL for instance, comes with an extensive test suite. [20:05] <soren> This test suite runs every time we upload a new version of mysql to Ubuntu. [20:06] <soren> If the test suite fails, the build fails, and the uploader gets an e-mail. [20:06] <soren> ...and it's all very obvious that something needs fixing. [20:06] <soren> This is great. [20:06] <soren> Well.. [20:06] <soren> Sort of. [20:06] <soren> The thing is, every package in Ubuntu has dependencies of some sort. [20:06] <soren> For instance, almost everything depends on libc [20:07] <soren> This means that a change in libc will inevitably affect MySQL somehow. [20:07] <soren> Luckily, if this causes problems, it is (hopefully) caught by MySQL's test suite. [20:07] <soren> Less luckily, this test suite, as I just mentioned.. [20:07] <soren> is run when MySQL is uploaded.. [20:07] <soren> not when libc is uploaded. [20:08] <soren> So we may not notice a problem until the next time someone uploads MySQL. This could be weeks or even months! [20:08] <soren> And trying to narrow down the change that broke something is hard with all the stuff doing on in Ubuntu development over the course of months. [20:08] <soren> So.. [20:09] <soren> to address this, we've set up and automated system that rebuilds MySQL ( and a bunch of other stuff) every night in a PPA. [20:09] <soren> That way, if we trust the test suite, we can relax and know that MySQL still works, despite any changes in its dependency chain. [20:09] <soren> We do the same for libvirt, php5, postgresql, etc. [20:10] <soren> Basically, anything that has a test suite that runs at build time and that causes the build to fail if it doesn't pass, should be added. [20:10] <soren> This at least makes me sleep better :) [20:11] <soren> So, the automated testing stuff in Lucid consists of two parts. [20:11] <soren> The above is the first part, which is pretty nice. [20:11] <soren> The second part is awesome: [20:11] <soren> :) [20:11] <soren> It's an automated ISO testing system. [20:11] <soren> ISO testing is the thankless and tedious job of installing Ubuntu from an ISO over and over again.. [20:12] <soren> ..with small adjustmets each time to make sure things haven't changed unexpectedly. [20:12] <soren> QUESTION: ~Shkodrani> why not run the test suite only when a packege on which, for instance MySQL relays on? [20:13] <soren> The cost of checking whether something in MySQL's dependency chain has changed is rather high. At the very least, it's tedious. [20:13] <soren> ..and just doing the rebuild is cheap and simple to get up and running. [20:13] <soren> It's all run by a 10 line shell script or thereabouts. [20:13] <soren> Ok, ISO testing.. [20:14] <soren> Every time we come close to an alpha, beta or any other kind of release.. [20:14] <soren> ..we all spend a lot of itme going through this install process. [20:14] <soren> Well, we /should/ anyway. I positively suck at getting it done, but there you go. [20:14] <soren> My fellow server team member, Mathias Gug, has had a preseed based setup running for a while now. [20:15] <soren> Basically, preseeding is a way to answer all of the installer's questions up front. [20:15] <soren> So, he takes all the answers.. [20:15] <soren> passes them to the install using clever hacks.. [20:15] <soren> ..and the install zips through the instlalation without bothering Mathias with questions. [20:15] <soren> In the end, he can log into the installed system and run the las tparts of the test cases. [20:16] <soren> This has served us well, and has probably saved us several man days (or weeks?) of testing tie over the last few years. [20:16] <soren> However, it doesn't actually test the same things as the ISO test cases describe. [20:16] <soren> The ISO test cases speak of the interaction between the user and the installer.. [20:16] <soren> However, the point of preseeding is to /avoid/ interaction, and to skip it entirely. [20:16] <soren> Don't get me wrong.. [20:17] <soren> Preseed testing is super valuable. [20:17] <soren> Installing that way is a supported install method, so having this well tested is wicked cool and really important. [20:17] <soren> ...but I wanted to test the interactivity as well. [20:18] <soren> So, being the virtualisation geek that I am.. [20:18] <soren> I decided to use the KVM autotest framework to do the ISO testing. [20:18] <soren> Now, KVM autotest was designed to test KVM. [20:19] <soren> KVM developers use it to install a bunch of different operating systems and test things to make sure they didn't change anything in KVM that broke functionality in one of the guest operating systems. [20:19] <soren> What we want to do, though, is somewhat the opposite. [20:19] <soren> We assume that KVM works and instead want to test the operating system. [20:20] <soren> So, the KVM autotest framework works by runing a virtual machine.. [20:20] <soren> grabs a screenshot every second.. [20:20] <soren> ..and when the screenshot looks a particular way (e.g. when a particular dialog comes up), [20:21] <soren> it can respond with a series of key presses or mouse events. [20:21] <soren> This way, we can emulate a complete, interactive install session. [20:21] <soren> Awesome stuff. [20:21] <soren> I've started documenting this, but haven't gotten all that far, since I kept changing things faster than I could update the docs :) [20:22] <soren> The documentation lives at https://wiki.ubuntu.com/AutomatedISOTesting [20:22] <soren> If you all open that page.. [20:22] <soren> ..and scroll down to the "step files" section.. [20:23] <soren> you can see a sample step from a "step file". [20:23] <soren> A step file is a description of a test case. [20:23] <soren> Now, looking at the sample, you can see a "step 9.45" and a "screendump" line. [20:23] <soren> They're pretty much just meta-data for the creator or editor of the step file [20:24] <soren> so don't worry about those. [20:24] <soren> The important lines are the "barrier_2" and "key" ones. [20:24] <soren> The barrier_2 line tells the testing system to wait.. [20:24] <soren> ..until the rectangle of size 117x34 of the screen, starting at 79x303.. [20:24] <soren> should have md5sum de7e18c10594ab288855a570dee7f159 within the next 47 seconds. [20:25] <soren> If this doesn't happen, the test will fail, and a report will be generated. [20:25] <soren> If it does pass, it goes on to the next step: "key ret" [20:25] <soren> As you can probably guess, "key ret" sends a keypress to the guest, namely Return. [20:26] <soren> The result of those two lines is: Wait for the language prompt right after boot to show up, and once it does, press return to accept the default "English". [20:26] <soren> Now, pretty soon, it became obvious that there was going to be a lot of duplication involved here. [20:26] <soren> ...all the installs would have to wait for that prompt and respond to it in the same way. [20:27] <soren> Even worse: If that prompt were to change, /every/ step file would need to be updated. [20:27] <soren> Even worse again: In the beginning there was no concept of "updating" step files. You had to start all over. [20:28] <soren> Starting over makes plain old ISO testing feel like a fun time. [20:28] <soren> It's not. [20:28] <soren> Just so you know. [20:28] <soren> I love people for doing it, but it's really not that much fun. :) [20:28] <soren> Ok, so to address the mass duplication of steps and stuff, I added a step file generator. [20:29] <soren> The step file generator generates a step file (you probably guessed this much) based on the task to be installed and the partitioning scheme to be used. [20:30] <soren> This means that I can tell the test frame work: Hey, please test an install of the LAMP task, with LVM partitioning and do it on amd64. [20:30] <soren> And it does so. [20:30] <soren> See, this is all running in a virtual machines. [20:30] <soren> Virtual machines are cool. [20:30] <soren> So cool, in fact... [20:30] <soren> That you can use them to make installer videos. [20:30] <soren> So, to see what happens during a test run, you can attach a recorder thingie and turn the result into an avi. [20:31] <soren> Now, like any decent TV chef, I've cheated and done this all in advance. [20:31] <soren> Now, unlike most decent TV chef's, what I did in advance failed. [20:31] <soren> And even more unlike TV chef's, I'm going to show it to you anyway, because it's useful. [20:32] <soren> Without further ado: [20:32] <soren> heh.. [20:32] <soren> wait for it.. [20:32] <soren> http://people.canonical.com/~soren/lamplvminstall.avi [20:32] <soren> There we go. [20:32] <soren> wget http://people.canonical.com/~soren/lamplvminstall.avi ; mplayer lamplvminstall.avi [20:32] <soren> This test case failed. [20:33] <soren> Somewhat surprisingly. [20:33] <soren> If you fast forward all the way to the end.. [20:33] <soren> (watch the rest as well, it's fun to watch the test system typing the username "John W. Doe III" and the password and whatnot) [20:34] <soren> ..at the end, you'll see if breaks off before the install actually finishes. [20:34] <soren> Like... seconds before it would have finished. [20:34] <soren> Honestly, I did not mean for this to happen, but it's a good learning experience :) [20:34] <soren> Ok, if we all look at.. [20:34] * soren digs through launchpad, bear with me. [20:34] <soren> http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/ [20:35] <soren> Those are the input files for the step file generator. [20:35] <soren> Yes, they are poorly named, but please appreciate that just days ago, they were all named "foo", "bar", "wibble", "wobble", etc. so this is a massive improvement. [20:36] <soren> QUESTION: That method could be used for UI testing in a *lot* of different GUI apps, not just ISO installations. Any plans to document/release it more generally? [20:36] <soren> (from rmunn) [20:36] <soren> Yes1 [20:36] <soren> ! [20:36] <soren> I meant to get that done for today, but the real world imposed and made a mockery of my plans. [20:36] <soren> This can totally be used to do GUI installs as well. [20:37] <soren> Looking at http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/ again.. [20:37] <soren> Specifically, 060-finish_install_and_reboot.steps [20:37] <soren> http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/annotate/head:/client/tests/kvm/generator_data/lucid/060-finish_install_and_reboot.steps [20:38] <soren> This is the step that failed. [20:38] <soren> For some reason (that I have yet to figure out, I only spotted this failure an hour ago) this times out. [20:38] <soren> It says 579, but perhaps those a special kind of seconds that are not as long as most people's seconds. [20:39] <soren> The point is this: I only have to change the timeout in this one place, and all the test cases will be updated. [20:39] <soren> < ~rmunn> QUESTION: I see a lot of keystrokes used to select various dialog widgets. Can the KVM testing system simulate mouse clicks and/or mouse [20:39] <soren> movements (e.g., for testing mouseover stuff) as well? [20:39] <soren> cut'n'paste for the lose :( [20:39] <soren> Well.. [20:39] <soren> Yes. [20:39] <soren> Sort of :) [20:40] <soren> The autotest framework supports it, I've added support for it to the frontend, but kvm has an.. um.. issue :) [20:40] <soren> It used to emulate a mouse, so it would move the cursor relative to the current position. [20:40] <soren> However, these days, GNOME and such give you... [20:40] <soren> mouse acceleration! [20:40] <soren> Yay! [20:40] <soren> No. Not yay. [20:41] <soren> Mouse acceleration is the enemy when you're actually warping the mouse from one place to another, because it thinks you just moved your mouse /really/ fast, and then moves it even further than you wanted it to. [20:41] <soren> This took me /forever/ to realise. [20:41] <soren> So, I've made it pretend to use a tablet. [20:41] <soren> Tablets offer absolute positioning, so this helped a lot. [20:42] <soren> However, the command to tell kvm to click on something internally translated into "mouse_event(click_button1, 0, 0, 0)", where 0,0,0 are the coordinates. [20:42] <soren> Now, if you're in relative positioning mode (using a regular mouse), this is good. [20:42] <soren> You want to click right where you are. [20:42] <soren> ..if you're using a tablet, it means you can only click in the top left corner. [20:42] <soren> No fun. [20:43] <soren> I wrote a patch for that, but I'm not sure it's in upstream KVM yet, but it'll be in Lucid half an hour after I start working on those GUI test cases :) [20:44] <soren> So, yes, GUI testing is totally an optoin. [20:44] <soren> option, too. [20:44] <soren> Another problem I had with this is that it was designed to test a variable kvm against a static set of OS's. [20:45] <soren> The OS's should look and act the same regardless of what changed in KVM. That is the whole point of these tests: To make sure they don't change. [20:45] <soren> However, we change the OS all the time. That's what we do :) [20:46] <soren> ..but since the designers of this test system never meant for it to be used this way, they didn't add an option to edit these step files very conveniently. [20:46] <soren> To fix this, I've added an option to the test system to fall back to the stepmaker (the GUI used to create step files) if a test fails. [20:46] <soren> This is great if you're running tests on your laptop or a machine you have direct access to rather than a machine running in a dusty corner of a data center. [20:47] <soren> It really comes in useful when the screens change (wording changes, extra/fewer dialogs, change of theme (in the GUI)). [20:47] <soren> Having to start over is, as I mentioned, no fun at all. [20:48] <soren> Please shoot any questions you may have. I haven't really prepared much more than this. [20:48] <soren> Still, questions belong in #ubuntu-classroom-chat [20:50] <soren> If there are no more questions, I'll sing for the rest of the time slot. [20:50] <soren> 21:50:25 < ~Omahn23> soren: QUESTION As an end user/sysadmin, is there anything I can do to help in testing with this new framework? [20:51] <soren> Well, seeing as these things run in virtual machines, running them in more places is not going to make much difference, so /running/ the tests is probably not something we need help with [20:51] <soren> However! [20:51] <soren> The more test cases we can include, the better. [20:51] <soren> The more, the merrier. [20:51] <soren> I'd love to have more test cases to include in our daily runs of this system. [20:52] <soren> 21:52:01 < hggdh> QUESTION: so we can automate pseudo-interactive testing. How to deal with the tests that require meat between the keyboard and the chair? [20:52] <soren> Examples? [20:52] <soren> 21:50:12 < Ramonster> soren: Any thoughts on testing servers while they actually perform one of the roles you talked about at the start ? [20:52] <soren> Ramonster: You mean functional testing of e.g. a LAMP server? [20:53] <soren> 21:52:23 < mscahill> QUESTION: you briefly mentioned PPA testing. what packages are included in this testing? [20:53] * soren looks that up. [20:53] <soren> 21:53:08 < Ramonster> soren: Yeah [20:53] <soren> alright. [20:53] <soren> Um, yes, but it's not part of this work I've been doing. [20:53] <soren> We're not very strong in that area at all. [20:53] <soren> ...and that's a shame. [20:54] <soren> PKGS="libvirt postgresql-8.3 postgresql-8.4 mysql-dfsg-5.0 mysql-dfsg-5.1 openldap php5 python2.6 atlas" [20:54] <soren> Is the list of packages built daily. [20:55] <soren> Well, the security team has a bunch of tests they run whenever they change anything. They often can't rely on anyone else to test anything (since they don't go through -proposed), so they need to be really thorough. [20:55] <soren> I'm working on getting those run every day as well. They should provide some amount of functional testing. [20:55] <soren> 21:53:12 < yltsrc> QUESTION: is writing test cases required for bugfixing? [20:55] <soren> Not per se. [20:56] <soren> Most test cases need updating once a bug is fixed, and most things I can think of would be covered by this, so new test cases (for this system, I mean) wouldn't be a requirement for bug fixes. [20:57] <soren> 21:54:55 < mscahill> QUESTION: are there plans to allow automated testing for package maintainers with their own PPA? [20:57] <soren> Sure, anyone is free to run that script and do their own testing. [20:57] <soren> Hm... I may not have published it anywhere. [20:57] * soren fixes that. [20:57] <soren> Well, /me makes a note to fix that [20:58] <soren> I do have a few ideas for doing functional testing of upgrades of various sort, but most of those ideas are only a few hours old, so they're not even half baked yet :) [20:59] <soren> Did I miss any questions? [20:59] <soren> 21:59:33 < Ramonster> soren: That's the problem atm everyone is walking around with these half-baked ideas :) [20:59] <soren> Whoops, didn't mean to post that here :) [21:00] <soren> Thanks for showing up, everyone. [21:00] <soren> that's it! |
Dev Week -- Automated server testing -- soren -- Tue, Jan 26
UTC
1 [20:00] * soren clears throat
2 [20:00] <soren> Hi, everyone.
3 [20:00] <soren> Thanks for coming to my session on Automated Server Testing.
4 [20:00] <soren> So..
5 [20:00] <soren> In the server team, we've traditionally had a problem with collecting test results.
6 [20:01] <soren> (question in #ubuntu-classroom-chat, by the way. please put "QUESTION" so that I will spot them)
7 [20:01] <soren> This is because our target audience and most of our users are using Ubuntu on servers that are being used to service real users.
8 [20:01] <soren> Real users, as you are probably aware, depend on their servers to work.
9 [20:01] <soren> They need mail server to be up and delivering mail so that they can get their daily dosage of spam..
10 [20:02] <soren> They need their file server to be around so they can get access to their music and various pirated software..
11 [20:02] <soren> They need their proxy server to work so that they can log onto facebook..
12 [20:02] <soren> They need the LDAP server to work so that they can look up the phone number for the pizza guy..
13 [20:02] <soren> And other important things.
14 [20:02] <soren> You get the idea.
15 [20:02] <soren> If something should fail, it means pain and suffering for the poor sysadmin.
16 [20:02] <soren> Hence, sysadmins are very hesitant to upgrade anything before it's been through lots and lots of QA.
17 [20:03] <soren> However, unless /some/ of them /do/ upgrade, there's not going to be much QA work done.
18 [20:03] <soren> This places us in a rather unfortunate situation, where a significant portion of our bug reports don't come in until after release.
19 [20:03] <soren> Anyone involved in Ubuntu development will know that this is a hassle, since fixing things after release is much more tedious than before release, since we have much less freedom to make changes.
20 [20:04] <soren> This is very difficult to change, and I haven't come up with a golden solution.
21 [20:04] <soren> However, the sooner we catch problems, the more time we have to work on fun stuff since we'll be putting out less fires in the end.
22 [20:04] <soren> See, while we're cursed with a user base that doesn't start testing our product until it's essentially too late..
23 [20:05] <soren> ..we areblessed with a type of software that traditionally comes with a good test suite.
24 [20:05] <soren> MySQL for instance, comes with an extensive test suite.
25 [20:05] <soren> This test suite runs every time we upload a new version of mysql to Ubuntu.
26 [20:06] <soren> If the test suite fails, the build fails, and the uploader gets an e-mail.
27 [20:06] <soren> ...and it's all very obvious that something needs fixing.
28 [20:06] <soren> This is great.
29 [20:06] <soren> Well..
30 [20:06] <soren> Sort of.
31 [20:06] <soren> The thing is, every package in Ubuntu has dependencies of some sort.
32 [20:06] <soren> For instance, almost everything depends on libc
33 [20:07] <soren> This means that a change in libc will inevitably affect MySQL somehow.
34 [20:07] <soren> Luckily, if this causes problems, it is (hopefully) caught by MySQL's test suite.
35 [20:07] <soren> Less luckily, this test suite, as I just mentioned..
36 [20:07] <soren> is run when MySQL is uploaded..
37 [20:07] <soren> not when libc is uploaded.
38 [20:08] <soren> So we may not notice a problem until the next time someone uploads MySQL. This could be weeks or even months!
39 [20:08] <soren> And trying to narrow down the change that broke something is hard with all the stuff doing on in Ubuntu development over the course of months.
40 [20:08] <soren> So..
41 [20:09] <soren> to address this, we've set up and automated system that rebuilds MySQL ( and a bunch of other stuff) every night in a PPA.
42 [20:09] <soren> That way, if we trust the test suite, we can relax and know that MySQL still works, despite any changes in its dependency chain.
43 [20:09] <soren> We do the same for libvirt, php5, postgresql, etc.
44 [20:10] <soren> Basically, anything that has a test suite that runs at build time and that causes the build to fail if it doesn't pass, should be added.
45 [20:10] <soren> This at least makes me sleep better :)
46 [20:11] <soren> So, the automated testing stuff in Lucid consists of two parts.
47 [20:11] <soren> The above is the first part, which is pretty nice.
48 [20:11] <soren> The second part is awesome:
49 [20:11] <soren> :)
50 [20:11] <soren> It's an automated ISO testing system.
51 [20:11] <soren> ISO testing is the thankless and tedious job of installing Ubuntu from an ISO over and over again..
52 [20:12] <soren> ..with small adjustmets each time to make sure things haven't changed unexpectedly.
53 [20:12] <soren> QUESTION: ~Shkodrani> why not run the test suite only when a packege on which, for instance MySQL relays on?
54 [20:13] <soren> The cost of checking whether something in MySQL's dependency chain has changed is rather high. At the very least, it's tedious.
55 [20:13] <soren> ..and just doing the rebuild is cheap and simple to get up and running.
56 [20:13] <soren> It's all run by a 10 line shell script or thereabouts.
57 [20:13] <soren> Ok, ISO testing..
58 [20:14] <soren> Every time we come close to an alpha, beta or any other kind of release..
59 [20:14] <soren> ..we all spend a lot of itme going through this install process.
60 [20:14] <soren> Well, we /should/ anyway. I positively suck at getting it done, but there you go.
61 [20:14] <soren> My fellow server team member, Mathias Gug, has had a preseed based setup running for a while now.
62 [20:15] <soren> Basically, preseeding is a way to answer all of the installer's questions up front.
63 [20:15] <soren> So, he takes all the answers..
64 [20:15] <soren> passes them to the install using clever hacks..
65 [20:15] <soren> ..and the install zips through the instlalation without bothering Mathias with questions.
66 [20:15] <soren> In the end, he can log into the installed system and run the las tparts of the test cases.
67 [20:16] <soren> This has served us well, and has probably saved us several man days (or weeks?) of testing tie over the last few years.
68 [20:16] <soren> However, it doesn't actually test the same things as the ISO test cases describe.
69 [20:16] <soren> The ISO test cases speak of the interaction between the user and the installer..
70 [20:16] <soren> However, the point of preseeding is to /avoid/ interaction, and to skip it entirely.
71 [20:16] <soren> Don't get me wrong..
72 [20:17] <soren> Preseed testing is super valuable.
73 [20:17] <soren> Installing that way is a supported install method, so having this well tested is wicked cool and really important.
74 [20:17] <soren> ...but I wanted to test the interactivity as well.
75 [20:18] <soren> So, being the virtualisation geek that I am..
76 [20:18] <soren> I decided to use the KVM autotest framework to do the ISO testing.
77 [20:18] <soren> Now, KVM autotest was designed to test KVM.
78 [20:19] <soren> KVM developers use it to install a bunch of different operating systems and test things to make sure they didn't change anything in KVM that broke functionality in one of the guest operating systems.
79 [20:19] <soren> What we want to do, though, is somewhat the opposite.
80 [20:19] <soren> We assume that KVM works and instead want to test the operating system.
81 [20:20] <soren> So, the KVM autotest framework works by runing a virtual machine..
82 [20:20] <soren> grabs a screenshot every second..
83 [20:20] <soren> ..and when the screenshot looks a particular way (e.g. when a particular dialog comes up),
84 [20:21] <soren> it can respond with a series of key presses or mouse events.
85 [20:21] <soren> This way, we can emulate a complete, interactive install session.
86 [20:21] <soren> Awesome stuff.
87 [20:21] <soren> I've started documenting this, but haven't gotten all that far, since I kept changing things faster than I could update the docs :)
88 [20:22] <soren> The documentation lives at https://wiki.ubuntu.com/AutomatedISOTesting
89 [20:22] <soren> If you all open that page..
90 [20:22] <soren> ..and scroll down to the "step files" section..
91 [20:23] <soren> you can see a sample step from a "step file".
92 [20:23] <soren> A step file is a description of a test case.
93 [20:23] <soren> Now, looking at the sample, you can see a "step 9.45" and a "screendump" line.
94 [20:23] <soren> They're pretty much just meta-data for the creator or editor of the step file
95 [20:24] <soren> so don't worry about those.
96 [20:24] <soren> The important lines are the "barrier_2" and "key" ones.
97 [20:24] <soren> The barrier_2 line tells the testing system to wait..
98 [20:24] <soren> ..until the rectangle of size 117x34 of the screen, starting at 79x303..
99 [20:24] <soren> should have md5sum de7e18c10594ab288855a570dee7f159 within the next 47 seconds.
100 [20:25] <soren> If this doesn't happen, the test will fail, and a report will be generated.
101 [20:25] <soren> If it does pass, it goes on to the next step: "key ret"
102 [20:25] <soren> As you can probably guess, "key ret" sends a keypress to the guest, namely Return.
103 [20:26] <soren> The result of those two lines is: Wait for the language prompt right after boot to show up, and once it does, press return to accept the default "English".
104 [20:26] <soren> Now, pretty soon, it became obvious that there was going to be a lot of duplication involved here.
105 [20:26] <soren> ...all the installs would have to wait for that prompt and respond to it in the same way.
106 [20:27] <soren> Even worse: If that prompt were to change, /every/ step file would need to be updated.
107 [20:27] <soren> Even worse again: In the beginning there was no concept of "updating" step files. You had to start all over.
108 [20:28] <soren> Starting over makes plain old ISO testing feel like a fun time.
109 [20:28] <soren> It's not.
110 [20:28] <soren> Just so you know.
111 [20:28] <soren> I love people for doing it, but it's really not that much fun. :)
112 [20:28] <soren> Ok, so to address the mass duplication of steps and stuff, I added a step file generator.
113 [20:29] <soren> The step file generator generates a step file (you probably guessed this much) based on the task to be installed and the partitioning scheme to be used.
114 [20:30] <soren> This means that I can tell the test frame work: Hey, please test an install of the LAMP task, with LVM partitioning and do it on amd64.
115 [20:30] <soren> And it does so.
116 [20:30] <soren> See, this is all running in a virtual machines.
117 [20:30] <soren> Virtual machines are cool.
118 [20:30] <soren> So cool, in fact...
119 [20:30] <soren> That you can use them to make installer videos.
120 [20:30] <soren> So, to see what happens during a test run, you can attach a recorder thingie and turn the result into an avi.
121 [20:31] <soren> Now, like any decent TV chef, I've cheated and done this all in advance.
122 [20:31] <soren> Now, unlike most decent TV chef's, what I did in advance failed.
123 [20:31] <soren> And even more unlike TV chef's, I'm going to show it to you anyway, because it's useful.
124 [20:32] <soren> Without further ado:
125 [20:32] <soren> heh..
126 [20:32] <soren> wait for it..
127 [20:32] <soren> http://people.canonical.com/~soren/lamplvminstall.avi
128 [20:32] <soren> There we go.
129 [20:32] <soren> wget http://people.canonical.com/~soren/lamplvminstall.avi ; mplayer lamplvminstall.avi
130 [20:32] <soren> This test case failed.
131 [20:33] <soren> Somewhat surprisingly.
132 [20:33] <soren> If you fast forward all the way to the end..
133 [20:33] <soren> (watch the rest as well, it's fun to watch the test system typing the username "John W. Doe III" and the password and whatnot)
134 [20:34] <soren> ..at the end, you'll see if breaks off before the install actually finishes.
135 [20:34] <soren> Like... seconds before it would have finished.
136 [20:34] <soren> Honestly, I did not mean for this to happen, but it's a good learning experience :)
137 [20:34] <soren> Ok, if we all look at..
138 [20:34] * soren digs through launchpad, bear with me.
139 [20:34] <soren> http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/
140 [20:35] <soren> Those are the input files for the step file generator.
141 [20:35] <soren> Yes, they are poorly named, but please appreciate that just days ago, they were all named "foo", "bar", "wibble", "wobble", etc. so this is a massive improvement.
142 [20:36] <soren> QUESTION: That method could be used for UI testing in a *lot* of different GUI apps, not just ISO installations. Any plans to document/release it more generally?
143 [20:36] <soren> (from rmunn)
144 [20:36] <soren> Yes1
145 [20:36] <soren> !
146 [20:36] <soren> I meant to get that done for today, but the real world imposed and made a mockery of my plans.
147 [20:36] <soren> This can totally be used to do GUI installs as well.
148 [20:37] <soren> Looking at http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/files/head:/client/tests/kvm/generator_data/lucid/ again..
149 [20:37] <soren> Specifically, 060-finish_install_and_reboot.steps
150 [20:37] <soren> http://bazaar.launchpad.net/~soren/autotest/automated-ubuntu-server-tests/annotate/head:/client/tests/kvm/generator_data/lucid/060-finish_install_and_reboot.steps
151 [20:38] <soren> This is the step that failed.
152 [20:38] <soren> For some reason (that I have yet to figure out, I only spotted this failure an hour ago) this times out.
153 [20:38] <soren> It says 579, but perhaps those a special kind of seconds that are not as long as most people's seconds.
154 [20:39] <soren> The point is this: I only have to change the timeout in this one place, and all the test cases will be updated.
155 [20:39] <soren> < ~rmunn> QUESTION: I see a lot of keystrokes used to select various dialog widgets. Can the KVM testing system simulate mouse clicks and/or mouse
156 [20:39] <soren> movements (e.g., for testing mouseover stuff) as well?
157 [20:39] <soren> cut'n'paste for the lose :(
158 [20:39] <soren> Well..
159 [20:39] <soren> Yes.
160 [20:39] <soren> Sort of :)
161 [20:40] <soren> The autotest framework supports it, I've added support for it to the frontend, but kvm has an.. um.. issue :)
162 [20:40] <soren> It used to emulate a mouse, so it would move the cursor relative to the current position.
163 [20:40] <soren> However, these days, GNOME and such give you...
164 [20:40] <soren> mouse acceleration!
165 [20:40] <soren> Yay!
166 [20:40] <soren> No. Not yay.
167 [20:41] <soren> Mouse acceleration is the enemy when you're actually warping the mouse from one place to another, because it thinks you just moved your mouse /really/ fast, and then moves it even further than you wanted it to.
168 [20:41] <soren> This took me /forever/ to realise.
169 [20:41] <soren> So, I've made it pretend to use a tablet.
170 [20:41] <soren> Tablets offer absolute positioning, so this helped a lot.
171 [20:42] <soren> However, the command to tell kvm to click on something internally translated into "mouse_event(click_button1, 0, 0, 0)", where 0,0,0 are the coordinates.
172 [20:42] <soren> Now, if you're in relative positioning mode (using a regular mouse), this is good.
173 [20:42] <soren> You want to click right where you are.
174 [20:42] <soren> ..if you're using a tablet, it means you can only click in the top left corner.
175 [20:42] <soren> No fun.
176 [20:43] <soren> I wrote a patch for that, but I'm not sure it's in upstream KVM yet, but it'll be in Lucid half an hour after I start working on those GUI test cases :)
177 [20:44] <soren> So, yes, GUI testing is totally an optoin.
178 [20:44] <soren> option, too.
179 [20:44] <soren> Another problem I had with this is that it was designed to test a variable kvm against a static set of OS's.
180 [20:45] <soren> The OS's should look and act the same regardless of what changed in KVM. That is the whole point of these tests: To make sure they don't change.
181 [20:45] <soren> However, we change the OS all the time. That's what we do :)
182 [20:46] <soren> ..but since the designers of this test system never meant for it to be used this way, they didn't add an option to edit these step files very conveniently.
183 [20:46] <soren> To fix this, I've added an option to the test system to fall back to the stepmaker (the GUI used to create step files) if a test fails.
184 [20:46] <soren> This is great if you're running tests on your laptop or a machine you have direct access to rather than a machine running in a dusty corner of a data center.
185 [20:47] <soren> It really comes in useful when the screens change (wording changes, extra/fewer dialogs, change of theme (in the GUI)).
186 [20:47] <soren> Having to start over is, as I mentioned, no fun at all.
187 [20:48] <soren> Please shoot any questions you may have. I haven't really prepared much more than this.
188 [20:48] <soren> Still, questions belong in #ubuntu-classroom-chat
189 [20:50] <soren> If there are no more questions, I'll sing for the rest of the time slot.
190 [20:50] <soren> 21:50:25 < ~Omahn23> soren: QUESTION As an end user/sysadmin, is there anything I can do to help in testing with this new framework?
191 [20:51] <soren> Well, seeing as these things run in virtual machines, running them in more places is not going to make much difference, so /running/ the tests is probably not something we need help with
192 [20:51] <soren> However!
193 [20:51] <soren> The more test cases we can include, the better.
194 [20:51] <soren> The more, the merrier.
195 [20:51] <soren> I'd love to have more test cases to include in our daily runs of this system.
196 [20:52] <soren> 21:52:01 < hggdh> QUESTION: so we can automate pseudo-interactive testing. How to deal with the tests that require meat between the keyboard and the chair?
197 [20:52] <soren> Examples?
198 [20:52] <soren> 21:50:12 < Ramonster> soren: Any thoughts on testing servers while they actually perform one of the roles you talked about at the start ?
199 [20:52] <soren> Ramonster: You mean functional testing of e.g. a LAMP server?
200 [20:53] <soren> 21:52:23 < mscahill> QUESTION: you briefly mentioned PPA testing. what packages are included in this testing?
201 [20:53] * soren looks that up.
202 [20:53] <soren> 21:53:08 < Ramonster> soren: Yeah
203 [20:53] <soren> alright.
204 [20:53] <soren> Um, yes, but it's not part of this work I've been doing.
205 [20:53] <soren> We're not very strong in that area at all.
206 [20:53] <soren> ...and that's a shame.
207 [20:54] <soren> PKGS="libvirt postgresql-8.3 postgresql-8.4 mysql-dfsg-5.0 mysql-dfsg-5.1 openldap php5 python2.6 atlas"
208 [20:54] <soren> Is the list of packages built daily.
209 [20:55] <soren> Well, the security team has a bunch of tests they run whenever they change anything. They often can't rely on anyone else to test anything (since they don't go through -proposed), so they need to be really thorough.
210 [20:55] <soren> I'm working on getting those run every day as well. They should provide some amount of functional testing.
211 [20:55] <soren> 21:53:12 < yltsrc> QUESTION: is writing test cases required for bugfixing?
212 [20:55] <soren> Not per se.
213 [20:56] <soren> Most test cases need updating once a bug is fixed, and most things I can think of would be covered by this, so new test cases (for this system, I mean) wouldn't be a requirement for bug fixes.
214 [20:57] <soren> 21:54:55 < mscahill> QUESTION: are there plans to allow automated testing for package maintainers with their own PPA?
215 [20:57] <soren> Sure, anyone is free to run that script and do their own testing.
216 [20:57] <soren> Hm... I may not have published it anywhere.
217 [20:57] * soren fixes that.
218 [20:57] <soren> Well, /me makes a note to fix that
219 [20:58] <soren> I do have a few ideas for doing functional testing of upgrades of various sort, but most of those ideas are only a few hours old, so they're not even half baked yet :)
220 [20:59] <soren> Did I miss any questions?
221 [20:59] <soren> 21:59:33 < Ramonster> soren: That's the problem atm everyone is walking around with these half-baked ideas :)
222 [20:59] <soren> Whoops, didn't mean to post that here :)
223 [21:00] <soren> Thanks for showing up, everyone.
224 [21:00] <soren> that's it!
MeetingLogs/devweek1001/AutoServerTests (last edited 2010-01-29 10:04:29 by i59F765F3)