AutoServerTests
Differences between revisions 2 and 3
161
Comment: page was renamed from MeetingLogs/devweek1001/devweek1001/AutoServerTests
|
7997
|
Deletions are marked like this. | Additions are marked like this. |
Line 5: | Line 5: |
(03:00:21 PM) soren: Hi, everyone. (03:00:21 PM) soren: Thanks for coming to my session on Automated Server Testing. (03:00:33 PM) soren: So.. (03:00:41 PM) soren: In the server team, we've traditionally had a problem with collecting test results. (03:01:02 PM) soren: (question in #ubuntu-classroom-chat, by the way. please put "QUESTION" so that I will spot them) (03:01:14 PM) soren: This is because our target audience and most of our users are using Ubuntu on servers that are being used to service real users. (03:01:30 PM) soren: Real users, as you are probably aware, depend on their servers to work. (03:01:46 PM) soren: They need mail server to be up and delivering mail so that they can get their daily dosage of spam.. (03:02:06 PM) soren: They need their file server to be around so they can get access to their music and various pirated software.. (03:02:19 PM) soren: They need their proxy server to work so that they can log onto facebook.. (03:02:32 PM) soren: They need the LDAP server to work so that they can look up the phone number for the pizza guy.. (03:02:37 PM) soren: And other important things. (03:02:41 PM) soren: You get the idea. (03:02:48 PM) soren: If something should fail, it means pain and suffering for the poor sysadmin. (03:02:57 PM) soren: Hence, sysadmins are very hesitant to upgrade anything before it's been through lots and lots of QA. (03:03:08 PM) soren: However, unless /some/ of them /do/ upgrade, there's not going to be much QA work done. (03:03:26 PM) soren: This places us in a rather unfortunate situation, where a significant portion of our bug reports don't come in until after release. (03:03:48 PM) soren: Anyone involved in Ubuntu development will know that this is a hassle, since fixing things after release is much more tedious than before release, since we have much less freedom to make changes. (03:04:03 PM) soren: This is very difficult to change, and I haven't come up with a golden solution. (03:04:30 PM) soren: However, the sooner we catch problems, the more time we have to work on fun stuff since we'll be putting out less fires in the end. (03:04:51 PM) soren: See, while we're cursed with a user base that doesn't start testing our product until it's essentially too late.. (03:04:54 PM) tedg left the room. (03:05:09 PM) soren: ..we areblessed with a type of software that traditionally comes with a good test suite. (03:05:26 PM) soren: MySQL for instance, comes with an extensive test suite. (03:05:39 PM) soren: This test suite runs every time we upload a new version of mysql to Ubuntu. (03:06:00 PM) soren: If the test suite fails, the build fails, and the uploader gets an e-mail. (03:06:07 PM) soren: ...and it's all very obvious that something needs fixing. (03:06:09 PM) soren: This is great. (03:06:18 PM) soren: Well.. (03:06:21 PM) soren: Sort of. (03:06:32 PM) soren: The thing is, every package in Ubuntu has dependencies of some sort. (03:06:43 PM) soren: For instance, almost everything depends on libc (03:07:00 PM) soren: This means that a change in libc will inevitably affect MySQL somehow. (03:07:28 PM) soren: Luckily, if this causes problems, it is (hopefully) caught by MySQL's test suite. (03:07:44 PM) soren: Less luckily, this test suite, as I just mentioned.. (03:07:49 PM) soren: is run when MySQL is uploaded.. (03:07:53 PM) soren: not when libc is uploaded. (03:08:13 PM) soren: So we may not notice a problem until the next time someone uploads MySQL. This could be weeks or even months! (03:08:38 PM) soren: And trying to narrow down the change that broke something is hard with all the stuff doing on in Ubuntu development over the course of months. (03:08:42 PM) soren: So.. (03:09:03 PM) soren: to address this, we've set up and automated system that rebuilds MySQL ( and a bunch of other stuff) every night in a PPA. (03:09:30 PM) soren: That way, if we trust the test suite, we can relax and know that MySQL still works, despite any changes in its dependency chain. (03:09:46 PM) soren: We do the same for libvirt, php5, postgresql, etc. (03:10:09 PM) soren: Basically, anything that has a test suite that runs at build time and that causes the build to fail if it doesn't pass, should be added. (03:10:23 PM) soren: This at least makes me sleep better :) (03:11:01 PM) soren: So, the automated testing stuff in Lucid consists of two parts. (03:11:10 PM) soren: The above is the first part, which is pretty nice. (03:11:15 PM) soren: The second part is awesome: (03:11:16 PM) soren: :) (03:11:25 PM) soren: It's an automated ISO testing system. (03:11:40 PM) soren: ISO testing is the thankless and tedious job of installing Ubuntu from an ISO over and over again.. (03:12:00 PM) soren: ..with small adjustmets each time to make sure things haven't changed unexpectedly. (03:12:36 PM) soren: QUESTION: ~Shkodrani> why not run the test suite only when a packege on which, for instance MySQL relays on? (03:13:03 PM) soren: The cost of checking whether something in MySQL's dependency chain has changed is rather high. At the very least, it's tedious. (03:13:18 PM) soren: ..and just doing the rebuild is cheap and simple to get up and running. (03:13:29 PM) soren: It's all run by a 10 line shell script or thereabouts. (03:13:48 PM) soren: Ok, ISO testing.. (03:14:05 PM) soren: Every time we come close to an alpha, beta or any other kind of release.. (03:14:14 PM) soren: ..we all spend a lot of itme going through this install process. (03:14:26 PM) soren: Well, we /should/ anyway. I positively suck at getting it done, but there you go. (03:14:45 PM) soren: My fellow server team member, Mathias Gug, has had a preseed based setup running for a while now. (03:15:01 PM) soren: Basically, preseeding is a way to answer all of the installer's questions up front. (03:15:11 PM) soren: So, he takes all the answers.. (03:15:18 PM) soren: passes them to the install using clever hacks.. (03:15:31 PM) soren: ..and the install zips through the instlalation without bothering Mathias with questions. (03:15:45 PM) soren: In the end, he can log into the installed system and run the las tparts of the test cases. (03:16:07 PM) soren: This has served us well, and has probably saved us several man days (or weeks?) of testing tie over the last few years. (03:16:18 PM) soren: However, it doesn't actually test the same things as the ISO test cases describe. (03:16:33 PM) soren: The ISO test cases speak of the interaction between the user and the installer.. (03:16:46 PM) soren: However, the point of preseeding is to /avoid/ interaction, and to skip it entirely. (03:16:54 PM) soren: Don't get me wrong.. (03:17:00 PM) soren: Preseed testing is super valuable. (03:17:23 PM) soren: Installing that way is a supported install method, so having this well tested is wicked cool and really important. (03:17:52 PM) soren: ...but I wanted to test the interactivity as well. (03:18:07 PM) soren: So, being the virtualisation geek that I am.. (03:18:10 PM) soren: I decided to use the KVM autotest framework to do the ISO testing. (03:18:18 PM) soren: Now, KVM autotest was designed to test KVM. (03:19:02 PM) soren: KVM developers use it to install a bunch of different operating systems and test things to make sure they didn't change anything in KVM that broke functionality in one of the guest operating systems. (03:19:16 PM) soren: What we want to do, though, is somewhat the opposite. (03:19:45 PM) soren: We assume that KVM works and instead want to test the operating system. (03:20:14 PM) soren: So, the KVM autotest framework works by runing a virtual machine.. (03:20:28 PM) soren: grabs a screenshot every second.. (03:20:47 PM) soren: ..and when the screenshot looks a particular way (e.g. when a particular dialog comes up), ... |
Dev Week -- Automated server testing -- soren -- Tue, Jan 26
UTC
(03:00:21 PM) soren: Hi, everyone. (03:00:21 PM) soren: Thanks for coming to my session on Automated Server Testing. (03:00:33 PM) soren: So.. (03:00:41 PM) soren: In the server team, we've traditionally had a problem with collecting test results. (03:01:02 PM) soren: (question in #ubuntu-classroom-chat, by the way. please put "QUESTION" so that I will spot them) (03:01:14 PM) soren: This is because our target audience and most of our users are using Ubuntu on servers that are being used to service real users. (03:01:30 PM) soren: Real users, as you are probably aware, depend on their servers to work. (03:01:46 PM) soren: They need mail server to be up and delivering mail so that they can get their daily dosage of spam.. (03:02:06 PM) soren: They need their file server to be around so they can get access to their music and various pirated software.. (03:02:19 PM) soren: They need their proxy server to work so that they can log onto facebook.. (03:02:32 PM) soren: They need the LDAP server to work so that they can look up the phone number for the pizza guy.. (03:02:37 PM) soren: And other important things. (03:02:41 PM) soren: You get the idea. (03:02:48 PM) soren: If something should fail, it means pain and suffering for the poor sysadmin. (03:02:57 PM) soren: Hence, sysadmins are very hesitant to upgrade anything before it's been through lots and lots of QA. (03:03:08 PM) soren: However, unless /some/ of them /do/ upgrade, there's not going to be much QA work done. (03:03:26 PM) soren: This places us in a rather unfortunate situation, where a significant portion of our bug reports don't come in until after release. (03:03:48 PM) soren: Anyone involved in Ubuntu development will know that this is a hassle, since fixing things after release is much more tedious than before release, since we have much less freedom to make changes. (03:04:03 PM) soren: This is very difficult to change, and I haven't come up with a golden solution. (03:04:30 PM) soren: However, the sooner we catch problems, the more time we have to work on fun stuff since we'll be putting out less fires in the end. (03:04:51 PM) soren: See, while we're cursed with a user base that doesn't start testing our product until it's essentially too late.. (03:04:54 PM) tedg left the room. (03:05:09 PM) soren: ..we areblessed with a type of software that traditionally comes with a good test suite. (03:05:26 PM) soren: MySQL for instance, comes with an extensive test suite. (03:05:39 PM) soren: This test suite runs every time we upload a new version of mysql to Ubuntu. (03:06:00 PM) soren: If the test suite fails, the build fails, and the uploader gets an e-mail. (03:06:07 PM) soren: ...and it's all very obvious that something needs fixing. (03:06:09 PM) soren: This is great. (03:06:18 PM) soren: Well.. (03:06:21 PM) soren: Sort of. (03:06:32 PM) soren: The thing is, every package in Ubuntu has dependencies of some sort. (03:06:43 PM) soren: For instance, almost everything depends on libc (03:07:00 PM) soren: This means that a change in libc will inevitably affect MySQL somehow. (03:07:28 PM) soren: Luckily, if this causes problems, it is (hopefully) caught by MySQL's test suite. (03:07:44 PM) soren: Less luckily, this test suite, as I just mentioned.. (03:07:49 PM) soren: is run when MySQL is uploaded.. (03:07:53 PM) soren: not when libc is uploaded. (03:08:13 PM) soren: So we may not notice a problem until the next time someone uploads MySQL. This could be weeks or even months! (03:08:38 PM) soren: And trying to narrow down the change that broke something is hard with all the stuff doing on in Ubuntu development over the course of months. (03:08:42 PM) soren: So.. (03:09:03 PM) soren: to address this, we've set up and automated system that rebuilds MySQL ( and a bunch of other stuff) every night in a PPA. (03:09:30 PM) soren: That way, if we trust the test suite, we can relax and know that MySQL still works, despite any changes in its dependency chain. (03:09:46 PM) soren: We do the same for libvirt, php5, postgresql, etc. (03:10:09 PM) soren: Basically, anything that has a test suite that runs at build time and that causes the build to fail if it doesn't pass, should be added. (03:10:23 PM) soren: This at least makes me sleep better :) (03:11:01 PM) soren: So, the automated testing stuff in Lucid consists of two parts. (03:11:10 PM) soren: The above is the first part, which is pretty nice. (03:11:15 PM) soren: The second part is awesome: (03:11:16 PM) soren: :) (03:11:25 PM) soren: It's an automated ISO testing system. (03:11:40 PM) soren: ISO testing is the thankless and tedious job of installing Ubuntu from an ISO over and over again.. (03:12:00 PM) soren: ..with small adjustmets each time to make sure things haven't changed unexpectedly. (03:12:36 PM) soren: QUESTION: ~Shkodrani> why not run the test suite only when a packege on which, for instance MySQL relays on? (03:13:03 PM) soren: The cost of checking whether something in MySQL's dependency chain has changed is rather high. At the very least, it's tedious. (03:13:18 PM) soren: ..and just doing the rebuild is cheap and simple to get up and running. (03:13:29 PM) soren: It's all run by a 10 line shell script or thereabouts. (03:13:48 PM) soren: Ok, ISO testing.. (03:14:05 PM) soren: Every time we come close to an alpha, beta or any other kind of release.. (03:14:14 PM) soren: ..we all spend a lot of itme going through this install process. (03:14:26 PM) soren: Well, we /should/ anyway. I positively suck at getting it done, but there you go. (03:14:45 PM) soren: My fellow server team member, Mathias Gug, has had a preseed based setup running for a while now. (03:15:01 PM) soren: Basically, preseeding is a way to answer all of the installer's questions up front. (03:15:11 PM) soren: So, he takes all the answers.. (03:15:18 PM) soren: passes them to the install using clever hacks.. (03:15:31 PM) soren: ..and the install zips through the instlalation without bothering Mathias with questions. (03:15:45 PM) soren: In the end, he can log into the installed system and run the las tparts of the test cases. (03:16:07 PM) soren: This has served us well, and has probably saved us several man days (or weeks?) of testing tie over the last few years. (03:16:18 PM) soren: However, it doesn't actually test the same things as the ISO test cases describe. (03:16:33 PM) soren: The ISO test cases speak of the interaction between the user and the installer.. (03:16:46 PM) soren: However, the point of preseeding is to /avoid/ interaction, and to skip it entirely. (03:16:54 PM) soren: Don't get me wrong.. (03:17:00 PM) soren: Preseed testing is super valuable. (03:17:23 PM) soren: Installing that way is a supported install method, so having this well tested is wicked cool and really important. (03:17:52 PM) soren: ...but I wanted to test the interactivity as well. (03:18:07 PM) soren: So, being the virtualisation geek that I am.. (03:18:10 PM) soren: I decided to use the KVM autotest framework to do the ISO testing. (03:18:18 PM) soren: Now, KVM autotest was designed to test KVM. (03:19:02 PM) soren: KVM developers use it to install a bunch of different operating systems and test things to make sure they didn't change anything in KVM that broke functionality in one of the guest operating systems. (03:19:16 PM) soren: What we want to do, though, is somewhat the opposite. (03:19:45 PM) soren: We assume that KVM works and instead want to test the operating system. (03:20:14 PM) soren: So, the KVM autotest framework works by runing a virtual machine.. (03:20:28 PM) soren: grabs a screenshot every second.. (03:20:47 PM) soren: ..and when the screenshot looks a particular way (e.g. when a particular dialog comes up), ...
MeetingLogs/devweek1001/AutoServerTests (last edited 2010-01-29 10:04:29 by i59F765F3)