VMTestEnvironments
Dev Week -- Using VMBuilder to create tests environments -- Soren Hansen & Nick Barcet -- Wed, Jan 21
Note: this transcript is also available a bit reformated on an external blog.
UTC -4 (EST)
(03:02:46 PM) JontheEchidna: Next up is g VMBuilder to create tests environments -- Søren Hansen and Nicolas Barcet will entertain you with virtual machines and the fantastic vmbuilder. Need a clean test environment for something? Don't want to run the latest development release on actual hardware yet? These two fine men have the answer for you. (03:03:05 PM) nijaba: Welcome everyone and thanks for attending this presentation about (03:03:14 PM) nijaba: __ ___ __ (03:03:14 PM) nijaba: /\ \ __ /\_ \ /\ \ (03:03:14 PM) nijaba: __ __ ___ ___\ \ \____ __ __/\_\\//\ \ \_\ \ __ _ __ (03:03:15 PM) nijaba: \ \ \_/ |/\ \/\ \/\ \ \ \L\ \ \ \_\ \ \ \ \_\ \_/\ \L\ \/\ __/\ \ \/ (03:03:16 PM) nijaba: \ \___/ \ \_\ \_\ \_\ \_,__/\ \____/\ \_\/\____\ \___,_\ \____\\ \_\ (03:03:17 PM) nijaba: \/__/ \/_/\/_/\/_/\/___/ \/___/ \/_/\/____/\/__,_ /\/____/ \/_/ (03:03:31 PM) ***nxvl waver (03:03:33 PM) nijaba: Originally Soren Hansen, the author and maintainer of vmbuilder was supposed to run this presentation with me, but he unfortunately got a bad flu and I'll therefore will have to do this alone. (03:03:34 PM) nxvl: waves* (03:03:59 PM) nijaba: Well not completely alone, I bet mathiaz and zul are around to make sure I don't say anything stupid ;) (03:04:12 PM) ***zul waves (03:04:19 PM) Koon: if it's really stupid, I will make sure everyone notices. (03:04:20 PM) nijaba: soren, what is vmbuilder? (you may ask) (03:04:45 PM) nijaba: ehe, I meant so not soren ^ :) (03:04:59 PM) nijaba: vmbuilder is a python tool that allows to create virtual machine images on the fly, without the need to start an installer in an hypervisor. (03:05:25 PM) nijaba: vmbuilder can create images for KVM, Xen, VMWare and for Amazon EC2. (03:05:43 PM) jmarsden|work: And soon hopefully for virtualbox too... (03:05:46 PM) nijaba: On a reasonably powerfull machine with a local package cache (for example apt-proxy or apt-cacher), the creation of a minimal virtual server (aka JeOS) can take less than a minute. (03:06:12 PM) nijaba: jmarsden|work: please use #ubuntu-classroom-chat for comments (03:06:35 PM) nijaba: As a python tool, it is first of all a library with a very powerfull plugin architecture. (03:06:49 PM) JontheEchidna left the room. (03:06:56 PM) nijaba: One of the plugins that is provided is a CLI (command line interface), and anyone could write another plugin to provide a GTK interface or use the vmbuilder library to create a web service. (03:07:25 PM) nijaba: Apart for the front end plugin mechanism, there are 2 other types of plugins that are used in vmbuilder: Hypervisors and Distributions. (03:07:53 PM) nijaba: The distribution plugin would allow for someone to easily extend vmbuilder for other distributions, but at this time the only available one is Ubuntu. (03:08:35 PM) nijaba: The same is true for the hypervisor plugin, and it seems that jmarsden|work is voluteering some work :) (03:08:48 PM) nijaba: Contributions are always welcome! (03:09:15 PM) nijaba: The first implementation of vmbuilder was on hardy as a shell script that used to be called ubuntu-vm-builder. (03:09:41 PM) nijaba: When Soren rewrote in python for intrepid, the source package was renamed python-vm-builder and contains a binary package called ubuntu-vm-builder which is just a wrapper to the old CLI to maintain upward compatibility. (03:10:18 PM) nijaba: Now that we have covered what vmbuilder is (in a nutshell), let's explore what you could use it for as a developer. (03:10:28 PM) weboide_ is now known as weboide (03:10:43 PM) nijaba: A couple scenarios come to mind, but I'm sure there are many others. (03:11:06 PM) nijaba: 1/let's imagine that you want to test something (a program that you wrote, an upgrade, a bug) in a clean environement. (03:11:26 PM) nijaba: chroot is generally great for this, but is not really what we could call a complete environment. (03:11:48 PM) nijaba: Using vmbuilder, you can generate a virtual machine on the fly that contains whichever package and configuration that you want almost as quickly as you would generate a chroot environment. (03:12:20 PM) nijaba: You can then start the machine in your favorite hypervisor and start testing. (03:12:52 PM) nijaba: Some of the great options vmbuilder provides are the ability to chose wich series you want (dapper, gutsy, hardy, intrepid, ...) and, to the difference of pre-built images that you would duplicate, is building machines from the latest versions of the packages in the repositories. (03:13:28 PM) nijaba: Since vmbuilder does not require any user interaction to run, this can be scripted and made part of build or test process. (03:14:06 PM) nijaba: 2/let's imagine that you want to deliver your package/packages/application/integration as a pre-built virtual machine for various hypervisor (providing so called "virtual-appliances", for example). (03:14:48 PM) nijaba: The traditional way would imply creating reference virtual images for each target environements and each time you updated your code, after building your package you would have to: (03:14:56 PM) nijaba: a- start each vm (03:15:05 PM) nijaba: b- update your package in each vm manually (03:15:20 PM) nijaba: c- clean up the vm (03:15:29 PM) nijaba: d- save it (03:15:44 PM) nijaba: e- send it to testers/users (03:16:11 PM) nijaba: Since vmbuilder knows about your various Hypervisors, it is very easy to integrate this work in your build scripts so that your virtual images are almost ready to ship once your build process is completed. (03:16:43 PM) nijaba: To make things even easier, vmbuilder includes three "sub-scripts" features that can be specified: (03:17:07 PM) nijaba: a- an "exec" script that is called during the machine build process just after finishing the the installation. The script can chroot within the target machine and make additional changes. (03:17:43 PM) nijaba: b- a "first-boot" script that is copied to the virtual machine and executed in batch mode the first time the vm will boot (03:18:04 PM) nijaba: c- a "first-login" script that is copied as well to the vm and executed as the user the first time someone logs-in to the vm (and therefore allows to perform some final interactive setup). (03:18:34 PM) nijaba: ok, now that I have covered the main use cases, I'll try to answer the first few questions before we move on (03:18:53 PM) nijaba: QUESTION: What do I need to use KVM? (03:19:21 PM) nijaba: You need a processor that supports hardware virtualization (03:19:52 PM) nijaba: there is a detailed step by step procedure to test this on http://help.ubuntu.com/communtity/KVM (03:20:05 PM) nijaba: err community (03:20:38 PM) nijaba: QUESTION: are there plans to also support qemu/kqemu? (03:21:08 PM) nijaba: That really should no be too hard to had, but it is not on any official roadmap, AFAIK (03:21:18 PM) nijaba: contributions, again, are very welcome (03:21:43 PM) nijaba: QUESTION: vmbuilder on gutsy? (03:22:06 PM) nijaba: unfortunately not. ubuntu-vm-builder was first developped on hardy (03:22:25 PM) nijaba: we badly need kpartx for it, and it is not available before hardy (03:23:02 PM) nijaba: QUESTION: which one should I choose between KVM, Xen, VMWare and for Amazon EC2 (if I got a machine that supports hardware virtualization and I want to do ubuntu package testing for instance) ??? (03:23:21 PM) nijaba: it all really depends on your use case (03:23:43 PM) nijaba: VMWARE (esx) is certainly the most complete virtualization solution available today (03:24:05 PM) nijaba: but it has some hard dependency, and is not really free software (03:24:24 PM) nijaba: Xen is nice if you processor does not have hardware virtualization extension (03:24:51 PM) nijaba: but then it will only provide paravirtualized os virtualization (03:25:07 PM) nijaba: and it is a BULKY thing to maintain in a distro (03:25:34 PM) nijaba: so that's mainly why, if your use case is server virtualization, we prefer KVM (03:25:47 PM) nijaba: ok, lets move on (03:26:01 PM) nijaba: To start using vmbuilder, you need to first install its package: (03:26:09 PM) nijaba: $sudo apt-get install python-vm-builder (03:26:41 PM) nijaba: note that if you want to create EC2 images, you will need to install the python-vm-builder-ec2, which, for licence reasons on the AMI tools, had to be made a separate package and placed into multiverse. (03:26:58 PM) nijaba: Once the install is complete, the first thing to do is to have a look a the help. (03:27:10 PM) nijaba: As I said before, python-vm-builder uses a plugin mechanism. (03:27:30 PM) nijaba: This implies that you will have different sets of options depending on the hypervisor and distribution that you pick. (03:28:04 PM) nijaba: This is the reason why the man page or a basic 'sudo vmbuilder --help' will only list options that are common to all hypervisors or distributions. (03:28:32 PM) nijaba: To get all options displayed, you need to specify your target hypervisor and distribution. For example try doing: (03:28:39 PM) nijaba: $sudo vmbuilder kvm ubuntu --help (03:28:47 PM) nijaba: or (03:28:57 PM) nijaba: $sudo vmbuilder xen ubuntu --help (03:29:15 PM) nijaba: and notice the additional options provided in each case. (03:29:39 PM) nijaba: So now, assuming that you all are going to use vmbuilder intensively, let's start setting up a package caching system. (03:29:58 PM) nijaba: Soren likes apt-cacher while I've preferred apt-proxy. (03:30:23 PM) nijaba: Since the later is what I know and is quite simple to enable, I'll invite you to: (03:30:33 PM) nijaba: $sudo apt-get install apt-proxy (03:30:50 PM) nijaba: once this is done, apt-proxy will immediately start listenning on all your interfaces on port 9999. (03:31:22 PM) nijaba: There are multiple options you can set on /etc/apt-proxy/apt-proxy.conf, but the default will work very nicely for our purpose. (03:32:39 PM) nijaba: In order for apt-proxy to be used by vmbuilder automatically, we could very well (03:32:39 PM) nijaba: add the option "--install-mirror http://127.0.0.1:9999/ubuntu" each time we would invoke vmbuilder, but since I am quite lazy, I have defined this as a permanent option in my ~/.vmbuilder.cfg file by putting in the following two lines: (03:33:05 PM) nijaba: [ubuntu] (03:33:05 PM) nijaba: install_mirror = http://127.0.0.1:9999/ubuntu (03:33:48 PM) nijaba: in fact, any command line option that can be passed to vmbuilder can added to this config file, and you can even specify an additional config file that uses the same format using the --config option. (03:34:31 PM) nijaba: not that the format of the config file changed between hardy and interpid... so I assume your are all on intrepid or jaunty here (03:35:01 PM) nijaba: Another option that speeds up the creation of vm is the --tmpfs one, which will use a ramdisk to store temporary file. (03:35:35 PM) nijaba: I have added in my ~/.vmbuilder.cfg file the following 2 lines: (03:35:49 PM) nijaba: [DEFAULT] (03:35:49 PM) nijaba: tmpfs = - (03:36:06 PM) nijaba: how is everyone doing so far? (03:36:39 PM) ***rmcbride nods (03:36:53 PM) nijaba: ok, so we are now ready to start building our first image. (03:37:21 PM) nijaba: For this example we'll build and equivalent of an Intrepid JeOS using the following command: (03:38:18 PM) nijaba: $sudo vmbuilder kvm ubuntu --suite intrepid --flavour virtual --arch i386 --verbose (03:38:50 PM) nijaba: I do not think the above options need much comments, do they? (03:39:26 PM) nijaba: The first time you will build a machine for a new suite, it will obviously take a bit longer as apt-proxy is not loaded. (03:39:58 PM) nijaba: While your vm is building, let's open the floor to questions. (03:40:13 PM) nijaba: In the absence of Soren, I may not be able to answer some of them (03:40:34 PM) nijaba: in which case I'll make sure to write them down and provide you with valid answers on my blog as soon as I can get them (03:40:54 PM) nijaba: QUESTION: sry to dwell on this but why could KVM only be used for server testing? cant I run X.org inside a KVM? (03:41:14 PM) nijaba: well, sure you can run X.org inside a KVM (03:41:47 PM) nijaba: but the interface has not really been tuned to provide you with the best experience possible for full desktop vritualization (03:42:11 PM) nijaba: but, I do use kvm for testing gnome once in a while (03:42:28 PM) nijaba: and apart from a few quirk, it is quite useable (03:42:48 PM) nijaba: I would not recommend it for any real usage on a daily basis though (03:43:26 PM) nijaba: qUESTION: what I have done in the past is boot from an .iso and then use rtegular tools to make mods (03:43:49 PM) nijaba: this is a CLI approach based on, listing all the pkgs you want to install in a text file, for example? (03:44:21 PM) nijaba: (please try to write your question starting with QUESTION without breaks) (03:44:41 PM) nijaba: yes, this is the general approach, yes (03:44:58 PM) nijaba: UESTION: when running vmbuilder, it printed a bunch of these: "Can not write log, openpty() failed (/dev/pts not mounted?)" is that a problem? (03:45:33 PM) nijaba: no, actually the reason why you saw those is because we enabled the --verbose option (03:45:58 PM) nijaba: but, unless vmbuilder stops before sayng it is done, these error are just transient (03:46:20 PM) nijaba: QUESTION what about custom downloads.. eg not debian/ubuntu packages.. not covered by this approach? (03:46:45 PM) nijaba: this is the reason why we have multiple scripts options (03:46:53 PM) nijaba: as I described earlier (03:47:07 PM) nijaba: you could have a --exec script (03:47:17 PM) nijaba: that download something with wget (03:47:24 PM) nijaba: and untars it wherever you want (03:48:08 PM) nijaba: ok, is everyone done building their first vm? (03:49:11 PM) ***sebp nods (03:49:30 PM) ***Quintasan nods (03:49:34 PM) nijaba: another very inresting part of vmbuilder are templates (03:50:02 PM) nijaba: have a look at what is inside /etc/vmbuilder/ (03:50:49 PM) nijaba: each of these directory contains base templates for file that we'll use for configuration during the build process (03:51:32 PM) nijaba: for example, to enable bridging in our vm, we'll need to modify the libvirt template (03:51:45 PM) nijaba: $mkdir -p mytemplates/libvirt (03:51:56 PM) nijaba: $cp /etc/vmbuilder/libvirt/* mytemplates/libvirt/ (03:52:17 PM) nijaba: we can then edit mytemplates/libvirt/libvirtxml.tmpl (03:52:43 PM) nijaba: and change the interface section of it to: (03:52:46 PM) nijaba: <interface type='bridge'> (03:52:46 PM) nijaba: <source bridge='br0'/> (03:52:46 PM) nijaba: </interface> (03:53:29 PM) nijaba: and then specify which template set to use by adding the following to our command line: (03:53:33 PM) nijaba: --templates mytemplates (03:54:19 PM) nijaba: QUESTION: how can I start the kvm I just built? (03:54:36 PM) nijaba: this brings me to the --libvirt option (03:54:48 PM) nijaba: assuming that libvirt is installed and running (03:55:25 PM) nijaba: (if it is not, have a look at https://help.ubuntu.com/community/KVM/Installation ) (03:56:03 PM) nijaba: then you can pass "--libvirt qemu:///system" to the command line (03:56:35 PM) nijaba: which will automatically add the generated vm as a livirt domain (03:56:46 PM) nijaba: which can be started immediately (03:57:48 PM) nijaba: https://help.ubuntu.com/community/KVM/Managing#From%20the%20shell gives an overview of how to manage running vm (03:58:28 PM) nijaba: I you do not want to use libvirt, see https://help.ubuntu.com/community/KVM/Directly (03:59:01 PM) nijaba: QUESTION: kvm is a standard'ish ubuntu VM I take it? (03:59:14 PM) nijaba: yes, it is the official virtualization technology since hardy (03:59:48 PM) nijaba: have a look at http://www.ubuntu.com/products/whatisubuntu/serveredition/technologies/virtualization (04:00:08 PM) nijaba: For additional reference, apart for the --help, I would like to point you to a tutorial I have writen on using VMBuilder which is available at https://help.ubuntu.com/community/JeOSVMBuilder (04:00:29 PM) nijaba: Hope this was a useful session for most of you and feel free to drop by #ubuntu-virt at any time if you have additional questions or just want to say hi :) (04:01:02 PM) Quintasan: Thanks nijaba, this is very helpful :) (04:01:25 PM) jdstrand: thanks nijaba! (04:01:41 PM) nijaba: Thanks for attending and asking very good questions everyone :) (04:01:54 PM) thekorn: thanks nijaba (04:01:54 PM) nijaba: I'm glad I could answer most of them (04:02:25 PM) mnemo: nijaba: hey, I started run.sh and I get to the prompt... but what is the password to my new VM ?? (04:02:30 PM) thekorn left the room (quit: Remote closed the connection). (04:02:49 PM) nijaba: mnemo: by default user and password are ubuntu (04:03:00 PM) mnemo: sweet (04:03:01 PM) mnemo: thanks (04:03:03 PM) nijaba: of course there is a command line option to change that too :)
MeetingLogs/devweek0901/VMTestEnvironments (last edited 2009-01-22 02:03:58 by mx)