Revision 4 as of 2010-01-27 19:05:30

Clear message

Dev Week -- Getting started with EC2 -- smoser -- Tue, Jan 26


(01:02:07 PM) smoser: Hello all
(01:02:16 PM) smoser: Ok, lets get started.
(01:02:22 PM) smoser: This is my first time doing one of these, so I ask for forgiveness in advance
(01:02:33 PM) smoser: I'm Scott Moser (
(01:02:42 PM) smoser: For the past few months I've been working on maintaining and improving Ubuntu's presense on Amazon's Cloud Offering, known as "EC2"
(01:02:56 PM) smoser: In this session, I hope to explain :
(01:03:01 PM) smoser:  * A little bit about what EC2 is (at least from my point of view)
(01:03:06 PM) smoser:  * What you need to do to use it
(01:03:13 PM) smoser:  * What you can do with it
(01:03:18 PM) smoser:  * Some warnings about issues you might find
(01:03:22 PM) smoser:  * Some miscellaneous things I'd like to mention
(01:03:31 PM) smoser: and then, have some time for questions
(01:03:37 PM) smoser:  * Questions
(01:04:14 PM) smoser: first off, I guess I'd like to kind of see a show of hands on who is interested in this session
(01:04:37 PM) smoser: if you are, simply a 'o/' or 'i am' ....
(01:05:04 PM) smoser: well, seeing none , we'll just go on
(01:05:10 PM) smoser: If you have questions along the way, please feel free to interrupt.  If it becomes too much, then we might have cjohnston help moderate.
(01:05:14 PM) cjohnston: There's quite a efw
(01:05:18 PM) cjohnston: quite a few
(01:05:40 PM) smoser: == A little bit about what EC2 is (at least from my point of view) ==
(01:05:51 PM) smoser: """Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers."""
(01:06:06 PM) smoser: What that means to me is:
(01:06:12 PM) smoser: * Virtual Servers, as many "clean instances" as you want, provisioned on your command
(01:06:25 PM) smoser: * ability to access permenant professional grade storage from inside the virtual servers (EBS or S3)
(01:06:52 PM) smoser: * access to fast network resources (much better than my cable modem)
(01:07:16 PM) smoser: You can use the above, and do whatever you want, then throw it away.
(01:07:27 PM) smoser: You get charged by the unit (hour/transfer/disk-size) for the resources as you are using them.
(01:08:00 PM) smoser: Depending on what your interest is, it may be tempting to simply calculate the charges for a month of m1.small (0.085 * 24 * 30 = $61.20), compare that to a virtual private server at and stop looking.
(01:08:22 PM) smoser: However, EC2 and "Cloud" is something quite different.  The biggest difference being the ability to start up and tear down instances in very little time.
(01:08:54 PM) smoser: Any questions so far?
(01:09:32 PM) cjohnston: < acicula> Question: define little, do they have spinning instances running or something?
(01:09:52 PM) smoser: acicula, 'little' is defined in terms of seconds , i'd say on average less than 100
(01:10:28 PM) smoser: i dont have hard numbers, and those vary based on time of day and load of ec2, and also (I think) popularity of the instance being requested
(01:10:45 PM) smoser: (the more popular an instance, the faster you get a copy)
(01:11:24 PM) smoser: moving on
(01:11:38 PM) smoser: === What you need to do to use it ===
(01:11:57 PM) smoser: Assuming you weren't completely turned off from my explanation above lets move on to how you can use this.
(01:12:27 PM) smoser: I'm really only going to cover command line usage via the ec2-api-tools, but there are several other ways to manage ec2
(01:12:37 PM) smoser: First thing you need is a credit card, and an AWS account with EC2 access.
(01:12:48 PM) smoser: The EC2Starters Guide ( has a good description on how to get signed up.  Make sure to generate, and download your X509 certificate, and key and keep them safe.
(01:13:13 PM) smoser: Next, you'll need the ec2 api tools. The latest version is available in Lucid and backports are available for karmic and hardy at . Install those tools by enabling multiverse and then:
(01:13:29 PM) smoser: $ apt-get install ec2-api-tools ec2-ami-tools
(01:14:08 PM) smoser: (note, for just "using" the cloud, the api-tools are probably enough. if you get interested in creating content for it, you will need to get the ec2-ami-tools)
(01:14:35 PM) mode (-o adiroiban ) by cjohnston
(01:14:39 PM) smoser: Commands I show will assume that you have EC2_PRIVATE_KEY and EC2_CERT in your environment.
(01:14:40 PM) mode (-o henninge ) by cjohnston
(01:14:59 PM) smoser: the EC2StartersGuide mentions how to set that up also
(01:15:29 PM) smoser: Access to instances "in the cloud" is via network only, and the primary method being ssh (at least here)
(01:15:37 PM) smoser: In order to access instances via ssh, you need to create a keypair.
(01:15:51 PM) smoser: This is a normal ssh private/public key, but amazon makes public portion available to the instance when you run it.
(01:16:29 PM) smoser: Create that keypair with:
(01:16:30 PM) smoser: $ ec2-add-keypair >
(01:16:46 PM) smoser: $ SSH_KEY=$PWD/
(01:17:00 PM) smoser: At this point, you're ready to go.  You can launch a new instance of the latest release of karmic on i386 with:
(01:17:13 PM) smoser: $ ec2-run-instances --instance-type m1.small --key ami-1515f67c
(01:17:54 PM) cjohnston: < bullgard45> QUESTION: What is an AWS account?
(01:18:24 PM) smoser: bullgard45, sorry, AWS Account is Amazon Web Services (
(01:18:48 PM) smoser: expanding on m1.small
(01:19:17 PM) cjohnston: < kamalmostafa> QUESTION: Please expound on "m1.small".
(01:19:44 PM) smoser: the above command says "give me a machine of type m1.small" and make the keypair that i just created able to access it.
(01:19:54 PM) smoser: m1.small is described at
(01:20:08 PM) smoser: and pricing information available at
(01:20:31 PM) smoser: m1.small is the cheapest, and is i386 virtual hardware
(01:21:34 PM) smoser: You will see output similar to the example at .
(01:21:44 PM) smoser: The magic 'ami-xxxxxx' number above is the image-id. You can see a list of available images via:
(01:21:49 PM) smoser: $ ec2-describe-images --all > images.list
(01:22:30 PM) smoser: I redirected output to a file because the list is quite long.  Another interface for looking at available machine images is
(01:23:08 PM) smoser: Official Ubuntu images are named such that you can easily identify them.  For more information on that see:
(01:23:55 PM) smoser: You'll note that after creation, you got an 'instance id' for the new instance (i-xxxxxx)
(01:24:01 PM) smoser: and that it was in 'pending' state.
(01:24:13 PM) smoser: At some point, your instance will move from 'pending' state to 'running' and will have a hostname associated with it.
(01:24:20 PM) smoser: you can watch the state with
(01:24:26 PM) smoser: $ ec2-describe-instances
(01:24:44 PM) smoser: Once you see 'running' and a hostname, then you can ssh to the instance
(01:24:52 PM) smoser: $ ssh -i ${SSH_KEY}
(01:25:11 PM) smoser: Ubuntu images accept only the 'ubuntu' user out of the box
(01:25:27 PM) smoser: and that user has passwordless sudo configured.
(01:25:55 PM) smoser: Hoorah! now you've got root access to your very own machine in the cloud, and it cost you $0.085 USD
(01:26:02 PM) smoser: now, you can shut that instance down with 'shutdown' or
(01:26:02 PM) smoser: $ ec2-terminate-instances i-xxxxxx
(01:26:28 PM) smoser: we'll pause for questions on the above
(01:26:37 PM) smoser: (i realize theres a lot for "getting started")
(01:27:19 PM) smoser: ok. going on then.
(01:27:22 PM) smoser: === What you can do with it ===
(01:27:32 PM) smoser: So, now that you've created this new machine and have access to it, what can you do?
(01:27:46 PM) smoser:  * Test : We make night nightly builds of the development release, and semi-nightly builds of previous releases available.
(01:28:13 PM) smoser: This makes it extremely easy to verify a change in a development release without any risk to your system
(01:28:23 PM) smoser:  * develop
(01:28:30 PM) smoser: If you don't have access to hardware supporting kvm, or just don't want to bother with keeping multiple images around of old releases or development, you can easily start up an instance, and start developing on it.
(01:28:55 PM) smoser: When you're done, you just throw it away.  A new one is only seconds away.
(01:29:03 PM) smoser: As an ubuntu developer, those two items above are the most interesting to me.  My laptop is not normally in a situation where it has a clean environment available for test or build.
(01:29:23 PM) smoser:  * Make your own images / devpay
(01:29:32 PM) smoser: You can make your own images and register them, and then boot your personal image.
(01:29:43 PM) smoser: You can "Make Money!" using DevPay (  If you add value to an image, enough that people will pay to use it, its easy to charge them for using their image.
(01:30:22 PM) smoser: I've not experimented at all with devpay, but it is a really easy way that you can make software appliances available to others and get paid!
(01:30:49 PM) smoser: I'll pause again for questions...
(01:31:08 PM) smoser: === Some warnings about issues you might find ===
(01:31:18 PM) smoser: The thing that most sticks out as different from other environments is that the root filesystem is *gone* once you shut a system down.
(01:31:42 PM) smoser: Amazon makes basically no promises for data stored on the "ephemeral storage" of an instance.
(01:32:03 PM) smoser: hardware failure that causes an instance to die would cause data there to be lost
(01:32:14 PM) smoser: If you want to keep data that you create on an instance, you have a few options:
(01:32:14 PM) smoser:  * Use S3 storage (from amazon)
(01:32:14 PM) smoser:  * Use EBS (and connect an ebs volume to your instance with ec2-attach-volume)
(01:32:14 PM) smoser:  * Use a (new) EBS Root instance.
(01:32:28 PM) smoser: In this new type of instance, your / is on a EBS volume, and you can stop your instance, and access that volume from another instance.
(01:33:03 PM) smoser: EBS Root Instances make the EC2 cloud much more similar to a "pay as you go Virtual Private Server"
(01:33:09 PM) smoser: We are planning on creating EBS volume IDs for Lucid, and Eric Hammond has made EBS boot AMIs available for Karmic and Hardy (
(01:34:05 PM) smoser: The "best practice" for using ec2 instances is to make all setup of your environment automated
(01:34:18 PM) smoser: so that you can start up new instances on failure or demand.
(01:34:49 PM) smoser: pause again. questions ?
(01:35:22 PM) smoser: === Some Miscellaneous Things I'd like to mention ===
(01:35:31 PM) smoser:  * User Data
(01:35:44 PM) smoser: Booting an exact copy of an instance over and over again is hardly any fun, and arguably not much use
(01:35:52 PM) smoser: In almost all cases, you will want to make some modifications to the instance immediately after boot.
(01:36:12 PM) smoser: You can do this by waiting util the machine is running, and ssh'ing in and running a set of commands.
(01:36:27 PM) smoser: But that may be less than idea.
(01:36:32 PM) smoser: This is where "User Data" comes in.  User data is provided to the instance via the '--user-data' or '--user-data-file' arguments to ec2-run-instances.
(01:37:10 PM) smoser: When the instance boots, it reads the user-data an can change its behavior based on that content.
(01:37:15 PM) smoser: When an ubuntu instance starts, if user-data begins with '#!' then it will be executed like an rc.local script.
(01:38:07 PM) smoser: This means that you can put together a shell script to turn the "base AMI" into your personal development box and pass that script as user-data to the instance on startup.  Then, when you ssh in, it will be all set up.
(01:38:23 PM) smoser: This is obviously quite useful for all sorts of purposes
(01:38:31 PM) smoser: We're adding features to how user-data is handled for lucid ( and for more information).
(01:38:48 PM) smoser: The big new feature will be the ability to configure the image on first boot via "config"  syntax rather than a shell script.
(01:39:06 PM) smoser:  * Spot Instances
(01:39:18 PM) smoser: If you're like me, you're cheap.  Spot instances allow you to run an instance in basically the same way as "normal instances" but at a reduced price.
(01:39:38 PM) smoser: The catch is that if there is high demand, your instance may be terminated.
(01:39:50 PM) smoser: I've began using spot instances for most of my tests.
(01:40:25 PM) smoser: When you make a request for a "spot instance" you state the maximum you will pay for that instance.  The rate you are charged varies based on demand for those resources.
(01:40:40 PM) smoser: if the rate goes above your maximum, your instance is terminated.
(01:40:57 PM) smoser:  * Desktop In the Cloud
(01:41:18 PM) smoser: In the lucid cycle, the Desktop team wanted to take advantage of the easily created test/reproduce/verify environment
(01:41:29 PM) smoser: so we started creating "desktop in the cloud" builds.
(01:41:56 PM) smoser: Those are created daily and registered to ec2
(01:42:06 PM) smoser: for relevant AMIs.
(01:42:51 PM) smoser: These images have a full gnome desktop installed and can be accessed via an NX client.  If you've never used NX before, it is surprising how well it performs.  Literally, you may forget that you are using a remote desktop.
(01:43:34 PM) smoser: has information on how to use these.
(01:43:52 PM) smoser: QUESTION: smoser, when you use spot instances for your testing, what rate do you typically offer to pay, and how often do your instances get terminated?
(01:43:52 PM) cjohnston: < kamalmostafa> QUESTION: smoser, when you use spot instances for your testing, what rate do you typically offer to pay, and how often do your instances get terminated?
(01:44:03 PM) smoser: I've never actually had one terminated.
(01:44:31 PM) smoser: has graphs of spot prices historically
(01:44:56 PM) smoser: and you can get that same data with : ec2-describe-spot-price-history
(01:45:29 PM) smoser: what I do is take the high price available and raise it by 10%
(01:45:42 PM) smoser: you'll notice spikes in the graphs at cloudexchange
(01:46:09 PM) smoser: this is where (I think) amazon had higher demand for non-spot instances and explicitly kicked off most spot instances
(01:46:42 PM) smoser: On average, spot instances run ~ 40% of the regular price.
(01:47:04 PM) smoser: This is all I had prepared, so the floor is open to questions
(01:47:06 PM) smoser: Questions
(01:48:08 PM) smoser: oh. One other thing I should mention
(01:48:16 PM) smoser: i can't believe i didn't plug UEC
(01:48:53 PM) smoser: UEC (Ubuntu Enterprise Cloud) is based on Eucalyptus and allows you to easily run a ec2 compatible cloud inside your own data center.
(01:49:07 PM) smoser: This has all sorts of neat uses.
(01:49:13 PM) cjohnston: < zul> QUESTION: how are the images built?
(01:49:23 PM) smoser: zul, thanks for asking :)
(01:49:52 PM) smoser: We build our images using vmbuilder
(01:50:20 PM) smoser: The complete setup for building identical images is available starting at :
(01:50:43 PM) smoser: and code we use to publish the builds is at
(01:51:13 PM) cjohnston: < dhillon-v10> smoser: how is the upcomming HTML5 going to affect the server cloud
(01:51:32 PM) smoser: hm.. I dont know.
(01:51:47 PM) smoser: Do you think it will increase demand for server resources? or reduce demand?
(01:52:19 PM) smoser: I guess I need more context on the question
(01:52:28 PM) cjohnston: < dhillon-v10> smoser: probably decrease the demand because now users can do more stuff offline
(01:53:09 PM) smoser: Maybe, but by the same note, the more you depend on "the cloud" the more important reliable resources are
(01:54:12 PM) smoser: Well, I guess to wrap up...
(01:54:14 PM) cjohnston: < dhillon-v10> smoser: possibly :)
(01:54:24 PM) cjohnston: < kamalmostafa> QUESTION: Can EC2 machine images be used by UEC installations?  (either with our without incurring charges at Amazon)?
(01:54:40 PM) smoser: kamalmostafa, yes
(01:55:08 PM) smoser: The only real difference is that UEC uses KVM and EC2 is xen based.
(01:55:31 PM) smoser: so the end result is that you'll have a different kernel, but ubuntu creates "ubuntu sauce" kernels for both varieties
(01:56:02 PM) smoser: i see UEC as very useful for developing machine images that can then be deployed publically on ec2
(01:56:08 PM) smoser: (and for other purposes too)
(01:56:21 PM) cjohnston: < kamalmostafa> QUESTION: Does one need an Amazon AWS account at all, in order to use those images in a UEC installation?
(01:56:22 PM) smoser: kamalmostafa, no
(01:56:37 PM) smoser: UEC is completely stand alone.
(01:56:51 PM) smoser: The requirements for it are basically just having hardware available.
(01:56:59 PM) smoser: you can create a "cloud" with 2 systems
(01:57:17 PM) smoser: and the ubuntu-server ISOs have an easy option for doing so.
(01:57:54 PM) smoser: I've probably represented UEC poorly here... maybe next developer week we need a UEC class
(01:58:31 PM) smoser: wrapping up...
(01:59:14 PM) cjohnston: < lbrinkma> QUESTION: Whats the api command for creating a spot instance?
(01:59:15 PM) smoser: If you're a ubuntu developer, I strongly urge you to take a look at using EC2 for easily running releases of ubuntu that you have no worry about destroying.
(01:59:40 PM) smoser: ec2-request-spot-instances is the api command
(01:59:51 PM) smoser: it basically looks like 'ec2-run-instances' with some extra args.
(02:00:15 PM) smoser: I've done sone work to make a wrapper 'ec2-cheap-run-instances'.  It needs some work, but then is a dropin replacement for ec2-run-instances
(02:00:26 PM) smoser:;a=blob;f=ec2/ec2-cheap-run-instances;hb=HEAD
(02:01:50 PM) smoser: If you're not a ubuntu developer but want to do "real stuff" with EC2, there are loads of people running businesses and servers on EC2's cloud.  I urge you to look into it further, and,of course, I urge you to start with the Official Ubuntu Images
(02:02:39 PM) smoser: If you have questions or need assistance, please feel free to ping me  #ubuntu-server or elsewhere.
(02:02:43 PM) cjohnston: Thanks smoser !