EC2

Revision 5 as of 2010-01-29 10:06:49

Clear message

Dev Week -- Getting started with EC2 -- smoser -- Tue, Jan 26

UTC

   1 [18:02] <smoser> Hello all
   2 [18:02] <smoser> Ok, lets get started.
   3 [18:02] <smoser> This is my first time doing one of these, so I ask for forgiveness in advance
   4 [18:02] <smoser> I'm Scott Moser (https://launchpad.net/~smoser).
   5 [18:02] <smoser> For the past few months I've been working on maintaining and improving Ubuntu's presense on Amazon's Cloud Offering, known as "EC2"
   6 [18:02] <smoser> In this session, I hope to explain :
   7 [18:03] <smoser>  * A little bit about what EC2 is (at least from my point of view)
   8 [18:03] <smoser>  * What you need to do to use it
   9 [18:03] <smoser>  * What you can do with it
  10 [18:03] <smoser>  * Some warnings about issues you might find
  11 [18:03] <smoser>  * Some miscellaneous things I'd like to mention
  12 [18:03] <smoser> and then, have some time for questions
  13 [18:03] <smoser>  * Questions
  14 [18:04] <smoser> first off, I guess I'd like to kind of see a show of hands on who is interested in this session
  15 [18:04] <smoser> if you are, simply a 'o/' or 'i am' ....
  16 [18:05] <smoser> well, seeing none , we'll just go on
  17 [18:05] <smoser> If you have questions along the way, please feel free to interrupt.  If it becomes too much, then we might have cjohnston help moderate.
  18 [18:05] <cjohnston> There's quite a efw
  19 [18:05] <cjohnston> quite a few
  20 [18:05] <smoser> == A little bit about what EC2 is (at least from my point of view) ==
  21 [18:05] <smoser> """Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers."""
  22 [18:06] <smoser> What that means to me is:
  23 [18:06] <smoser> * Virtual Servers, as many "clean instances" as you want, provisioned on your command
  24 [18:06] <smoser> * ability to access permenant professional grade storage from inside the virtual servers (EBS or S3)
  25 [18:06] <smoser> * access to fast network resources (much better than my cable modem)
  26 [18:07] <smoser> You can use the above, and do whatever you want, then throw it away.
  27 [18:07] <smoser> You get charged by the unit (hour/transfer/disk-size) for the resources as you are using them.
  28 [18:08] <smoser> Depending on what your interest is, it may be tempting to simply calculate the charges for a month of m1.small (0.085 * 24 * 30 = $61.20), compare that to a virtual private server at http://www.linode.com and stop looking.
  29 [18:08] <smoser> However, EC2 and "Cloud" is something quite different.  The biggest difference being the ability to start up and tear down instances in very little time.
  30 [18:08] <smoser> Any questions so far?
  31 [18:09] <cjohnston> < acicula> Question: define little, do they have spinning instances running or something?
  32 [18:09] <smoser> acicula, 'little' is defined in terms of seconds , i'd say on average less than 100
  33 [18:10] <smoser> i dont have hard numbers, and those vary based on time of day and load of ec2, and also (I think) popularity of the instance being requested
  34 [18:10] <smoser> (the more popular an instance, the faster you get a copy)
  35 [18:11] <smoser> moving on
  36 [18:11] <smoser> === What you need to do to use it ===
  37 [18:11] <smoser> Assuming you weren't completely turned off from my explanation above lets move on to how you can use this.
  38 [18:12] <smoser> I'm really only going to cover command line usage via the ec2-api-tools, but there are several other ways to manage ec2
  39 [18:12] <smoser> First thing you need is a credit card, and an AWS account with EC2 access.
  40 [18:12] <smoser> The EC2Starters Guide (https://help.ubuntu.com/community/EC2StartersGuide) has a good description on how to get signed up.  Make sure to generate, and download your X509 certificate, and key and keep them safe.
  41 [18:13] <smoser> Next, you'll need the ec2 api tools. The latest version is available in Lucid and backports are available for karmic and hardy at https://launchpad.net/~ubuntu-on-ec2/+archive/ec2-tools . Install those tools by enabling multiverse and then:
  42 [18:13] <smoser> $ apt-get install ec2-api-tools ec2-ami-tools
  43 [18:14] <smoser> (note, for just "using" the cloud, the api-tools are probably enough. if you get interested in creating content for it, you will need to get the ec2-ami-tools)
  44 [18:14] <smoser> Commands I show will assume that you have EC2_PRIVATE_KEY and EC2_CERT in your environment.
  45 [18:14] <smoser> the EC2StartersGuide mentions how to set that up also
  46 [18:15] <smoser> Access to instances "in the cloud" is via network only, and the primary method being ssh (at least here)
  47 [18:15] <smoser> In order to access instances via ssh, you need to create a keypair.
  48 [18:15] <smoser> This is a normal ssh private/public key, but amazon makes public portion available to the instance when you run it.
  49 [18:16] <smoser> Create that keypair with:
  50 [18:16] <smoser> $ ec2-add-keypair ec2-keypair.us-east-1 > ec2-keypair.us-east-1.pem
  51 [18:16] <smoser> $ SSH_KEY=$PWD/ec2-keypair.us-east-1.pem
  52 [18:17] <smoser> At this point, you're ready to go.  You can launch a new instance of the latest release of karmic on i386 with:
  53 [18:17] <smoser> $ ec2-run-instances --instance-type m1.small --key ec2-keypair.us-east-1 ami-1515f67c
  54 [18:17] <cjohnston> < bullgard45> QUESTION: What is an AWS account?
  55 [18:18] <smoser> bullgard45, sorry, AWS Account is Amazon Web Services (http://aws.amazon.com/)
  56 [18:18] <smoser> expanding on m1.small
  57 [18:19] <cjohnston> < kamalmostafa> QUESTION: Please expound on "m1.small".
  58 [18:19] <smoser> the above command says "give me a machine of type m1.small" and make the keypair that i just created able to access it.
  59 [18:19] <smoser> m1.small is described at http://aws.amazon.com/ec2/instance-types/
  60 [18:20] <smoser> and pricing information available at http://aws.amazon.com/ec2/#pricing
  61 [18:20] <smoser> m1.small is the cheapest, and is i386 virtual hardware
  62 [18:21] <smoser> You will see output similar to the example at https://wiki.ubuntu.com/UbuntuDeveloperWeek/Sessions/GettingStartedWithEC2 .
  63 [18:21] <smoser> The magic 'ami-xxxxxx' number above is the image-id. You can see a list of available images via:
  64 [18:21] <smoser> $ ec2-describe-images --all > images.list
  65 [18:22] <smoser> I redirected output to a file because the list is quite long.  Another interface for looking at available machine images is http://thecloudmarket.com/
  66 [18:23] <smoser> Official Ubuntu images are named such that you can easily identify them.  For more information on that see: https://wiki.ubuntu.com/UEC/Images/NamingConvention
  67 [18:23] <smoser> You'll note that after creation, you got an 'instance id' for the new instance (i-xxxxxx)
  68 [18:24] <smoser> and that it was in 'pending' state.
  69 [18:24] <smoser> At some point, your instance will move from 'pending' state to 'running' and will have a hostname associated with it.
  70 [18:24] <smoser> you can watch the state with
  71 [18:24] <smoser> $ ec2-describe-instances
  72 [18:24] <smoser> Once you see 'running' and a hostname, then you can ssh to the instance
  73 [18:24] <smoser> $ ssh -i ${SSH_KEY} ubuntu@ec2-174-129-61-44.compute-1.amazonaws.com
  74 [18:25] <smoser> Ubuntu images accept only the 'ubuntu' user out of the box
  75 [18:25] <smoser> and that user has passwordless sudo configured.
  76 [18:25] <smoser> Hoorah! now you've got root access to your very own machine in the cloud, and it cost you $0.085 USD
  77 [18:26] <smoser> now, you can shut that instance down with 'shutdown' or
  78 [18:26] <smoser> $ ec2-terminate-instances i-xxxxxx
  79 [18:26] <smoser> we'll pause for questions on the above
  80 [18:26] <smoser> (i realize theres a lot for "getting started")
  81 [18:27] <smoser> ok. going on then.
  82 [18:27] <smoser> === What you can do with it ===
  83 [18:27] <smoser> So, now that you've created this new machine and have access to it, what can you do?
  84 [18:27] <smoser>  * Test : We make night nightly builds of the development release, and semi-nightly builds of previous releases available.
  85 [18:28] <smoser> This makes it extremely easy to verify a change in a development release without any risk to your system
  86 [18:28] <smoser>  * develop
  87 [18:28] <smoser> If you don't have access to hardware supporting kvm, or just don't want to bother with keeping multiple images around of old releases or development, you can easily start up an instance, and start developing on it.
  88 [18:28] <smoser> When you're done, you just throw it away.  A new one is only seconds away.
  89 [18:29] <smoser> As an ubuntu developer, those two items above are the most interesting to me.  My laptop is not normally in a situation where it has a clean environment available for test or build.
  90 [18:29] <smoser>  * Make your own images / devpay
  91 [18:29] <smoser> You can make your own images and register them, and then boot your personal image.
  92 [18:29] <smoser> You can "Make Money!" using DevPay (http://aws.amazon.com/devpay/).  If you add value to an image, enough that people will pay to use it, its easy to charge them for using their image.
  93 [18:30] <smoser> I've not experimented at all with devpay, but it is a really easy way that you can make software appliances available to others and get paid!
  94 [18:30] <smoser> I'll pause again for questions...
  95 [18:31] <smoser> === Some warnings about issues you might find ===
  96 [18:31] <smoser> The thing that most sticks out as different from other environments is that the root filesystem is *gone* once you shut a system down.
  97 [18:31] <smoser> Amazon makes basically no promises for data stored on the "ephemeral storage" of an instance.
  98 [18:32] <smoser> hardware failure that causes an instance to die would cause data there to be lost
  99 [18:32] <smoser> If you want to keep data that you create on an instance, you have a few options:
 100 [18:32] <smoser>  * Use S3 storage (from amazon)
 101 [18:32] <smoser>  * Use EBS (and connect an ebs volume to your instance with ec2-attach-volume)
 102 [18:32] <smoser>  * Use a (new) EBS Root instance.
 103 [18:32] <smoser> In this new type of instance, your / is on a EBS volume, and you can stop your instance, and access that volume from another instance.
 104 [18:33] <smoser> EBS Root Instances make the EC2 cloud much more similar to a "pay as you go Virtual Private Server"
 105 [18:33] <smoser> We are planning on creating EBS volume IDs for Lucid, and Eric Hammond has made EBS boot AMIs available for Karmic and Hardy (http://alestic.com/2010/01/public-ebs-boot-amis-for-ubuntu-on-amazon-ec2)
 106 [18:34] <smoser> The "best practice" for using ec2 instances is to make all setup of your environment automated
 107 [18:34] <smoser> so that you can start up new instances on failure or demand.
 108 [18:34] <smoser> pause again. questions ?
 109 [18:35] <smoser> === Some Miscellaneous Things I'd like to mention ===
 110 [18:35] <smoser>  * User Data
 111 [18:35] <smoser> Booting an exact copy of an instance over and over again is hardly any fun, and arguably not much use
 112 [18:35] <smoser> In almost all cases, you will want to make some modifications to the instance immediately after boot.
 113 [18:36] <smoser> You can do this by waiting util the machine is running, and ssh'ing in and running a set of commands.
 114 [18:36] <smoser> But that may be less than idea.
 115 [18:36] <smoser> This is where "User Data" comes in.  User data is provided to the instance via the '--user-data' or '--user-data-file' arguments to ec2-run-instances.
 116 [18:37] <smoser> When the instance boots, it reads the user-data an can change its behavior based on that content.
 117 [18:37] <smoser> When an ubuntu instance starts, if user-data begins with '#!' then it will be executed like an rc.local script.
 118 [18:38] <smoser> This means that you can put together a shell script to turn the "base AMI" into your personal development box and pass that script as user-data to the instance on startup.  Then, when you ssh in, it will be all set up.
 119 [18:38] <smoser> This is obviously quite useful for all sorts of purposes
 120 [18:38] <smoser> We're adding features to how user-data is handled for lucid (https://wiki.ubuntu.com/ServerLucidCloudBoothooks and https://wiki.ubuntu.com/ServerLucidCloudConfig for more information).
 121 [18:38] <smoser> The big new feature will be the ability to configure the image on first boot via "config"  syntax rather than a shell script.
 122 [18:39] <smoser>  * Spot Instances
 123 [18:39] <smoser> If you're like me, you're cheap.  Spot instances allow you to run an instance in basically the same way as "normal instances" but at a reduced price.
 124 [18:39] <smoser> The catch is that if there is high demand, your instance may be terminated.
 125 [18:39] <smoser> I've began using spot instances for most of my tests.
 126 [18:40] <smoser> When you make a request for a "spot instance" you state the maximum you will pay for that instance.  The rate you are charged varies based on demand for those resources.
 127 [18:40] <smoser> if the rate goes above your maximum, your instance is terminated.
 128 [18:40] <smoser>  * Desktop In the Cloud
 129 [18:41] <smoser> In the lucid cycle, the Desktop team wanted to take advantage of the easily created test/reproduce/verify environment
 130 [18:41] <smoser> so we started creating "desktop in the cloud" builds.
 131 [18:41] <smoser> Those are created daily and registered to ec2
 132 [18:42] <smoser> http://uec-images.ubuntu.com/desktop/lucid/ for relevant AMIs.
 133 [18:42] <smoser> These images have a full gnome desktop installed and can be accessed via an NX client.  If you've never used NX before, it is surprising how well it performs.  Literally, you may forget that you are using a remote desktop.
 134 [18:43] <smoser> http://www.stgraber.org/2009/12/12/ubuntu-desktop-cloud-now-working-lucid has information on how to use these.
 135 [18:43] <smoser> QUESTION: smoser, when you use spot instances for your testing, what rate do you typically offer to pay, and how often do your instances get terminated?
 136 [18:43] <cjohnston> < kamalmostafa> QUESTION: smoser, when you use spot instances for your testing, what rate do you typically offer to pay, and how often do your instances get terminated?
 137 [18:44] <smoser> I've never actually had one terminated.
 138 [18:44] <smoser> http://cloudexchange.org/ has graphs of spot prices historically
 139 [18:44] <smoser> and you can get that same data with : ec2-describe-spot-price-history
 140 [18:45] <smoser> what I do is take the high price available and raise it by 10%
 141 [18:45] <smoser> you'll notice spikes in the graphs at cloudexchange
 142 [18:46] <smoser> this is where (I think) amazon had higher demand for non-spot instances and explicitly kicked off most spot instances
 143 [18:46] <smoser> On average, spot instances run ~ 40% of the regular price.
 144 [18:47] <smoser> This is all I had prepared, so the floor is open to questions
 145 [18:47] <smoser> Questions
 146 [18:48] <smoser> oh. One other thing I should mention
 147 [18:48] <smoser> i can't believe i didn't plug UEC
 148 [18:48] <smoser> UEC (Ubuntu Enterprise Cloud) is based on Eucalyptus and allows you to easily run a ec2 compatible cloud inside your own data center.
 149 [18:49] <smoser> This has all sorts of neat uses.
 150 [18:49] <cjohnston> < zul> QUESTION: how are the images built?
 151 [18:49] <smoser> zul, thanks for asking :)
 152 [18:49] <smoser> We build our images using vmbuilder
 153 [18:50] <smoser> The complete setup for building identical images is available starting at : https://code.launchpad.net/~ubuntu-on-ec2/vmbuilder/automated-ec2-builds
 154 [18:50] <smoser> and code we use to publish the builds is at https://code.edge.launchpad.net/~ubuntu-on-ec2/ubuntu-on-ec2/ec2-publishing-scripts
 155 [18:51] <cjohnston> < dhillon-v10> smoser: how is the upcomming HTML5 going to affect the server cloud
 156 [18:51] <smoser> hm.. I dont know.
 157 [18:51] <smoser> Do you think it will increase demand for server resources? or reduce demand?
 158 [18:52] <smoser> I guess I need more context on the question
 159 [18:52] <cjohnston> < dhillon-v10> smoser: probably decrease the demand because now users can do more stuff offline
 160 [18:53] <smoser> Maybe, but by the same note, the more you depend on "the cloud" the more important reliable resources are
 161 [18:54] <smoser> Well, I guess to wrap up...
 162 [18:54] <cjohnston> < dhillon-v10> smoser: possibly :)
 163 [18:54] <cjohnston> < kamalmostafa> QUESTION: Can EC2 machine images be used by UEC installations?  (either with our without incurring charges at Amazon)?
 164 [18:54] <smoser> kamalmostafa, yes
 165 [18:55] <smoser> The only real difference is that UEC uses KVM and EC2 is xen based.
 166 [18:55] <smoser> so the end result is that you'll have a different kernel, but ubuntu creates "ubuntu sauce" kernels for both varieties
 167 [18:56] <smoser> i see UEC as very useful for developing machine images that can then be deployed publically on ec2
 168 [18:56] <smoser> (and for other purposes too)
 169 [18:56] <cjohnston> < kamalmostafa> QUESTION: Does one need an Amazon AWS account at all, in order to use those images in a UEC installation?
 170 [18:56] <smoser> kamalmostafa, no
 171 [18:56] <smoser> UEC is completely stand alone.
 172 [18:56] <smoser> The requirements for it are basically just having hardware available.
 173 [18:56] <smoser> you can create a "cloud" with 2 systems
 174 [18:57] <smoser> and the ubuntu-server ISOs have an easy option for doing so.
 175 [18:57] <smoser> I've probably represented UEC poorly here... maybe next developer week we need a UEC class
 176 [18:58] <smoser> wrapping up...
 177 [18:59] <cjohnston> < lbrinkma> QUESTION: Whats the api command for creating a spot instance?
 178 [18:59] <smoser> If you're a ubuntu developer, I strongly urge you to take a look at using EC2 for easily running releases of ubuntu that you have no worry about destroying.
 179 [18:59] <smoser> ec2-request-spot-instances is the api command
 180 [18:59] <smoser> it basically looks like 'ec2-run-instances' with some extra args.
 181 [19:00] <smoser> I've done sone work to make a wrapper 'ec2-cheap-run-instances'.  It needs some work, but then is a dropin replacement for ec2-run-instances
 182 [19:00] <smoser> http://smoser.brickies.net/git/?p=misc-starter-tools.git;a=blob;f=ec2/ec2-cheap-run-instances;hb=HEAD
 183 [19:01] <smoser> If you're not a ubuntu developer but want to do "real stuff" with EC2, there are loads of people running businesses and servers on EC2's cloud.  I urge you to look into it further, and,of course, I urge you to start with the Official Ubuntu Images
 184 [19:02] <smoser> If you have questions or need assistance, please feel free to ping me  #ubuntu-server or elsewhere.
 185 [19:02] <cjohnston> Thanks smoser !