Ubuntu Wiki

   1 [17:00] <nigelb> Next up is kim0, the man behind the clouds ;)
   2 [17:01] <kim0> Hello Hello everyone o/
   3 [17:01] <ClassBot> Logs for this session will be available at http://irclogs.ubuntu.com/2011/03/04/%23ubuntu-classroom.html following the conclusion of the session.
   4 [17:01] <nigelb> Kim0 is going to talk about Introducing boto EC2 Cloud API.  kim0, the stage is all yours :)
   5 [17:01] <kim0> Thanks nigelb
   6 === ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Developer Week - Current Session: Introducing boto EC2 Cloud API - Instructors: kim0
   7 [17:02] <kim0> It's really a pleasure to be presenting in UDW
   8 [17:02] <kim0> So the topic of the session is "intro to boto ec2 api"
   9 [17:02] <kim0> basically what that means is
  10 [17:02] <kim0> how to control the Amazon EC2 cloud
  11 [17:02] <kim0> from the ease and comfort of your python shell
  12 [17:03] <kim0> Please feel free to shoot any questions to me in #ubuntu-classroom-chat
  13 [17:03] <kim0> just prepend your questions with QUESTION:
  14 [17:03] <kim0> I'll give a few lines intro
  15 [17:03] <kim0> since many might not be totally familiar with cloud means
  16 [17:04] <kim0> basically, cloud computing has tons of different definitions
  17 [17:04] <kim0> however, almost everyone will agree
  18 [17:04] <kim0> resources have to be allocated by an api call
  19 [17:04] <kim0> and that resource allocation is instantaneuous
  20 [17:04] <kim0> and that it should be elastic and almost infinite
  21 [17:05] <kim0> Amazon ec2 cloud meets those conditions
  22 [17:05] <kim0> so basically
  23 [17:05] <kim0> through an api call
  24 [17:05] <kim0> you're able to very quickly allocate computing resources
  25 [17:05] <kim0> i.e. servers, networking gear, IPs, storage space
  26 [17:05] <kim0> ...etc
  27 [17:05] <kim0> you use them for as much as you want
  28 [17:05] <kim0> then you simply "delete" them
  29 [17:06] <kim0> So in real life
  30 [17:06] <kim0> assuming you (as a developer) were tasked with the compute heavy task of say converting 100k text files into PDFs
  31 [17:07] <kim0> a "typical" implementation would be to spawn 20 servers in the cloud
  32 [17:07] <kim0> kick them crunching on the conversion .. and finish in say 2 hours
  33 [17:07] <kim0> then delete that infrastructure!
  34 [17:07] <kim0> and yes, you only pay for the used resources (40 hours of compute time)
  35 [17:07] <kim0> lovely, isn't it :)
  36 [17:08] <kim0> Any questions so far on the general concepts
  37 [17:08] <kim0> before I start digging into boto
  38 [17:09] <kim0> Ok .. assuming no one has questions
  39 [17:09] <kim0> let's get started
  40 [17:09] <kim0> the first step would be to install the python-boto package
  41 [17:10] <kim0> sudo apt-get install pyton-boto
  42 [17:10] <kim0> I also prefer to work in the ipython envrionment
  43 [17:10] <kim0> sudo apt-get install ipython
  44 [17:11] <kim0> short break .. taking questions
  45 [17:11] <ClassBot> techbreak asked: short introduction to urself please.. we wonder who our session master is :)
  46 [17:11] <kim0> hi techbreak :) I'm Ahmed Kamal, the ubuntu cloud community liaison
  47 [17:11] <kim0> My role is to help the ubuntu cloud community grow and succeed
  48 [17:12] <kim0> I always hang-out at #ubuntu-cloud
  49 [17:12] <kim0> feel free to ping me anytime
  50 [17:12] <ClassBot> techbreak asked: i have seen examples of cloud to be "facebook" too.. how facebook can be a cloud  ?
  51 [17:13] <kim0> well yes, FB is considered a cloud app
  52 [17:13] <kim0> It is a cloud "application"
  53 [17:13] <kim0> not a cloud platform
  54 [17:13] <kim0> the likes of Google apps (gmail, gdocs...) facebook ..
  55 [17:13] <kim0> are all considered cloud apps
  56 [17:13] <kim0> because the data and code are distributed in highly distributed system
  57 [17:14] <kim0> I could go into all sorts of details, but that question is a bit offtopic .. feel free to pick it up later :)
  58 [17:14] <ClassBot> techbreak asked: ipython ? iron ptyhon ?
  59 [17:14] <kim0> ipython is "interactive python shell" afaik
  60 [17:14] <kim0> ok .. assuming you're all set
  61 [17:15] <kim0> → back to boto
  62 === msnsachin12 is now known as msnsachin
  63 [17:15] <kim0> In order to actually execute the steps here .. you'd need to have an amazon ec2 account
  64 [17:15] <kim0> setup and have generated secret keys and stored your variables in ~/.ec2/ec2rc
  65 [17:16] <kim0> this is outside the scope of this tutorial however
  66 [17:16] <kim0> for those interested
  67 [17:16] <kim0> follow along with https://help.ubuntu.com/community/EC2StartersGuide
  68 [17:16] <kim0> If you spot any wrong info, let me know
  69 [17:16] <kim0> In all examples, I will copy/paste
  70 [17:16] <kim0> the api calls and the results
  71 [17:17] <kim0> so that everyone can follow along easily
  72 [17:17] <kim0> $ ipython
  73 [17:17] <kim0> we're now inside the ipython interperter
  74 [17:17] <kim0> import boto
  75 [17:17] <kim0> the boto python module is imported (ready to be used)
  76 [17:17] <kim0> The Amazon cloud is composed of multiple regions
  77 [17:17] <kim0> similar to "data centers" all around the world
  78 === ampelbein_ is now known as Ampelbein
  79 [17:18] <kim0> we need to pick one to connect to
  80 [17:18] <kim0> let's see how to do that
  81 [17:19] <kim0> from boto import ec2
  82 [17:19] <kim0> regions = ec2.regions()
  83 [17:19] <kim0> Out[7]:
  84 [17:19] <kim0> [RegionInfo:eu-west-1,
  85 [17:19] <kim0>  RegionInfo:us-east-1,
  86 [17:19] <kim0>  RegionInfo:ap-northeast-1,
  87 [17:19] <kim0>  RegionInfo:us-west-1,
  88 [17:19] <kim0>  RegionInfo:ap-southeast-1]
  89 [17:19] <kim0> What you see are the Amazon regions (data-centers) around the world
  90 [17:19] <kim0> we'll pick the one in us-east-1 to work with and connect to!
  91 [17:20] <kim0> >> useast = regions[1]
  92 [17:20] <kim0> I will prepend all python input code with (>>) to make it easily distinguishable
  93 [17:21] <kim0> >>useast.endpoint
  94 [17:21] <kim0>  u'ec2.us-east-1.amazonaws.com'
  95 [17:21] <kim0> Let's connect
  96 [17:21] <kim0> >> useconn = useast.connect()
  97 [17:21] <kim0> Awesome
  98 [17:22] <kim0> we're now connected
  99 [17:22] <kim0> let's do something useful
 100 [17:22] <kim0> by default .. Amazon configure its cloud firewall
 101 [17:22] <kim0> to block all incoming port connections
 102 [17:22] <kim0> the rule-sets .. are called "secutiy groups" in its jargon
 103 [17:23] <kim0> let's get a list of security groups available
 104 [17:23] <kim0> >> useconn.get_all_security_groups()
 105 [17:23] <kim0> Out[14]: [SecurityGroup:default]
 106 [17:23] <kim0> the result is a single group called "default" ... makes sense!
 107 [17:25] <kim0> just a little note, for anyone trying to setup a new account with amazon, you're gonna need a credit card
 108 [17:26] <kim0> while they offer a completely free instance (micro type) free for a year
 109 [17:26] <kim0> i.e. you most likely won't be charged anything .. but you still need a valid one
 110 [17:26] <kim0> try to follow along with me in the session, I'll be pasting all input on output
 111 [17:26] <kim0> back on track
 112 [17:26] <kim0> so let's get our security group (firewall rule)
 113 [17:26] <kim0> sg=useconn.get_all_security_groups()[0]
 114 [17:27] <kim0> let's "open" port 22
 115 [17:27] <kim0> that's for ssh
 116 [17:27] <kim0> >> sg.authorize('tcp', 22, 22, '0.0.0.0/0')
 117 [17:28] <kim0> Ok ..
 118 [17:28] <kim0> so let's quickly recap
 119 [17:29] <kim0> what we've done
 120 [17:29] <kim0> we've enumerated amazon cloud datacenters .. and chose to connect to the one in us-east
 121 [17:29] <kim0> and we're manipulated the firewall to open port 22
 122 [17:29] <kim0> all using API calls .. all on-demand and elastic
 123 [17:30] <kim0> let's start the really cool stuff :)
 124 [17:30] <kim0> let's start our own cloud server
 125 [17:30] <kim0> a little intro
 126 [17:30] <kim0> Ubuntu created official ubuntu cloud images, and publishes them to the amazon cloud
 127 [17:30] <kim0> each published image is called AMI
 128 [17:30] <kim0> Amazon Machine Image
 129 [17:31] <kim0> and each AMI .. has its own "id" .. which is like a long number identifying it
 130 [17:31] <kim0> when you want to start a new cloud server, you tell amazon what ami you want to use .. then it clones that ami and starts an "instance" of that ami for you!
 131 [17:31] <kim0> so let's do that
 132 [17:32] <kim0> >> ubuimages=useconn.get_all_images(owners= ['099720109477', ])
 133 [17:32] <kim0> note that useconn .. is the us-east connection we had setup
 134 [17:32] <kim0> get_all_images() is the call to get a list of images from amazon
 135 [17:32] <kim0> the "owners=['099720109477', ]" part .. is basically a filter .. that number is the ID for Canonical .. so that you only get official ubuntu images
 136 [17:33] <kim0> while you can start using fancy code to filter the huge list
 137 [17:33] <kim0> of images to what you want!
 138 [17:33] <kim0> You could use code like
 139 [17:33] <kim0> nattymachines = [ x for x in ubuimages if (x.type == 'machine' and re.search("atty", str(x.name))) ]
 140 [17:33] <kim0> I prefer a simpler approach
 141 [17:34] <kim0> Visit the Ubuntu cloud portal
 142 [17:34] <kim0> http://cloud.ubuntu.com/ami/
 143 [17:34] <kim0> this page shows a listing of all publicly available ubuntu images
 144 [17:34] <kim0> You just use the search box on the top right to search for what you want
 145 [17:35] <kim0> In my case, I searched for "us-east natty"
 146 [17:35] <kim0> the ami ID is shown in the table and you can simply copy it!
 147 [17:35] <kim0> For me it's    ami-7e4ab917
 148 [17:36] <kim0> so let's filter the list using that ID
 149 [17:36] <kim0> >> natty = [ x for x in ubuimages if x.id == 'ami-7e4ab917' ][0]
 150 [17:36] <kim0> >>  natty.name
 151 [17:36] <kim0> Out[22]: u'ebs/ubuntu-images-milestone/ubuntu-natty-alpha3-amd64-server-20110302.2'
 152 [17:36] <kim0> voila .. as you can see
 153 [17:36] <kim0> it's a natty image .. alpha3!
 154 [17:36] <kim0> hot from the oven :)
 155 [17:37] <kim0> Let's go ahead and start a server
 156 [17:37] <kim0> >> reservation = natty.run(key_name='default',instance_type='t1.micro')
 157 [17:37] <kim0> natty.run() starts an image
 158 [17:37] <kim0> the key_name .. is your ssh key .. this is part of setting up your Amazon ec2 account
 159 [17:38] <kim0> that key is injected into the ubuntu instance as it boots, so that you're able to ssh into it later
 160 [17:38] <kim0> The instance_type parameter .. is the "size" of the server you want
 161 [17:38] <kim0> in my case, I'm starting the smallest one .. micro
 162 [17:38] <kim0> since I'm executing this live
 163 [17:38] <kim0> the server must have been created right now
 164 [17:39] <kim0> in a matter of seconds
 165 [17:39] <kim0> the API call
 166 [17:39] <kim0> returns as "reservation"
 167 [17:39] <kim0> let's interrogate that
 168 [17:40] <kim0> >> instance = reservation.instances[0]
 169 [17:40] <kim0> Let's see if the server is ready
 170 [17:40] <kim0> >>  instance.state
 171 [17:40] <kim0> Out[26]: u'pending'
 172 [17:40] <kim0> oh .. that's interesting
 173 [17:40] <kim0> state pending means it's still being allocated
 174 [17:40] <kim0> any questions so far ?
 175 [17:41] <kim0> now is a good time while amazon allocates the instance :)
 176 [17:41] <kim0> not that it takes more than a few seconds actually
 177 [17:41] <kim0> Great
 178 [17:41] <kim0> it's ready
 179 [17:41] <kim0> >> : instance.update()
 180 [17:41] <kim0> Out[28]: u'running'
 181 [17:42] <kim0> The serer has been created, booted, my ssh key "default" injected into it
 182 [17:42] <kim0> how do we login you say
 183 [17:42] <kim0> let's ask about the serer's name
 184 [17:42] <kim0> >>  instance.public_dns_name
 185 [17:42] <kim0> Out[29]: u'ec2-184-72-132-193.compute-1.amazonaws.com'
 186 [17:43] <kim0> I can now ssh into that Ubuntu cloud server just like that
 187 [17:43] <kim0> ssh ubuntu@ec2-184-72-132-193.compute-1.amazonaws.com
 188 [17:44] <kim0> If I had used a different ssh key, I would nice ssh "-i" parameter .. but I'm using my default "~/.ssh/id_rsa" .. so no need to do anything
 189 [17:46] <kim0> I've just configured the server to allow
 190 [17:46] <kim0> you guys to ssh into it
 191 [17:46] <kim0> Go ahead
 192 [17:46] <kim0> ssh session@ec2-184-72-132-193.compute-1.amazonaws.com
 193 [17:47] <kim0> password: session
 194 [17:47] <kim0> please don't do anything nasty :)
 195 [17:47] <kim0> Once logged in .. feel free to start byobu
 196 [17:47] <kim0> a nice colorful gnu/screen customization
 197 [17:47] <kim0> I'm ready to take some questions
 198 [17:49] <ClassBot> ranamalo asked: when new packages are released are they incorporated in the ubuntu ami's?  Like if openssh is currently 5.3 and 5.4 is released today, if fire up the ubuntu ami tomorrow will i have 5.4 installed?
 199 [17:49] <kim0> ranamalo: Well, with package being updated
 200 [17:49] <kim0> new AMIs are pushed
 201 [17:50] <kim0> remember that AMI-ID we got   ami-7e4ab917
 202 [17:50] <kim0> that we got from cloud.ubuntu.com/ami
 203 [17:50] <kim0> every couple of weeks or so
 204 [17:50] <kim0> updates will be pushed
 205 [17:50] <kim0> and a new image will be created
 206 [17:50] <kim0> however
 207 [17:50] <kim0> if you're running an older image .. there's nothing preventing you from apt-get dist-upgrade 'ing it
 208 [17:51] <ClassBot> There are 10 minutes remaining in the current session.
 209 [17:51] <kim0> this is espeically true if you're running natty or a recent maverick (pvgrub booted instance) .. not too important for now
 210 [17:52] <ClassBot> techbreak asked: if you are copy pasting can you link us to the python code ? so that we can have the code and you can explain one by one /
 211 [17:52] <kim0> Here you are
 212 [17:52] <kim0> http://paste.ubuntu.com/575608/
 213 [17:52] <kim0> That's all the commands I've written so far
 214 [17:52] <kim0> you can practice on your own later
 215 [17:53] <kim0> and if you need any help .. everyone at #ubuntu-cloud is more than helpful (at least I hope so) :)
 216 [17:54] <ClassBot> mhall119 asked: does boto work on UEC as well?
 217 [17:54] <kim0> mhall119: yes it does AFAIK
 218 [17:54] <kim0> UEC is Ubuntu Enterprise Cloud
 219 [17:54] <kim0> it's a private cloud product
 220 [17:54] <kim0> that you can use to run your own cloud (like Aamzon)
 221 [17:54] <kim0> on your own hardware
 222 [17:54] <kim0> It is based on the eucalyptus open source project
 223 [17:54] <kim0> for more info on using boto with UEC .. check out this link http://open.eucalyptus.com/wiki/ToolsEcosystem_boto
 224 [17:55] <ClassBot> ranamalo asked: what advantages are there to using boto over ec2-ami-tools and ec2-api-tools?
 225 [17:56] <ClassBot> There are 5 minutes remaining in the current session.
 226 [17:56] <kim0> ranamalo: well, you can use command line tools in those packages
 227 [17:56] <kim0> to effectively do the same thing
 228 [17:56] <kim0> however the benefit of using python bindings
 229 [17:56] <kim0> is clear .. if you're writing a tool
 230 [17:56] <kim0> or some program
 231 [17:56] <kim0> running external commands
 232 [17:56] <kim0> and parsing results from the stdout text
 233 [17:56] <kim0> is effectively hell
 234 [17:56] <kim0> for programmers
 235 [17:57] <kim0> an API provides a consistent clean interface
 236 [17:57] <ClassBot> akshatj asked: What would be the advantages of deploying dmedia( https://launchpad.net/dmedia ) on clous?
 237 [17:57]  * kim0 checking that out
 238 [17:58] <kim0> akshatj: I'm not familiar with that project
 239 [17:58] <kim0> however the generic advantages would be
 240 [17:58] <kim0> you don't worry about hardware
 241 [17:58] <kim0> you don't manage the infrastrucutre
 242 [17:58] <kim0> you can scale-up/down easily cheaply
 243 [17:58] <ClassBot> ranamalo asked: Is there a howto url you can give us?
 244 [17:59]  * kim0 racing to answer :)
 245 [17:59] <kim0> howto url for ?
 246 [17:59] <kim0> a boto howto
 247 [17:59] <kim0> well .. nothing special .. just googling you'll find tons of stuff
 248 [18:00] <kim0> many useful info on the wiki
 249 [18:00] <kim0> like https://help.ubuntu.com/community/UEC
 250 [18:00] <kim0> If you're interested in learning more or contributing
 251 [18:00] <kim0> ping me anytime
 252 [18:00] <kim0> on #ubuntu-cloud
 253 [18:00] <kim0> Thanks everyone
 254 [18:00] <kim0> hope this was useful
 255 [18:00] <kim0> don't forget to terminate your instances
 256 [18:01] <kim0> or you keep paying for them :)
 257 [18:01] <kim0> bye bye