== Dev Week -- Introducing boto EC2 Cloud API -- kim0 -- Fri, Mar 4th, 2011 == {{{#!irc [17:00] Next up is kim0, the man behind the clouds ;) [17:01] Hello Hello everyone o/ [17:01] Logs for this session will be available at http://irclogs.ubuntu.com/2011/03/04/%23ubuntu-classroom.html following the conclusion of the session. [17:01] Kim0 is going to talk about Introducing boto EC2 Cloud API. kim0, the stage is all yours :) [17:01] Thanks nigelb === ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Developer Week - Current Session: Introducing boto EC2 Cloud API - Instructors: kim0 [17:02] It's really a pleasure to be presenting in UDW [17:02] So the topic of the session is "intro to boto ec2 api" [17:02] basically what that means is [17:02] how to control the Amazon EC2 cloud [17:02] from the ease and comfort of your python shell [17:03] Please feel free to shoot any questions to me in #ubuntu-classroom-chat [17:03] just prepend your questions with QUESTION: [17:03] I'll give a few lines intro [17:03] since many might not be totally familiar with cloud means [17:04] basically, cloud computing has tons of different definitions [17:04] however, almost everyone will agree [17:04] resources have to be allocated by an api call [17:04] and that resource allocation is instantaneuous [17:04] and that it should be elastic and almost infinite [17:05] Amazon ec2 cloud meets those conditions [17:05] so basically [17:05] through an api call [17:05] you're able to very quickly allocate computing resources [17:05] i.e. servers, networking gear, IPs, storage space [17:05] ...etc [17:05] you use them for as much as you want [17:05] then you simply "delete" them [17:06] So in real life [17:06] assuming you (as a developer) were tasked with the compute heavy task of say converting 100k text files into PDFs [17:07] a "typical" implementation would be to spawn 20 servers in the cloud [17:07] kick them crunching on the conversion .. and finish in say 2 hours [17:07] then delete that infrastructure! [17:07] and yes, you only pay for the used resources (40 hours of compute time) [17:07] lovely, isn't it :) [17:08] Any questions so far on the general concepts [17:08] before I start digging into boto [17:09] Ok .. assuming no one has questions [17:09] let's get started [17:09] the first step would be to install the python-boto package [17:10] sudo apt-get install pyton-boto [17:10] I also prefer to work in the ipython envrionment [17:10] sudo apt-get install ipython [17:11] short break .. taking questions [17:11] techbreak asked: short introduction to urself please.. we wonder who our session master is :) [17:11] hi techbreak :) I'm Ahmed Kamal, the ubuntu cloud community liaison [17:11] My role is to help the ubuntu cloud community grow and succeed [17:12] I always hang-out at #ubuntu-cloud [17:12] feel free to ping me anytime [17:12] techbreak asked: i have seen examples of cloud to be "facebook" too.. how facebook can be a cloud ? [17:13] well yes, FB is considered a cloud app [17:13] It is a cloud "application" [17:13] not a cloud platform [17:13] the likes of Google apps (gmail, gdocs...) facebook .. [17:13] are all considered cloud apps [17:13] because the data and code are distributed in highly distributed system [17:14] I could go into all sorts of details, but that question is a bit offtopic .. feel free to pick it up later :) [17:14] techbreak asked: ipython ? iron ptyhon ? [17:14] ipython is "interactive python shell" afaik [17:14] ok .. assuming you're all set [17:15] → back to boto === msnsachin12 is now known as msnsachin [17:15] In order to actually execute the steps here .. you'd need to have an amazon ec2 account [17:15] setup and have generated secret keys and stored your variables in ~/.ec2/ec2rc [17:16] this is outside the scope of this tutorial however [17:16] for those interested [17:16] follow along with https://help.ubuntu.com/community/EC2StartersGuide [17:16] If you spot any wrong info, let me know [17:16] In all examples, I will copy/paste [17:16] the api calls and the results [17:17] so that everyone can follow along easily [17:17] $ ipython [17:17] we're now inside the ipython interperter [17:17] import boto [17:17] the boto python module is imported (ready to be used) [17:17] The Amazon cloud is composed of multiple regions [17:17] similar to "data centers" all around the world === ampelbein_ is now known as Ampelbein [17:18] we need to pick one to connect to [17:18] let's see how to do that [17:19] from boto import ec2 [17:19] regions = ec2.regions() [17:19] Out[7]: [17:19] [RegionInfo:eu-west-1, [17:19] RegionInfo:us-east-1, [17:19] RegionInfo:ap-northeast-1, [17:19] RegionInfo:us-west-1, [17:19] RegionInfo:ap-southeast-1] [17:19] What you see are the Amazon regions (data-centers) around the world [17:19] we'll pick the one in us-east-1 to work with and connect to! [17:20] >> useast = regions[1] [17:20] I will prepend all python input code with (>>) to make it easily distinguishable [17:21] >>useast.endpoint [17:21] u'ec2.us-east-1.amazonaws.com' [17:21] Let's connect [17:21] >> useconn = useast.connect() [17:21] Awesome [17:22] we're now connected [17:22] let's do something useful [17:22] by default .. Amazon configure its cloud firewall [17:22] to block all incoming port connections [17:22] the rule-sets .. are called "secutiy groups" in its jargon [17:23] let's get a list of security groups available [17:23] >> useconn.get_all_security_groups() [17:23] Out[14]: [SecurityGroup:default] [17:23] the result is a single group called "default" ... makes sense! [17:25] just a little note, for anyone trying to setup a new account with amazon, you're gonna need a credit card [17:26] while they offer a completely free instance (micro type) free for a year [17:26] i.e. you most likely won't be charged anything .. but you still need a valid one [17:26] try to follow along with me in the session, I'll be pasting all input on output [17:26] back on track [17:26] so let's get our security group (firewall rule) [17:26] sg=useconn.get_all_security_groups()[0] [17:27] let's "open" port 22 [17:27] that's for ssh [17:27] >> sg.authorize('tcp', 22, 22, '0.0.0.0/0') [17:28] Ok .. [17:28] so let's quickly recap [17:29] what we've done [17:29] we've enumerated amazon cloud datacenters .. and chose to connect to the one in us-east [17:29] and we're manipulated the firewall to open port 22 [17:29] all using API calls .. all on-demand and elastic [17:30] let's start the really cool stuff :) [17:30] let's start our own cloud server [17:30] a little intro [17:30] Ubuntu created official ubuntu cloud images, and publishes them to the amazon cloud [17:30] each published image is called AMI [17:30] Amazon Machine Image [17:31] and each AMI .. has its own "id" .. which is like a long number identifying it [17:31] when you want to start a new cloud server, you tell amazon what ami you want to use .. then it clones that ami and starts an "instance" of that ami for you! [17:31] so let's do that [17:32] >> ubuimages=useconn.get_all_images(owners= ['099720109477', ]) [17:32] note that useconn .. is the us-east connection we had setup [17:32] get_all_images() is the call to get a list of images from amazon [17:32] the "owners=['099720109477', ]" part .. is basically a filter .. that number is the ID for Canonical .. so that you only get official ubuntu images [17:33] while you can start using fancy code to filter the huge list [17:33] of images to what you want! [17:33] You could use code like [17:33] nattymachines = [ x for x in ubuimages if (x.type == 'machine' and re.search("atty", str(x.name))) ] [17:33] I prefer a simpler approach [17:34] Visit the Ubuntu cloud portal [17:34] http://cloud.ubuntu.com/ami/ [17:34] this page shows a listing of all publicly available ubuntu images [17:34] You just use the search box on the top right to search for what you want [17:35] In my case, I searched for "us-east natty" [17:35] the ami ID is shown in the table and you can simply copy it! [17:35] For me it's ami-7e4ab917 [17:36] so let's filter the list using that ID [17:36] >> natty = [ x for x in ubuimages if x.id == 'ami-7e4ab917' ][0] [17:36] >> natty.name [17:36] Out[22]: u'ebs/ubuntu-images-milestone/ubuntu-natty-alpha3-amd64-server-20110302.2' [17:36] voila .. as you can see [17:36] it's a natty image .. alpha3! [17:36] hot from the oven :) [17:37] Let's go ahead and start a server [17:37] >> reservation = natty.run(key_name='default',instance_type='t1.micro') [17:37] natty.run() starts an image [17:37] the key_name .. is your ssh key .. this is part of setting up your Amazon ec2 account [17:38] that key is injected into the ubuntu instance as it boots, so that you're able to ssh into it later [17:38] The instance_type parameter .. is the "size" of the server you want [17:38] in my case, I'm starting the smallest one .. micro [17:38] since I'm executing this live [17:38] the server must have been created right now [17:39] in a matter of seconds [17:39] the API call [17:39] returns as "reservation" [17:39] let's interrogate that [17:40] >> instance = reservation.instances[0] [17:40] Let's see if the server is ready [17:40] >> instance.state [17:40] Out[26]: u'pending' [17:40] oh .. that's interesting [17:40] state pending means it's still being allocated [17:40] any questions so far ? [17:41] now is a good time while amazon allocates the instance :) [17:41] not that it takes more than a few seconds actually [17:41] Great [17:41] it's ready [17:41] >> : instance.update() [17:41] Out[28]: u'running' [17:42] The serer has been created, booted, my ssh key "default" injected into it [17:42] how do we login you say [17:42] let's ask about the serer's name [17:42] >> instance.public_dns_name [17:42] Out[29]: u'ec2-184-72-132-193.compute-1.amazonaws.com' [17:43] I can now ssh into that Ubuntu cloud server just like that [17:43] ssh ubuntu@ec2-184-72-132-193.compute-1.amazonaws.com [17:44] If I had used a different ssh key, I would nice ssh "-i" parameter .. but I'm using my default "~/.ssh/id_rsa" .. so no need to do anything [17:46] I've just configured the server to allow [17:46] you guys to ssh into it [17:46] Go ahead [17:46] ssh session@ec2-184-72-132-193.compute-1.amazonaws.com [17:47] password: session [17:47] please don't do anything nasty :) [17:47] Once logged in .. feel free to start byobu [17:47] a nice colorful gnu/screen customization [17:47] I'm ready to take some questions [17:49] ranamalo asked: when new packages are released are they incorporated in the ubuntu ami's? Like if openssh is currently 5.3 and 5.4 is released today, if fire up the ubuntu ami tomorrow will i have 5.4 installed? [17:49] ranamalo: Well, with package being updated [17:49] new AMIs are pushed [17:50] remember that AMI-ID we got ami-7e4ab917 [17:50] that we got from cloud.ubuntu.com/ami [17:50] every couple of weeks or so [17:50] updates will be pushed [17:50] and a new image will be created [17:50] however [17:50] if you're running an older image .. there's nothing preventing you from apt-get dist-upgrade 'ing it [17:51] There are 10 minutes remaining in the current session. [17:51] this is espeically true if you're running natty or a recent maverick (pvgrub booted instance) .. not too important for now [17:52] techbreak asked: if you are copy pasting can you link us to the python code ? so that we can have the code and you can explain one by one / [17:52] Here you are [17:52] http://paste.ubuntu.com/575608/ [17:52] That's all the commands I've written so far [17:52] you can practice on your own later [17:53] and if you need any help .. everyone at #ubuntu-cloud is more than helpful (at least I hope so) :) [17:54] mhall119 asked: does boto work on UEC as well? [17:54] mhall119: yes it does AFAIK [17:54] UEC is Ubuntu Enterprise Cloud [17:54] it's a private cloud product [17:54] that you can use to run your own cloud (like Aamzon) [17:54] on your own hardware [17:54] It is based on the eucalyptus open source project [17:54] for more info on using boto with UEC .. check out this link http://open.eucalyptus.com/wiki/ToolsEcosystem_boto [17:55] ranamalo asked: what advantages are there to using boto over ec2-ami-tools and ec2-api-tools? [17:56] There are 5 minutes remaining in the current session. [17:56] ranamalo: well, you can use command line tools in those packages [17:56] to effectively do the same thing [17:56] however the benefit of using python bindings [17:56] is clear .. if you're writing a tool [17:56] or some program [17:56] running external commands [17:56] and parsing results from the stdout text [17:56] is effectively hell [17:56] for programmers [17:57] an API provides a consistent clean interface [17:57] akshatj asked: What would be the advantages of deploying dmedia( https://launchpad.net/dmedia ) on clous? [17:57] * kim0 checking that out [17:58] akshatj: I'm not familiar with that project [17:58] however the generic advantages would be [17:58] you don't worry about hardware [17:58] you don't manage the infrastrucutre [17:58] you can scale-up/down easily cheaply [17:58] ranamalo asked: Is there a howto url you can give us? [17:59] * kim0 racing to answer :) [17:59] howto url for ? [17:59] a boto howto [17:59] well .. nothing special .. just googling you'll find tons of stuff [18:00] many useful info on the wiki [18:00] like https://help.ubuntu.com/community/UEC [18:00] If you're interested in learning more or contributing [18:00] ping me anytime [18:00] on #ubuntu-cloud [18:00] Thanks everyone [18:00] hope this was useful [18:00] don't forget to terminate your instances [18:01] or you keep paying for them :) [18:01] bye bye }}}