== Dev Week -- Deploy your App to the cloud, Writing Ensemble formulas 101 -- kim0 -- Thu, Jul 14th, 2011 == {{{#!irc [17:01] Good morning, good afternoon and good evening everyone [17:02] I'm here to introduce you to Ensemble → https://ensemble.ubuntu.com/ [17:02] Ensemble is a cloud orchestration framework [17:02] I am assuming you know what cloud is, and maybe played around a bit with Amazon ec2 [17:03] if you'd like me to provide a quick intro about that as well .. let me know [17:03] just ping me in #ubuntu-classroom-chat or leave a question [17:03] I will be hosting a demo of Ensemble (yay!) [17:04] You can login to this shared screen session → ssh guest@ec2-107-20-18-151.compute-1.amazonaws.com [17:04] password: guest [17:04] or you may view the web UI → https://ec2-107-20-18-151.compute-1.amazonaws.com/ [17:05] So .. as a developer, why would Ensemble interest you ? [17:05] - Ensemble enables you to quickly deploy your code to the cloud for testing/qa purposes [17:05] - You're able to easily test various different versions of operating systems [17:06] - And most importantly .. If deployment instructions for your application are not too simple, Ensemble provides an easy and consistent way [17:06] for your users to deploy your application [17:06] It's like having that big red "easy" button to hit, to deploy your code [17:07] As a developer, you're also ensured that users have deployed your application in exactly the way you would recommend [17:07] If there is any questions so far .. shoot [17:08] Let me jump over a couple of ensemble concepts quickly [17:08] then we'll do the demo [17:08] this session should be short and sweet [17:08] - Ensemble concepts [17:08] * The first thing to note, is that since Ensemble was written in the cloud age [17:09] it does not really care about "servers" .. [17:09] rather it focuses on "services" [17:09] services being .. [17:09] mysql : a database service [17:09] memcached : a high performance caching service [17:09] munin : a monitoring service [17:09] and so on and so forth [17:09] You application, of course interacts with many of those services [17:10] * The second important concept is that Ensemble completely "encapsulates" those services [17:10] that is, if you have no idea how to get munin running [17:10] if you ask ensemble to deploy it, you would have it running within minutes [17:11] and you can connect it (read: relate it) to other services [17:11] and it would start graphing performance metrics from all around your infrastructure [17:11] you do not need to know how to control munin, it is encapsulated [17:12] * The third important concept, is that with Ensemble services are "composable" [17:12] that is, services have well defined interfaces [17:12] such that you can connect/relate many services together .. to form a large infrastructure [17:12] you can replace infrastructure components with others .. such as replace mysql with pgsql if you so wish [17:13] and if both of them implement the same interface! [17:13] In the future, Ensemblse would even handle backing up and migrating your data! [17:13] Cool! [17:13] Enough talk .. let's jump to demo land [17:14] If there's any questions pre demo .. shoot [17:14] * kim0 waits a minute .. and prepares demo land [17:17] = demo begin = [17:17] Please make sure you're logged into the shared ssh session or web UI [17:17] for instructions, scroll up [17:17] Alright! let's get started [17:18] The very fist step, is as I just did in the shared screen [17:18] ensemble bootstrap [17:18] What this does, is it starts a "management" node for Ensemble [17:18] this is the brains for the deployment [17:18] it is a utility node [17:19] of course ec2 needs some time to create this node .. [17:19] so I need to check it is available and ready before proceeding [17:19] I can do this with [17:19] ensemble status [17:20] The node seems to be bootstrapping [17:21] While the node is bootstrapping [17:21] I would like to introduce "formulas" [17:22] Formulas, are Ensemble's way of codifying the experience of deploying code and orchestrating the full service [17:22] So, if you would like to deploy a mysql service [17:22] You would need a mysql formula [17:22] this is the same for any other service [17:23] and if you as a developer, would like your application to be deployable by Ensemble, you would need to write a formula for it [17:23] the "very cool" part! [17:23] is that formulas are language indepndant [17:23] that is, you do not need to learn yet another programming language [17:24] to write those formulas, formulas can be written in python, bash, php, c++ or whatever! [17:24] Very well .. the bootstrap has of course finish bootstrapping [17:24] I will paste ensemble commands and output here for archiving purposes [17:24] $ ensemble status [17:24] 2011-07-14 17:24:16,265 INFO Connecting to environment. [17:24] machines: [17:24] 0: {dns-name: ec2-50-17-153-3.compute-1.amazonaws.com, instance-id: i-f3c13e92} [17:25] services: {} [17:25] 2011-07-14 17:24:18,137 INFO 'status' command finished successfully [17:25] machine number "0" is our bootstrap node [17:25] let us now quickly deploy a wordpress service and a drupal application [17:25] How hard do you think that is? [17:26] Here it is [17:26] ensemble deploy --repository=. drupal mycms [17:26] This deploys an instance of "drupal" .. calls the instance "mycms" [17:27] $ ensemble deploy --repository=. mysql mydb [17:27] This also deploys mysql, calling it mydb! [17:27] doesn't really get much simpler than that! [17:27] let's check out ensemble status now [17:27] $ ensemble status [17:27] 2011-07-14 17:27:34,085 INFO Connecting to environment. [17:27] machines: [17:27] 0: {dns-name: ec2-50-17-153-3.compute-1.amazonaws.com, instance-id: i-f3c13e92} [17:27] 1: {dns-name: ec2-107-20-30-230.compute-1.amazonaws.com, instance-id: i-0537c864} [17:27] 2: {dns-name: ec2-50-19-190-104.compute-1.amazonaws.com, instance-id: i-e537c884} [17:27] services: [17:28] mycms: [17:28] formula: local:drupal-9 [17:28] relations: {} [17:28] units: [17:28] mycms/0: [17:28] machine: 1 [17:28] relations: {} [17:28] state: null [17:28] mydb: [17:28] formula: local:mysql-97 [17:28] relations: {} [17:28] units: [17:28] mydb/0: [17:28] machine: 2 [17:28] relations: {} [17:28] state: null [17:28] 2011-07-14 17:27:35,995 INFO 'status' command finished successfully [17:28] Let us quickly check out the meaning of this status message [17:28] In the machines section .. we now have 3 machines .. 0 1 and 2 [17:28] In the services section, we have two services deployed [17:28] mycms and mydb [17:29] You can see mycms (drupal) is running on machine 1 (from the machines section, it's url is ec2-107-20-30-230.compute-1.amazonaws.com) [17:29] and mydb (mysql) is running on machine 2 [17:30] Let me know paste an updated status [17:30] $ ensemble status [17:30] 2011-07-14 17:29:44,206 INFO Connecting to environment. [17:30] machines: [17:30] 0: {dns-name: ec2-50-17-153-3.compute-1.amazonaws.com, instance-id: i-f3c13e92} [17:30] 1: {dns-name: ec2-107-20-30-230.compute-1.amazonaws.com, instance-id: i-0537c864} [17:30] 2: {dns-name: ec2-50-19-190-104.compute-1.amazonaws.com, instance-id: i-e537c884} [17:30] services: [17:30] mycms: [17:30] formula: local:drupal-9 [17:30] relations: {} [17:30] units: [17:30] mycms/0: [17:30] machine: 1 [17:30] relations: {} [17:30] state: started [17:30] mydb: [17:30] formula: local:mysql-97 [17:30] relations: {} [17:30] units: [17:30] mydb/0: [17:30] machine: 2 [17:30] relations: {} [17:30] state: started [17:30] 2011-07-14 17:29:46,435 INFO 'status' command finished successfully [17:31] Now you can the state of both services is "started" [17:31] which means they have been successfully installed and started [17:31] Note however that till now, the two services are deployed "separately" [17:31] that is mysql knows nothing about drupal and vice versa [17:31] The big bang (a ha) moment happens when you connect both services together! [17:31] let's do just that [17:32] Here is how I just did it [17:32] $ ensemble add-relation mycms mydb:db [17:32] 2011-07-14 17:32:03,640 INFO Connecting to environment. [17:32] 2011-07-14 17:32:04,759 INFO Added mysql relation to all service units. [17:32] 2011-07-14 17:32:04,760 INFO 'add_relation' command finished successfully [17:32] I am using add-relation to connect the two services together [17:32] Once the connection happens [17:33] mysql recognizes drupal needs a database service [17:33] mysql creates a new database for drupal [17:33] sends over the connection information (ip, username, password) needed to connect to the newly created database! cool! [17:33] drupal receives this information, configures itself to connect to that service [17:34] the end result .. with that single command === zyga is now known as zyga-afk [17:34] and with Ensemble in the background [17:34] you have deployed a multi-tier service [17:35] As we just mentioned the url for the mycms machine is http://ec2-107-20-30-230.compute-1.amazonaws.com/ensemble/ [17:35] and you can indeed hit that url with your browser right now [17:35] and you should get drupal waiting for you .. hurray [17:36] I'd like to quickly show you as well .. the formula for drupal [17:36] i.e. how simple it is to write such a formula [17:36] I chose to write this formula in bash, however as mentioned, you can use any language you fancy! [17:38] The install hook (what runs to install the service) is now shown in the shared screen [17:38] I am however .. pasting it as well for archiving purposes [17:38] #!/bin/bash [17:38] set -eux # -x for verbose logging to ensemble debug-log [17:38] ensemble-log "Installing drush,apache2,php via apt-get" [17:38] apt-get -y install drush apache2 php5-gd libapache2-mod-php5 php5-cgi mysql-client-core-5.1 [17:38] a2enmod php5 [17:38] /etc/init.d/apache2 restart [17:38] ensemble-log "Using drush to download latest Drupal" [17:38] cd /var/www && drush dl drupal --drupal-project-rename=ensemble [17:39] As you can see, it simply uses apt-get to install some packages, enables php, restarts apache [17:39] and then uses "drush" the drupal shell, to download a fresh version of drupal [17:39] it's interesting to see that ensemble does NOT force you to only use ubuntu packages [17:39] you can mix and match like I just did [17:40] as a developer, you can grab any code branch and build it and mix that with package installations, if you so wish [17:40] Very well .. [17:40] remember the magic that happened when we connected the two services together [17:40] the "add-relation" command [17:40] let us see the code that runs from drupal's perspective [17:41] I have just displayed it on the shared screen [17:41] I will paste it in chunks [17:41] #!/bin/bash [17:41] set -eux # -x for verbose logging to ensemble debug-log [17:41] hooksdir=$PWD [17:41] user=`relation-get user` [17:41] password=`relation-get password` [17:41] host=`relation-get host` [17:41] database=`relation-get database` [17:42] these last 4 lines, read various settings from Ensemble provided communication channels [17:42] they read: user, password, host, database [17:42] these settings are provided by the mysql service upon establishing the connection. [17:42] drupal reads them, to configure itself to connect to that DB [17:43] let us see how the connection happens [17:43] ensemble-log "Setting up Drupal for the first time" [17:43] cd /var/www/ensemble && drush site-install -y standard \ [17:43] --db-url=mysql://$user:$password@$host/$database \ [17:43] --site-name=Ensemble --clean-url=0 [17:43] easy enough! [17:43] we just use drush to install drupal passing the need parameters we just consumed [17:43] What is really important to notice here [17:44] is that I might have never touched mysql before [17:44] I do not have to know how it works or how to configure it [17:44] or I cared about, was the Ensemble provided interface .. those 4 parameters in this case [17:45] The actual script is slightly longer, since it checks whether or not this is first time drupal is being set up [17:45] i.e. are we starting the very first drupal node, or just scaling it up [17:45] oh yeah .. because scaling it up is as simple as [17:45] ensemble add-unit mycms [17:45] yep .. that's it [17:46] So I hope this was a useful introduction to Ensemble [17:46] I'll provide a couple of pointers and answer question [17:46] questions* [17:46] You can read more about Ensemble at: https://ensemble.ubuntu.com/ [17:46] Documentation: https://ensemble.ubuntu.com/docs/ [17:46] Mailing list: https://lists.ubuntu.com/mailman/listinfo/ensemble [17:47] IRC Channel: #ubuntu-ensemble @ Freenode [17:47] we are a very welcoming community .. it's a party and everyone is invited :) [17:47] So start hacking on your own formula today! [17:47] You can see a list of existing formulas at: https://code.launchpad.net/principia [17:48] Our goal is to cover all of free software! [17:48] Alright .. taking questions [17:48] alexm asked: can emsemble be used to deploy outside the cloud? [17:48] alexm: Ensemble uses multiple "providers" to handle what to deploy to [17:49] ec2 cloud is currently the one that is ready, a provider for LXC (Linux container) is being worked on heavily [17:49] and myunderstanding is that it will be ready during this 11.10 cycle [17:49] you can also deploy to private clouds such as openstack or eucalyptus .. not sure how well tested that is, but should mostly work [17:50] Also, Ensemble is being currently integrated with Orchestra [17:50] the Ubuntu server hardware deployment tool [17:50] so you will be able to install to your own hardware farm as well [17:50] I hope that answers your question [17:51] There are 10 minutes remaining in the current session. [17:51] chadadavis asked: Does ensemble have upstart's concepts of services vs tasks. Backup seems more like a service, whereas migration more like a task. [17:51] Things are a bit too early in Ensemble development right now .. Currently everything is a service [17:52] however the dev team recognize the need that some operation have special needs [17:52] "machine services" are being debated [17:52] various types of services (or whatever they'd be called) can definitely arise [17:52] ahs3 asked: each service seems to be a separate machine; can i have multiple services on the same machine? [17:53] ahs3: good question .. With the LXC provider that will become possible [17:53] there is also design work currently to colocate chosen services together [17:53] interested code hackers are invited to hack on ensemble core to make this happen faster :) [17:53] #ubuntu-ensemble [17:54] Ensemble itself is written in python using twisted framework afaik [17:54] ahs3 asked: how do i find out what services are available? [17:54] ahs3: As mentioned this is the current list [17:54] https://code.launchpad.net/principia [17:54] this only started a few weeks back [17:54] so it is growing rapidly === janimo` is now known as janimo [17:55] I am more than ready to help anyone write his own first formula [17:55] just grab me (kim0) on #ubuntu-ensemble [17:55] and start hacking on your own formula :) [17:55] alexm asked: how does mycms know that it has to use mydb? i.e. mycms is deployed with another db before connecting to mydb? [17:55] alexm: good question [17:55] it doesn't [17:55] There are 5 minutes remaining in the current session. [17:55] I tell it to use mydb [17:55] when I issued [17:56] add-relation mycms mydb:db [17:56] This is exactly what "connects" or "relates" them together [17:56] before add-relation they have no idea about each other [17:56] and mycms does NOT have another DB .. in fact it has no db at all and is mostly misconfigured [17:57] Awesome .. that concludes this session [17:57] feel free to ask me any further questions in #ubuntu-ensemble [17:57] till next time .. c ya }}}