Ubuntu Wiki

   1 [17:01] <kim0> Good morning, good afternoon and good evening everyone
   2 [17:02] <kim0> I'm here to introduce you to Ensemble → https://ensemble.ubuntu.com/
   3 [17:02] <kim0> Ensemble is a cloud orchestration framework
   4 [17:02] <kim0> I am assuming you know what cloud is, and maybe played around a bit with Amazon ec2
   5 [17:03] <kim0> if you'd like me to provide a quick intro about that as well .. let me know
   6 [17:03] <kim0> just ping me in #ubuntu-classroom-chat or leave a question
   7 [17:03] <kim0> I will be hosting a demo of Ensemble (yay!)
   8 [17:04] <kim0> You can login to this shared screen session → ssh guest@ec2-107-20-18-151.compute-1.amazonaws.com
   9 [17:04] <kim0> password: guest
  10 [17:04] <kim0> or you may view the web UI → https://ec2-107-20-18-151.compute-1.amazonaws.com/
  11 [17:05] <kim0> So .. as a developer, why would Ensemble interest you ?
  12 [17:05] <kim0> - Ensemble enables you to quickly deploy your code to the cloud for testing/qa purposes
  13 [17:05] <kim0> - You're able to easily test various different versions of operating systems
  14 [17:06] <kim0> - And most importantly .. If deployment instructions for your application are not too simple, Ensemble provides an easy and consistent way
  15 [17:06] <kim0> for your users to deploy your application
  16 [17:06] <kim0> It's like having that big red "easy" button to hit, to deploy your code
  17 [17:07] <kim0> As a developer, you're also ensured that users have deployed your application in exactly the way you would recommend
  18 [17:07] <kim0> If there is any questions so far .. shoot
  19 [17:08] <kim0> Let me jump over a couple of ensemble concepts quickly
  20 [17:08] <kim0> then we'll do the demo
  21 [17:08] <kim0> this session should be short and sweet
  22 [17:08] <kim0> - Ensemble concepts
  23 [17:08] <kim0> * The first thing to note, is that since Ensemble was written in the cloud age
  24 [17:09] <kim0> it does not really care about "servers" ..
  25 [17:09] <kim0> rather it focuses on "services"
  26 [17:09] <kim0> services being ..
  27 [17:09] <kim0> mysql : a database service
  28 [17:09] <kim0> memcached : a high performance caching service
  29 [17:09] <kim0> munin : a monitoring service
  30 [17:09] <kim0> and so on and so forth
  31 [17:09] <kim0> You application, of course interacts with many of those services
  32 [17:10] <kim0> * The second important concept is that Ensemble completely "encapsulates" those services
  33 [17:10] <kim0> that is, if you have no idea how to get munin running
  34 [17:10] <kim0> if you ask ensemble to deploy it, you would have it running within minutes
  35 [17:11] <kim0> and you can connect it (read: relate it) to other services
  36 [17:11] <kim0> and it would start graphing performance metrics from all around your infrastructure
  37 [17:11] <kim0> you do not need to know how to control munin, it is encapsulated
  38 [17:12] <kim0> * The third important concept, is that with Ensemble services are "composable"
  39 [17:12] <kim0> that is, services have well defined interfaces
  40 [17:12] <kim0> such that you can connect/relate many services together .. to form a large infrastructure
  41 [17:12] <kim0> you can replace infrastructure components with others .. such as replace mysql with pgsql if you so wish
  42 [17:13] <kim0> and if both of them implement the same interface!
  43 [17:13] <kim0> In the future, Ensemblse would even handle backing up and migrating your data!
  44 [17:13] <kim0> Cool!
  45 [17:13] <kim0> Enough talk .. let's jump to demo land
  46 [17:14] <kim0> If there's any questions pre demo .. shoot
  47 [17:14]  * kim0 waits a minute .. and prepares demo land
  48 [17:17] <kim0> = demo begin =
  49 [17:17] <kim0> Please make sure you're logged into the shared ssh session or web UI
  50 [17:17] <kim0> for instructions, scroll up
  51 [17:17] <kim0> Alright! let's get started
  52 [17:18] <kim0> The very fist step, is as I just did in the shared screen
  53 [17:18] <kim0> ensemble bootstrap
  54 [17:18] <kim0> What this does, is it starts a "management" node for Ensemble
  55 [17:18] <kim0> this is the brains for the deployment
  56 [17:18] <kim0> it is a utility node
  57 [17:19] <kim0> of course ec2 needs some time to create this node ..
  58 [17:19] <kim0> so I need to check it is available and ready before proceeding
  59 [17:19] <kim0> I can do this with
  60 [17:19] <kim0> ensemble status
  61 [17:20] <kim0> The node seems to be bootstrapping
  62 [17:21] <kim0> While the node is bootstrapping
  63 [17:21] <kim0> I would like to introduce "formulas"
  64 [17:22] <kim0> Formulas, are Ensemble's way of codifying the experience of deploying code and orchestrating the full service
  65 [17:22] <kim0> So, if you would like to deploy a mysql service
  66 [17:22] <kim0> You would need a mysql formula
  67 [17:22] <kim0> this is the same for any other service
  68 [17:23] <kim0> and if you as a developer, would like your application to be deployable by Ensemble, you would need to write a formula for it
  69 [17:23] <kim0> the "very cool" part!
  70 [17:23] <kim0> is that formulas are language indepndant
  71 [17:23] <kim0> that is, you do not need to learn yet another programming language
  72 [17:24] <kim0> to write those formulas, formulas can be written in python, bash, php, c++ or whatever!
  73 [17:24] <kim0> Very well .. the bootstrap has of course finish bootstrapping
  74 [17:24] <kim0> I will paste ensemble commands and output here for archiving purposes
  75 [17:24] <kim0> $ ensemble status
  76 [17:24] <kim0> 2011-07-14 17:24:16,265 INFO Connecting to environment.
  77 [17:24] <kim0> machines:
  78 [17:24] <kim0>   0: {dns-name: ec2-50-17-153-3.compute-1.amazonaws.com, instance-id: i-f3c13e92}
  79 [17:25] <kim0> services: {}
  80 [17:25] <kim0> 2011-07-14 17:24:18,137 INFO 'status' command finished successfully
  81 [17:25] <kim0> machine number "0" is our bootstrap node
  82 [17:25] <kim0> let us now quickly deploy a wordpress service and a drupal application
  83 [17:25] <kim0> How hard do you think that is?
  84 [17:26] <kim0> Here it is
  85 [17:26] <kim0> ensemble deploy --repository=. drupal mycms
  86 [17:26] <kim0> This deploys an instance of "drupal" .. calls the instance "mycms"
  87 [17:27] <kim0> $ ensemble deploy --repository=. mysql mydb
  88 [17:27] <kim0> This also deploys mysql, calling it mydb!
  89 [17:27] <kim0> doesn't really get much simpler than that!
  90 [17:27] <kim0> let's check out ensemble status now
  91 [17:27] <kim0> $ ensemble status
  92 [17:27] <kim0> 2011-07-14 17:27:34,085 INFO Connecting to environment.
  93 [17:27] <kim0> machines:
  94 [17:27] <kim0>   0: {dns-name: ec2-50-17-153-3.compute-1.amazonaws.com, instance-id: i-f3c13e92}
  95 [17:27] <kim0>   1: {dns-name: ec2-107-20-30-230.compute-1.amazonaws.com, instance-id: i-0537c864}
  96 [17:27] <kim0>   2: {dns-name: ec2-50-19-190-104.compute-1.amazonaws.com, instance-id: i-e537c884}
  97 [17:27] <kim0> services:
  98 [17:28] <kim0>   mycms:
  99 [17:28] <kim0>     formula: local:drupal-9
 100 [17:28] <kim0>     relations: {}
 101 [17:28] <kim0>     units:
 102 [17:28] <kim0>       mycms/0:
 103 [17:28] <kim0>         machine: 1
 104 [17:28] <kim0>         relations: {}
 105 [17:28] <kim0>         state: null
 106 [17:28] <kim0>   mydb:
 107 [17:28] <kim0>     formula: local:mysql-97
 108 [17:28] <kim0>     relations: {}
 109 [17:28] <kim0>     units:
 110 [17:28] <kim0>       mydb/0:
 111 [17:28] <kim0>         machine: 2
 112 [17:28] <kim0>         relations: {}
 113 [17:28] <kim0>         state: null
 114 [17:28] <kim0> 2011-07-14 17:27:35,995 INFO 'status' command finished successfully
 115 [17:28] <kim0> Let us quickly check out the meaning of this status message
 116 [17:28] <kim0> In the machines section .. we now have 3 machines .. 0 1 and 2
 117 [17:28] <kim0> In the services section, we have two services deployed
 118 [17:28] <kim0> mycms and mydb
 119 [17:29] <kim0> You can see mycms (drupal) is running on machine 1 (from the machines section, it's url is ec2-107-20-30-230.compute-1.amazonaws.com)
 120 [17:29] <kim0> and mydb (mysql) is running on machine 2
 121 [17:30] <kim0> Let me know paste an updated status
 122 [17:30] <kim0> $ ensemble status
 123 [17:30] <kim0> 2011-07-14 17:29:44,206 INFO Connecting to environment.
 124 [17:30] <kim0> machines:
 125 [17:30] <kim0>   0: {dns-name: ec2-50-17-153-3.compute-1.amazonaws.com, instance-id: i-f3c13e92}
 126 [17:30] <kim0>   1: {dns-name: ec2-107-20-30-230.compute-1.amazonaws.com, instance-id: i-0537c864}
 127 [17:30] <kim0>   2: {dns-name: ec2-50-19-190-104.compute-1.amazonaws.com, instance-id: i-e537c884}
 128 [17:30] <kim0> services:
 129 [17:30] <kim0>   mycms:
 130 [17:30] <kim0>     formula: local:drupal-9
 131 [17:30] <kim0>     relations: {}
 132 [17:30] <kim0>     units:
 133 [17:30] <kim0>       mycms/0:
 134 [17:30] <kim0>         machine: 1
 135 [17:30] <kim0>         relations: {}
 136 [17:30] <kim0>         state: started
 137 [17:30] <kim0>   mydb:
 138 [17:30] <kim0>     formula: local:mysql-97
 139 [17:30] <kim0>     relations: {}
 140 [17:30] <kim0>     units:
 141 [17:30] <kim0>       mydb/0:
 142 [17:30] <kim0>         machine: 2
 143 [17:30] <kim0>         relations: {}
 144 [17:30] <kim0>         state: started
 145 [17:30] <kim0> 2011-07-14 17:29:46,435 INFO 'status' command finished successfully
 146 [17:31] <kim0> Now you can the state of both services is "started"
 147 [17:31] <kim0> which means they have been successfully installed and started
 148 [17:31] <kim0> Note however that till now, the two services are deployed "separately"
 149 [17:31] <kim0> that is mysql knows nothing about drupal and vice versa
 150 [17:31] <kim0> The big bang (a ha) moment happens when you connect both services together!
 151 [17:31] <kim0> let's do just that
 152 [17:32] <kim0> Here is how I just did it
 153 [17:32] <kim0> $ ensemble add-relation mycms mydb:db
 154 [17:32] <kim0> 2011-07-14 17:32:03,640 INFO Connecting to environment.
 155 [17:32] <kim0> 2011-07-14 17:32:04,759 INFO Added mysql relation to all service units.
 156 [17:32] <kim0> 2011-07-14 17:32:04,760 INFO 'add_relation' command finished successfully
 157 [17:32] <kim0> I am using add-relation to connect the two services together
 158 [17:32] <kim0> Once the connection happens
 159 [17:33] <kim0> mysql recognizes drupal needs a database service
 160 [17:33] <kim0> mysql creates a new database for drupal
 161 [17:33] <kim0> sends over the connection information (ip, username, password) needed to connect to the newly created database! cool!
 162 [17:33] <kim0> drupal receives this information, configures itself to connect to that service
 163 [17:34] <kim0> the end result .. with that single command
 164 === zyga is now known as zyga-afk
 165 [17:34] <kim0> and with Ensemble in the background
 166 [17:34] <kim0> you have deployed a multi-tier service
 167 [17:35] <kim0> As we just mentioned the url for the mycms machine is http://ec2-107-20-30-230.compute-1.amazonaws.com/ensemble/
 168 [17:35] <kim0> and you can indeed hit that url with your browser right now
 169 [17:35] <kim0> and you should get drupal waiting for you .. hurray
 170 [17:36] <kim0> I'd like to quickly show you as well .. the formula for drupal
 171 [17:36] <kim0> i.e. how simple it is to write such a formula
 172 [17:36] <kim0> I chose to write this formula in bash, however as mentioned, you can use any language you fancy!
 173 [17:38] <kim0> The install hook (what runs to install the service) is now shown in the shared screen
 174 [17:38] <kim0> I am however .. pasting it as well for archiving purposes
 175 [17:38] <kim0> #!/bin/bash
 176 [17:38] <kim0> set -eux # -x for verbose logging to ensemble debug-log
 177 [17:38] <kim0> ensemble-log "Installing drush,apache2,php via apt-get"
 178 [17:38] <kim0> apt-get -y install drush apache2 php5-gd libapache2-mod-php5 php5-cgi mysql-client-core-5.1
 179 [17:38] <kim0> a2enmod php5
 180 [17:38] <kim0> /etc/init.d/apache2 restart
 181 [17:38] <kim0> ensemble-log "Using drush to download latest Drupal"
 182 [17:38] <kim0> cd /var/www && drush dl drupal --drupal-project-rename=ensemble
 183 [17:39] <kim0> As you can see, it simply uses apt-get to install some packages, enables php, restarts apache
 184 [17:39] <kim0> and then uses "drush" the drupal shell, to download a fresh version of drupal
 185 [17:39] <kim0> it's interesting to see that ensemble does NOT force you to only use ubuntu packages
 186 [17:39] <kim0> you can mix and match like I just did
 187 [17:40] <kim0> as a developer, you can grab any code branch and build it and mix that with package installations, if you so wish
 188 [17:40] <kim0> Very well ..
 189 [17:40] <kim0> remember the magic that happened when we connected the two services together
 190 [17:40] <kim0> the "add-relation" command
 191 [17:40] <kim0> let us see the code that runs from drupal's perspective
 192 [17:41] <kim0> I have just displayed it on the shared screen
 193 [17:41] <kim0> I will paste it in chunks
 194 [17:41] <kim0> #!/bin/bash
 195 [17:41] <kim0> set -eux # -x for verbose logging to ensemble debug-log
 196 [17:41] <kim0> hooksdir=$PWD
 197 [17:41] <kim0> user=`relation-get user`
 198 [17:41] <kim0> password=`relation-get password`
 199 [17:41] <kim0> host=`relation-get host`
 200 [17:41] <kim0> database=`relation-get database`
 201 [17:42] <kim0> these last 4 lines, read various settings from Ensemble provided communication channels
 202 [17:42] <kim0> they read: user, password, host, database
 203 [17:42] <kim0> these settings are provided by the mysql service upon establishing the connection.
 204 [17:42] <kim0> drupal reads them, to configure itself to connect to that DB
 205 [17:43] <kim0> let us see how the connection happens
 206 [17:43] <kim0>     ensemble-log "Setting up Drupal for the first time"
 207 [17:43] <kim0>     cd /var/www/ensemble && drush site-install -y standard \
 208 [17:43] <kim0>     --db-url=mysql://$user:$password@$host/$database \
 209 [17:43] <kim0>     --site-name=Ensemble --clean-url=0
 210 [17:43] <kim0> easy enough!
 211 [17:43] <kim0> we just use drush to install drupal passing the need parameters we just consumed
 212 [17:43] <kim0> What is really important to notice here
 213 [17:44] <kim0> is that I might have never touched mysql before
 214 [17:44] <kim0> I do not have to know how it works or how to configure it
 215 [17:44] <kim0> or I cared about, was the Ensemble provided interface .. those 4 parameters in this case
 216 [17:45] <kim0> The actual script is slightly longer, since it checks whether or not this is first time drupal is being set up
 217 [17:45] <kim0> i.e. are we starting the very first drupal node, or just scaling it up
 218 [17:45] <kim0> oh yeah .. because scaling it up is as simple as
 219 [17:45] <kim0> ensemble add-unit mycms
 220 [17:45] <kim0> yep .. that's it
 221 [17:46] <kim0> So I hope this was a useful introduction to Ensemble
 222 [17:46] <kim0> I'll provide a couple of pointers and answer question
 223 [17:46] <kim0> questions*
 224 [17:46] <kim0> You can read more about Ensemble at: https://ensemble.ubuntu.com/
 225 [17:46] <kim0> Documentation: https://ensemble.ubuntu.com/docs/
 226 [17:46] <kim0> Mailing list: https://lists.ubuntu.com/mailman/listinfo/ensemble
 227 [17:47] <kim0> IRC Channel: #ubuntu-ensemble @ Freenode
 228 [17:47] <kim0> we are a very welcoming community .. it's a party and everyone is invited :)
 229 [17:47] <kim0> So start hacking on your own formula today!
 230 [17:47] <kim0> You can see a list of existing formulas at: https://code.launchpad.net/principia
 231 [17:48] <kim0> Our goal is to cover all of free software!
 232 [17:48] <kim0> Alright .. taking questions
 233 [17:48] <ClassBot> alexm asked: can emsemble be used to deploy outside the cloud?
 234 [17:48] <kim0> alexm: Ensemble uses multiple "providers" to handle what to deploy to
 235 [17:49] <kim0> ec2 cloud is currently the one that is ready, a provider for LXC (Linux container) is being worked on heavily
 236 [17:49] <kim0> and myunderstanding is that it will be ready during this 11.10 cycle
 237 [17:49] <kim0> you can also deploy to private clouds such as openstack or eucalyptus .. not sure how well tested that is, but should mostly work
 238 [17:50] <kim0> Also, Ensemble is being currently integrated with Orchestra
 239 [17:50] <kim0> the Ubuntu server hardware deployment tool
 240 [17:50] <kim0> so you will be able to install to your own hardware farm as well
 241 [17:50] <kim0> I hope that answers your question
 242 [17:51] <ClassBot> There are 10 minutes remaining in the current session.
 243 [17:51] <ClassBot> chadadavis asked: Does ensemble have upstart's concepts of services vs tasks. Backup seems more like a service, whereas migration more like a task.
 244 [17:51] <kim0> Things are a bit too early in Ensemble development right now .. Currently everything is a service
 245 [17:52] <kim0> however the dev team recognize the need that some operation have special needs
 246 [17:52] <kim0> "machine services" are being debated
 247 [17:52] <kim0> various types of services (or whatever they'd be called) can definitely arise
 248 [17:52] <ClassBot> ahs3 asked: each service seems to be a separate machine; can i have multiple services on the same machine?
 249 [17:53] <kim0> ahs3: good question .. With the LXC provider that will become possible
 250 [17:53] <kim0> there is also design work currently to colocate chosen services together
 251 [17:53] <kim0> interested code hackers are invited to hack on ensemble core to make this happen faster :)
 252 [17:53] <kim0> #ubuntu-ensemble
 253 [17:54] <kim0> Ensemble itself is written in python using twisted framework afaik
 254 [17:54] <ClassBot> ahs3 asked: how do i find out what services are available?
 255 [17:54] <kim0> ahs3: As mentioned this is the current list
 256 [17:54] <kim0> https://code.launchpad.net/principia
 257 [17:54] <kim0> this only started a few weeks back
 258 [17:54] <kim0> so it is growing rapidly
 259 === janimo` is now known as janimo
 260 [17:55] <kim0> I am more than ready to help anyone write his own first formula
 261 [17:55] <kim0> just grab me (kim0) on #ubuntu-ensemble
 262 [17:55] <kim0> and start hacking on your own formula :)
 263 [17:55] <ClassBot> alexm asked: how does mycms know that it has to use mydb? i.e. mycms is deployed with another db before connecting to mydb?
 264 [17:55] <kim0> alexm: good question
 265 [17:55] <kim0> it doesn't
 266 [17:55] <ClassBot> There are 5 minutes remaining in the current session.
 267 [17:55] <kim0> I tell it to use mydb
 268 [17:55] <kim0> when I issued
 269 [17:56] <kim0> add-relation mycms mydb:db
 270 [17:56] <kim0> This is exactly what "connects" or "relates" them together
 271 [17:56] <kim0> before add-relation they have no idea about each other
 272 [17:56] <kim0> and mycms does NOT have another DB .. in fact it has no db at all and is mostly misconfigured
 273 [17:57] <kim0> Awesome .. that concludes this session
 274 [17:57] <kim0> feel free to ask me any further questions in #ubuntu-ensemble
 275 [17:57] <kim0> till next time .. c ya