What is Ensemble? - Presentation and Demo -- SpamapS

   1 [18:01] <ClassBot> Logs for this session will be available at http://irclogs.ubuntu.com/2011/03/23/%23ubuntu-classroom.html following the conclusion of the session.
   2 [18:02] <SpamapS> So, I have prepared a short set of slides to try and explain what Ensemble is here: http://spamaps.org/files/Ensemble%20Presentation.pdf
   3 [18:02] <SpamapS> I will elaborate here in channel.
   4 [18:03] <SpamapS> Ensemble is an implementation of Service Management
   5 [18:03] <SpamapS> up until now this has also been called "Orchestration", and the term is not all that inaccurate, though I feel that Service Management is more appropriate
   6 [18:03] <SpamapS> "What is Service Management?"
   7 [18:04] <SpamapS> Service Management is focused on the things that servers do that end users consume
   8 [18:04] <SpamapS> Users connect to websites, dns servers, or (at a lower level) databases, cache services, etc
   9 [18:04] <SpamapS> Ensemble models how services relate to one another.
  10 [18:05] <SpamapS> Web applications need to connect to a number of remote resources. Load balancers need to connect to web application servers.. monitoring services need to connect to services and test that they're working.
  11 [18:05] <SpamapS> Ensemble models all of these in what we call "formulas" (more on this later)
  12 [18:06] <SpamapS> If this starts to sound like Configuration Management, you won't be the first to make that mistake.
  13 [18:06] <SpamapS> However, this sits at a higher level than configuration management.
  14 [18:06] <SpamapS> "Contrast With Configuration Management"
  15 [18:07] <SpamapS> Configuration management grew from the time when we had a few servers that were expensive to buy/lease/provision, and lived a long time.
  16 [18:07] <SpamapS> Because of this, system administrators modeled system configuration very deeply. Puppet, chef, etc., first and foremost, model how to configure *a server*
  17 [18:08] <SpamapS> As the networks grew and became more dependent on one another, the config management systems have grown the ability to share data about servers.
  18 [18:08] <SpamapS> However the model is still focused on "how do I get my server configured"
  19 [18:09] <SpamapS> Ensemble seeks to configure the service.
  20 [18:09] <SpamapS> With the cloud, we have the ability to rapidly provision and de-provision servers. So service management is tightly coupled with provisioning.
  21 [18:10] <SpamapS> Chef, in particular, from the config management world, has done a good job of adding this in with their management tools.
  22 [18:10] <SpamapS> However, where we start to see a lot of duplication of work in configmanagement, is in sharing of the knowledge of service configuration.
  23 [18:11] <SpamapS> Puppet and Chef both have the ability to share their "recipes" or "cookbooks"
  24 [18:11] <SpamapS> However, most of these are filled with comments and variables "change this for your site"
  25 [18:11] <SpamapS> The knowledge of how and when and why is hard to encode in these systems.
  26 [18:12] <SpamapS> Ensemble doesn't compete directly with them on this level. Ensemble can actually utilize configuration management to do service management.
  27 [18:12] <SpamapS> The comparison is similar to what we all used to do 15+ years ago with open source software
  28 [18:13] <SpamapS> download tarball, extract, cd, ./configure --with-dep1=/blah && make && sudo make install
  29 [18:13] <SpamapS> This would be an iterative process where we would figure out how to make the software work for our particular server every time.
  30 [18:13] <SpamapS> Then distributions came along and created packaging, and repositories, to relieve us from the burden of doing this for *most* low level dependencies.
  31 [18:14] <SpamapS> So ensemble seeks to give us, in the cloud, what we have on the single server.. something like 'apt-get install'
  32 [18:14] <SpamapS> "Terms"
  33 [18:15] <SpamapS> "service unit" - for the most part this means "a server", but it really just means one member of a service deployment. If you have 3 identical web app servers, these are 3 service units, in one web app service deployment.
  34 [18:16] <SpamapS> "formula" - this is the sharable, ".deb" for the cloud. It encodes the relationships and runtime environment required to configure a service
  35 [18:17] <SpamapS> "environment" - in ensemble, this defines the machine provider and settings for deploying services together. Right now this means your ec2 credentials and what instance type. But it could mean a whole host of things.
  36 [18:17] <SpamapS> "bootstrap" - ensemble's first job in any deployment is to "boostrap" itself. You run the CLI tool to boostrap it
  37 [18:18] <SpamapS> that means it starts a machine that runs the top level ensemble agent that you will communicate with going forward
  38 [18:18] <SpamapS> "Basic Workflow"
  39 [18:19] <SpamapS> This is how we see people using ensemble, though we have to imagine the details of this will change as ensemble grows, since it hasn't even been "released" yet.
  40 [18:19] <SpamapS> (though, as a side note, it is working perfectly well, and available for lucid at https://launchpad.net/~ensemble/+archive/ppa)
  41 [18:20] <SpamapS> 0. (let this out of the slide) - configure your environment. This means establish AWS credentials, and record them in .ensemble/environment.yaml
  42 === daker is now known as daker_
  43 [18:20] <SpamapS> 1. Bootstrap (ensemble bootstrap) - this connects to your machine provider (EC2 right now) and spawns an instance, and seeds it using cloud-init to install ensemble and its dependencies
  44 [18:21] <SpamapS> 2. Deploy Services (ensemble deploy mysql wiki-db; ensemble deploy mediawiki demo-wiki)
  45 [18:21] <SpamapS> This actually spawns nodes with the machine provider, and runs the ensemble agent on them, telling them what service they're a part of and running the service "install" hooks to get them ready to participate in the service
  46 === niemeyer is now known as niemeyer_bbl
  47 [18:22] <SpamapS> 3. Relate Services (ensemble add-relation demo-wiki:db wiki-db:db)
  48 [18:23] <SpamapS> This part won't always be necessary. Automatic relationship resolution is being worked on right now. But sometimes you will want to be explicit, or do a relation that is optional.
  49 [18:23] <SpamapS> In the example above, this tells demo-wiki and wiki-db about eachother. I will pastebin a formula example to clear this up.
  50 [18:24] <SpamapS> http://paste.ubuntu.com/584424/
  51 [18:24] <SpamapS> This is the metadata portion of the mediawiki formula, which I created recently as part of the "Principia" project, which is a collection of formulas for ensemble: https://launchpad.net/principia
  52 [18:25] <SpamapS> If you look there, you see that it 'requires:' a relationship called 'db'
  53 [18:25] <SpamapS> the interface for that relationship is "mysql"
  54 [18:25] <SpamapS> These interface names are used to ensure you don't relate two things which have different interfaces
  55 [18:26] <SpamapS> (almost done will take questions shortly)
  56 [18:26] <SpamapS> http://paste.ubuntu.com/584425/
  57 [18:26] <SpamapS> This is the corresponding metadata for mysql..
  58 [18:27] <SpamapS> as you see, it provides a relationship called 'db' as well, which uses the interface 'mysql'
  59 [18:27] <SpamapS> What this means is that the 'requires' side of the formula can expect certain data to be passed to it when it joins this relationship
  60 [18:28] <SpamapS> and likewise, the provides side knows that its consumers will need certain bits
  61 [18:28] <SpamapS> When this relationship is added, "hooks" are fired
  62 [18:29] <SpamapS> These are just scripts that are run at certain events in the relationship lifecycle
  63 [18:29] <SpamapS> These scripts use helper programs from ensemble to read and write data over the two-way communication channel.
  64 [18:30] <SpamapS> In the case of mysql, whenever a service unit joins a relationship, it creates a database for the service if it doesn't exist, and then creates a username/password/etc. and sends that to the consumer
  65 [18:31] <SpamapS> and the mediawiki hook for the relationship will configure mediawiki to use that database
  66 [18:31] <SpamapS> The code for all of this is in lp:principia if you are curious.
  67 [18:32] <SpamapS> the final slide is just an overview of ensemble's architecture under the hood.
  68 [18:32] <SpamapS> I will take questions now...
  69 [18:34] <SpamapS> marktma: GREAT question. Definitely. One of the goals is to make it easy to write new "machine providers". By doing EC2 first though, we should have a reasonable chance at working with UEC/Eucalyptus and maybe even OpenStack out of the box.
  70 [18:35] <ClassBot> marktma asked: is there any chance ensemble will be used for private clouds as well?
  71 [18:35] <SpamapS> hah, ok, see answer ^^
  72 [18:35] <ClassBot> kim0 asked: What does the interface: mysql .. actually mean
  73 [18:35] <SpamapS> I think I may have answered that already in the ensuing description..
  74 [18:36] <SpamapS> but essentially its a loose contract between providers/requirerers/peers on what will be passed through the communication channel
  75 [18:37] <ClassBot> EvilPhoenix asked: (for kim0): that contract .. is it defined somewhere
  76 [18:38] <SpamapS> It is only defined via the formulas. It is intentially kept as a loose coupling to make formulas flexible. I could see it being strengthened a bit in the future.
  77 [18:39] <SpamapS> Now, I wanted to stream my desktop to demo ensemble in action..
  78 [18:39] <SpamapS> but that has proven difficult given the 20 minutes I had to attempt to set that up.
  79 [18:39] <SpamapS> So I will paste bin the terminal output of an ensemble run...
  80 [18:40] <SpamapS> I have setup a lucid instance for this, and the only commands not seen here are: sudo add-apt-repository ppa:ensemble/ppa ; apt-get update ; apt-get install ensemble ; bzr branch lp:principia ; cat > aws.sh
  81 [18:40] <SpamapS> the last bit is to store my aws credentials
  82 [18:41] <SpamapS> http://paste.ubuntu.com/584430/
  83 [18:41] <SpamapS> this is the boostrap phase
  84 [18:41] <SpamapS> bootstrap even
  85 [18:42] <SpamapS> I now need to wait for EC2 to start an instance
  86 === daneroo_ is now known as daneroo
  87 [18:42] <SpamapS> ubuntu@ip-10-203-81-87:~$ ensemble status
  88 [18:42] <SpamapS> 2011-03-23 18:42:02,263 INFO Connecting to environment.
  89 [18:42] <SpamapS> 2011-03-23 18:42:18,586 INFO Environment still initializing.  Will wait.
  90 [18:43] <SpamapS> And now it has spawned my bootstrap
  91 [18:43] <SpamapS> there will be live DNS names here, so hopefully my security groups will keep your prying eyes out..
  92 [18:43] <SpamapS> machines: 0: {dns-name: ec2-50-17-142-155.compute-1.amazonaws.com, instance-id: i-10f63f7f}
  93 [18:43] <SpamapS> services: {}
  94 [18:43] <SpamapS> 2011-03-23 18:42:50,216 INFO 'status' command finished successfully
  95 [18:43] <ClassBot> TeTeT asked: what would a system administrators task with ensemble be - write formulas or just deploy them or a mix?
  96 [18:44] <SpamapS> I'd imagine sysadmins would write the formulas for an organization's own services which consume existing services.
  97 [18:44] <SpamapS> The common scenario is a LAMP application which takes advantage of memcached, mysql, and has a load balancer
  98 [18:45] <SpamapS> The lamp app needs to have its config files written with the db, cache servers, etc., so the sysadmin would write the relation hooks for mysql and memcached. OR a developer could write these. The devops paradigm kind of suggests that they work together on this.
  99 [18:46] <SpamapS> Ok now I'll run my "demo.sh" script which builds a full mediawiki stack
 100 [18:47] <SpamapS> While this is going, I will stress that this is *unreleased* alpha software, though the dev team has been very dilligent and the code is of a very high quality (written in python with twisted, and available at lp:ensemble
 101 [18:47] <SpamapS> http://paste.ubuntu.com/584433/
 102 [18:47] <SpamapS> Now we'll need to wait a few minutes while all of those nodes spawn
 103 [18:49] <SpamapS> Now, I'm using t1.micro, so these provision *fast* .. we can watch their hooks run w/ debug-log...
 104 [18:49] <SpamapS> However they may already be done..
 105 [18:50] <SpamapS> Ideally, we'll have a wiki accessible at the address of 'wiki-balancer' .. lets see
 106 [18:51] <ClassBot> There are 10 minutes remaining in the current session.
 107 [18:51] <ClassBot> TeTeT asked: is the deployment through ensemble itself or via cloud-init or puppet or other config tools?
 108 [18:52] <SpamapS> http://paste.ubuntu.com/584437/
 109 [18:52] <SpamapS> While you guys try to decipher that I'll answer TeTeT
 110 [18:52] <SpamapS> TeTeT: the nodes are configured via cloud-init to run ensemble's agent. After that, ensemble is in control running hooks. The formulas are pushed into S3, and then downloaded by the agent once it starts.
 111 [18:53] <SpamapS> So unfortunately, our load balancer has failed.. it is "machine 4" http://ec2-50-17-47-115.compute-1.amazonaws.com/ ... but the individual mediawiki nodes *are* working..
 112 [18:53] <SpamapS> http://ec2-204-236-202-35.compute-1.amazonaws.com/mediawiki/index.php/Main_Page
 113 [18:55] <SpamapS> Ahh, there was a bug in my demo.sh :)
 114 [18:55] <SpamapS> $ENSEMBLE add-relation wiki-balancer demo-wiki:reverseproxy
 115 [18:55] <SpamapS> mediawiki has no relation named reverseproxy
 116 [18:55] <SpamapS> 2011-03-23 18:46:37,900 INFO Connecting to environment.
 117 [18:55] <SpamapS> No matching endpoints
 118 [18:55] <SpamapS> 2011-03-23 18:46:38,473 ERROR No matching endpoints
 119 [18:55] <SpamapS> 2011-03-23 18:46:38,865 INFO Connecting to environment.
 120 [18:56] <SpamapS> We actually had that error but missed it. ;)
 121 [18:56] <SpamapS> lets relate the load balancer now
 122 [18:56] <ClassBot> There are 5 minutes remaining in the current session.
 123 [18:56] <SpamapS> ubuntu@ip-10-203-81-87:~$ ensemble add-relation wiki-balancer:reverseproxy demo-wiki:website
 124 [18:56] <SpamapS> 2011-03-23 18:56:28,059 INFO Connecting to environment.
 125 [18:56] <SpamapS> 2011-03-23 18:56:28,691 INFO Added http relation to all service units.
 126 [18:56] <ClassBot> kim0 asked: Can't a cache and a wiki service-units share the same ec2 instance
 127 [18:56] <SpamapS> 2011-03-23 18:56:28,691 INFO 'add_relation' command finished successfully
 128 [18:57] <SpamapS> kim0: the idea is that in that instance, its simpler to use something like LXC containers to make it easier to write formulas. However, in the case of purely non-conflicting formulas, there should be a way in the future to do that yes
 129 [18:57] <SpamapS> http://ec2-50-17-47-115.compute-1.amazonaws.com/mediawiki/index.php/Main_Page
 130 [18:57] <SpamapS> And there you have a working mediawiki
 131 [18:58] <ClassBot> TeTeT asked: will ensemble also provide service monitoring, or is that better left to munin/nagios and alike
 132 [18:59] <SpamapS> TeTeT: The latter. nagios/munin/etc are just services in themselves. And they speak the same protocols as consuming services. If a formula wants to explicitly expose *more* over a monitoring interface they certainly can
 133 [18:59] <SpamapS> I think thats about all the time we have
 134 [19:00] <SpamapS> Thanks so much for taking the time to listen. https://launchpad.net/ensemble has more information!

UbuntuCloudDays/23032011/WhatIsEnsemble (last edited 2011-03-26 17:02:24 by nigelbabu)