ServerMaverickCloudUtils

Summary

In the 10.04 Lucid cycle, we added a package called 'cloud-utils' that provided programs to address some of the largest pain points in working with images and instances on UEC. Those included:

  • ssh-import-lp-id
  • uec-publish-image
  • uec-publish-tarball
  • uec-query-build
  • uec-resize-image

We would like to continue adding nicer utilities for interacting with UEC/EC2 clouds.

Release Note

Ubuntu 10.10 includes utilities to make interacting with UEC and EC2 easier. Tasks such as "run an instance and wait until it is up" and "cleanly rebundle an image" can be accomplished with one command.

Rationale

The ec2-api-tools, ec2-ami-tools, and euca2ools are complex and comprehensive commands. Howerver, they're not the most user friendly commands. In order to make UEC and EC2 more easily available we will develop some tools that simplify complex operations.

User stories

  • Jenna wants to run an instance of Ubuntu 10.10 on EC2 and then do something with it. She is not so much interested in knowing when the request has been made as when the system is available to ssh to and use.
  • Joey wants to create an appliance based on an Ubuntu image. He is not quite sure about how to go about it. There is {euca|ec2}-bundle-vol and ec2-create-image, but these commands are either complex to use or require cleaning of an already booted system. He would like to launch a single command to modify a pristine Ubuntu image and make a re-bundled image from that.

Assumptions

Design

The following is a list of functionality we would like to add. It is generally ordered by perceived value. Changes will primarily be done as additions to the existing 'cloud-utils' package.

  • rebundle image (ec2-ubuntu-base)

    • support for re-bundling an image to EBS
    • support for re-bundling an image to instance store
    • support for "let the user do some things by hand" to the image then continue. This would basically do everything that you need to do to bundle an image cleanly, but allow the user to do some things by hand in the middle (as opposed to by a script).
  • run instance
    • and wait till it is reachable via ssh
    • and attach IP address
    • and run command, then tear down (this might not be so wise as network might be unreliable for long running command)
  • name based launching, or globally s/id/name (add name stored in s3 wherever an id is all thats available. for the places where user-defineable string isavailable (ie, ec2-register --nam), then use it.

  • 556528 presence of a ~/.eucarc breaks environment based variable settings

  • uec-rebundle image
    • published image to tarfile that can then be consumed by uec-publish-tarball
  • uec-cc-setup (euca_conf --get-credentials, ln -sf .euca/eucarc ~/.eucarc, euca-add-keypair, euca-authorize-default ssh, setup ~/.ssh/config to use keypair for each address in euca-describe-addresses)
  • kill all instances
  • delete instance (by ami or name), optionally unregister kernel/ramdisk
    • delete all in a bucket (unregistering)
  • run instance
    • add data to s3 (expiring URL) pass in user-data

Implementation

Individual Tasks will be added to the Work Item tracker, see the whiteboard at server-maverick-cloud-utils

Code Changes

Migration

Users rebundle Ubuntu images in a number of different ways:

We would like to be able to move users of these methods to the "suggested method". That will include making users aware of the preferred method (blog/documentation).

Test/Demo Plan

TODO Where possible, we'd like for these utilities to be tested by our tests, and even our nightly build process (automated-ec2-builds, ec2-publishing-tools). That would serve as a good amount of test of the utilities.

  • rebundle image: rebundle an existing image, register it, make sure it boots and represents the changes that were made.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

BoF agenda and discussion

In lucid, we added a package named 'cloud-utils', that was a box for placing utilities that made dealing with life in UEC or ec2 easier.   This included 'uec-publish-image' and 'uec-publish-tarball', 'uec-run-instances', and 'ssh-import-lp-id'.  Certainly there are things that can be done to continue making things even better.  In this session we would really like for people to mention their pain points with UEC or EC2, and discuss how we can address those.

Some issues are:
  * images are referred to by a non-human friendly number prefixed by 'emi-' or 'ami-'.  This number differs across instances of a cloud.  It might be nice to a.) provide a way to give a symbolic name to an id such that it could be resolved for the target cloud b.) host a set of well known names/ids.
  * after deploying a fresh UEC, the admin immediately has to do some things to make it functional (http://testcases.qa.ubuntu.com/Install/ServerENode)
  * no automated mechanism for adding users --[nurmi says coming in 10.10 !!!]
  * presence of a ~/.eucarc breaks environment based variable settings (smoser needs to open a bug for this)

Thoughts of utilities for making UEC life easier:
 * uec-rebundle-image.  This would provide much the same purpose of euca-rebundle-vol, but would do so by downloading pristine tar image, mounting it, and then allowing the user to modify it either via script or possibly interactively.  The benefit would be in working with the un-booted image, that hasn't done "first boot" already. (lp:~smoser/+junk/ec2-ubuntu-base)
 * uec-cc-setup : When doing many cloud installs, a few things are often done before it can actually be used.  This includes 'euca_conf get-credentials', unzip/setup credentials, euca-add-key, modify .ssh/config to provide that key for hosts listed in describe-addresses...
 * name based launching of EC2 images:
  * uec-run-instances lucid i386 released
  * uec-run-instances maverick x86_64 daily
  *  uec-run-instance-and-wait
  *  uec-run-instances
    * add support for launching by name (via querying http://uec-images.ubuntu.com/query/)
  * published image to tarfile that can then be consumed by uec-publish-tarball
  * delete (unregister and remove) by ami id
  * kill all instances
  
  
   * naming of ebs volumes, or generic naming of anything that has a silly uid
   * etienne : request bench mark utilities ... an idea of "how much overhead is there"

CategorySpec

ServerMaverickCloudUtils (last edited 2010-05-26 15:06:05 by d14-69-66-169)