Ubuntu Wiki

• Launchpad Entry: server-lucid-xc2

• Created: 2009-11-24

• Contributors: Scott Moser

• Packages affected: ubuntu-cloud-utils (to be packaged/created)

Summary

This spec will cover adding a new package to ubuntu that includes more featureful command line utilities for interacting with EC2 and/or UEC. Currently there exist ec2-{ami,api}-tools and work-alike euca2ools.

This spec covers providing some additional tools to make interacting with UEC or ec2 more convienent on ubuntu. The tools are possibly ubuntu specific.

Release Note

New command line utilities are provided for interacting with UEC and EC2. These utilities allow higher level commands, making it easier to launch instances, upload images, or otherwise work with EC2 or UEC.

Rationale

Interacting with ec2 and UEC with the command line tools that are provided in ec2-ami-tools, ec2-api-tools, or euca2ools is often tedious and error prone. The changes here will make the following items much easier:

• Launch a ubuntu instance of a given release without knowing ami/aki
• specify configuration changes to make to the image (additional packages)
• launch an instance and when it is available securely connect to it via ssh

User stories

• As a ubuntu developer, I would like to easily start an instance of the latest ubuntu development series with some additional packages installed.

Assumptions

Design

This will likely be implemented in 3 different parts:

• xc2 : this provides a very basic wrapper around ec2-ami-tools/ec2-api-tools or euca2ools.
  • Script utilities can use this layer of indirection to get one or the other.
  • It can also possibly fix small incompatibilities between the two sets of tools.
  • Caching would be greatly beneficial for scripts that invoke ec2-describe-images as this command takes several seconds, but server side data changes fairly infrequently.
  • config file based configuration rather than environment (ie, by default read ~/.xc2rc. or allow '--config ~/my-aws-config.txt). Ideally config file format is compatible with what we decide on in euca2ools

• ubec2 (<insert-updated-name-here>) : this utility will sit on top of top of the euca2ools python library and/or boto and provide higher level tools.

  • ubec2 launch latest ubuntu karmic-daily
  • provide defaults for values such as size (--instance-type) and key-pair (--key) based on config
  • ubec2
• general tools for working with ec2/uec (uploading images, resizing images ...)

Implementation

xc2

For xc2, there exists a reasonalbe starting point, the new funciton to add to it are listed as work items in the whiteboard.

ubec2

This tool will require interoperation with other tools or data:

• getting the latest released image requires repository that the tool can query for that information. (EC2UpgradesSpec)

• config file syntax of images (ServerLucidCloudConfig)

• ssh key passing syntax (ServerLucidBootHooks)

The following are items to support:

• mechanism of specifying ubuntu release/arch by name rather than ami id. This is coupled with server side data.

  • ubec2 --release karmic --arch i386 launch
    ubec2 --release 8.10 --arch amd64 launch

• mechanism for writing config file syntax based on command line input. The following sets the 'foo' config item to 'bar' and 'wark' item to 'abcde fghi'

  • ubec2 launch --config-foo=bar --config-wark="abcde fghi"

• mechanism for generating ssh keys on launch, and passing them to image:

  • ubec2 launch --ssh-gen

• There should be some mechanism for also writing the ssh keys that were generated to a file that can be used as an ssh known-hosts file. This will have to happen after the instance has a hostname, so without a block, that can't happen immediately on launch.

• ssh key generation will have to support multiple instances (--instance-count)
• combined "start instance, wait for it to come up, VERIFY THE SSH KEY FINGERPRINT, and ssh into it" command

other tools

• publish-image, perhaps others lp:~ubuntu-on-ec2/ubuntu-on-ec2/ec2-publishing-scripts/

• resize-uec-image lp:~ubuntu-on-ec2/ubuntu-on-ec2/uec-tools

• register-uec-tarball lp:~ubuntu-on-ec2/ubuntu-on-ec2/uec-tools

Migration

This is all new function/feature so no migration is necessary.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

Updated Plan 20100202

Due to recent features in the ec2-api-tools (EBS root), the lack of those features in UEC, instead of many of the above, I suggest to add:

• add a cloud-utils package, a place for uec and/or ec2 tools that don't fall elsewhere

• add resize-uec-image to cloud-utils

• add register-uec-tarball

• add published data of ec2 ami at uec-images.ubuntu.com that can be queried to find "latest released image" and such
• add 'ubuntu-ec2-query' command that would

  • support "<release> <region>" outputting the latest AMI

• add tab completion to euca2ools
• MIR for cloud-utils
• add support for specifying '--key' flag to euca2ools via config
• add run-instance-and-wait command
• add run-instance-and-connect command (or flag to above)

BoF agenda and discussion

Agenda

• euca2ools compatibility with EC2
• euca2ools syntax compatibility with ec2-api-tools, ec2-ami-tools
• higher level CLI tools
  • specify AMI by release/type/etc. rather than hex string
  • read configuration files rather than lots of environment variables
• configuration based keys (read ~/.xc2rc)
• aggregate "bundle upload and register"
• name/aliases for aki/ami/ari
• configuration include
  • access key
  • secret key
  • ec2_cert ec2_private key path
  • keys
  • EC2 URL
• combined "start instance, wait for it to come up, VERIFY THE SSH KEY FINGERPRINT, and ssh into it" command
  • and check fingerprints
• tab completion
• snapshottin
• ec2-consistent-snapshot
• un-register and delete
• wishlist: expire snapshot
• generate ssh key locally and configure the instance to use it?
• Completely terminate/delete/clean up after instance

other examples

• read the ruby gems from rightscale

Example

ubuntu-ec2 start
        -r/--release karmic
        -t/--instance-type large
        --apply-updates
        --add-packages ...
        --ssh
        ...

ubuntu-ec2 run-instances ...

CategorySpec