Ubuntu Wiki

This document covers the ec2 publishing process. The commands listed below are intended to be run on the canonical build machine (nectarine), but should be able to run on other systems that have the requirements listed below.

They ensure that builds are published according to NamingConvention.

Canonical AWS Credentials are described at on the canonical wiki.

Requirements

The vmbuilder user that does the daily builds has environment set up as described below. If you are attempting this from another system you will need the following:

• Source / Tools

  • ec2-publishing-scripts

    • bzr branch lp:~ubuntu-on-ec2/ubuntu-on-ec2/ec2-publishing-scripts

  • xc2

    • git clone http://smoser.brickies.net/git/misc-starter-tools.git

  • ec2-ami-tools, ec2-api-tools (from multiverse)

    • apt-get install ec2-ami-tools ec2-api-tools

  • vm-builder branch 0.11.

  • recent boto (1.9b works, a dependency for some xc2 operations)
  • s3cmd (s3cmd-0.9.9.91 is version used)

  • For kernel and ramdisk publishing, the user must have ec2 tools for 'ec2-bundle-kernel' and 'ec2-bundle-ramdisk'. These must also be available in PATH (these tools are available only under amazon NDA, but are not needed after 10.04 as we use amazon's pv-grub kernels).

• AWS Credentials
  • The user must have their environment set up such that xc2 runs with the correct account without a '-c' argument. Specifically, that means that one of
  • ~/.xc2rc exists and is setup correctly

  • environment variable XC2_DEF_CFG is defined, pointing to a valid xc2 config file

• PATH

  • on the build machine, there is a a path/environment setup wrapper named 'cronrun available in '~vmbuilder/bin/cronrun'. It comes from automated-ec2-builds. Below, the shell var 'cronrun' is expected to be set or path otherwise set up correctly.

    cronrun=/home/vmbuilder/bin/cronrun

  • One way or another, PATH is expected to include 'xc2' and 'ec2-publishing-scripts'.
• build output. To get build output, you can:

  • rsync -aP uec-images.ubuntu.com::uec-images/karmic/current . 

    Note, daily-build-publisher expects that 'current' is a symbolic link to a directory whose path ends in YYYYMMDD format.

Important

Please read this!

Any long running commands described on this page should be run from inside a GNU screen session. Connectivity can be dropped for any reason, and you don't want to lose state because of that.

Publishing

Daily

Daily builds are generally automatically published by the daily build script (uec-build-launcher), which invokes publish-build when it is through. If you so desire, you can invoke a daily publish by hand with:

sudo -u vmbuilder ${cronrun} \
   publish-build --verbose lucid server daily /srv/ec2-images/server/lucid/current

publish-build calls 'publicize-build', which gives launch access to 'all' and aso updates the query data in /srv/ec2-images/query .

Milestone / Release Publishing

A release milestone or release build is published using the same tools that the daily build launcher invokes, simply with different arguments.

Because publishing is a fairly long process which can take upwards of 2 hours, it is recommended that a publish is done ahead of time without giving launch permissions. Then, later, when the release is ready to go 'launch-permission' is given to 'all' for the previously registered images and kernels/ramdisks.

Pre-publish the build, this will publish to EC2 and take care of all operations that take a large amount of time:

LABEL=beta2 # or beta1 or alpha1 or 'release'
sudo -u vmbuilder ${cronrun} promote-daily ${LABEL} ${BUILD_D}

Make the build public:

sudo -u vmbuilder ${cronrun} promote-daily --verbose --make-public ${LABEL} ${BUILD_D}

It will write a file in '${BUILD_D}' named 'published-ec2-release.txt' with the ids of the newly published images. (Note, '-release' is for alpha, beta, rc, for daily it will be 'published-ec2-daily.txt').

If the command above fails due to network issues or some other sporadic failure, it can safely be re-run with:

sudo -u vmbuilder ${cronrun} promote-daily ${LABEL} ${BUILD_D} --verbose --allow-existing [--make-public] 

Publishing release on uec-images

Use promote-daily as above, but with 'release' as the label.

Manually starting Builds

Builds are done with regular schedules via cron. The crontab that dictates this is managed in the 'automated-ec2-builds' branch here.

If you need to start a build manually, that can be done by

$ SUITE=maverick
$ BUILD_NAME=server
$ ~vmbuilder/bin/vbcron build-daily ${SUITE} ${BUILD_NAME}

After the build is finished, it can be promoted as described above.

Updating Amazon Pages

This is a manual process, unfortunately. To reduce likelyhood of human error when editing the pages, I've added the pages to revision control at lp:~ubuntu-on-ec2/ubuntu-on-ec2/ami-pages/. I edit the pages there, and then start up firefox and copy and paste the content.

Below are AMI pages that are registered at AWS:

Follow the link, log in (as canonical editor account) and edit ("Manage Document") the page. These pages are created by adding a page from Amazon's Submit a Resource

Publishing Hardy Kernels

To collect kernel and ramdisk, for each arch, you will need to:

• launch an instance and ssh to it (choose the lastest released image from uec-images)

• Then install kernel and collect ramdisk:

  ## MODIFY THIS ###
  kver="2.6.24-11-xen"
  
  # update and upgrade to be current (some utils copied to ramdisk)
  sudo apt-get update && sudo apt-get dist-upgrade
  
  # add ubuntu-on-ec2 ppa  and key 9EE6D873 
  echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ppa/ubuntu hardy main" |
     sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2.list
  sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
  sudo apt-get update
  
  # download and install the kernel with apt:
  sudo apt-get install linux-image-${kver} linux-modules-${kver}
  # OR, download and dpkg
  sudo dpkg -i linux-image-${kver}*.deb linux-modules-${kver}*.deb
  
  rel="hardy"
  fmt="ubuntu-${rel}-\${Architecture}-\${Package}-v-\${Version}"
  bname=$(dpkg-query --show --showformat "${fmt}" linux-image-${kver})
  rdnum=$(date +%Y%m%d)
  
  ## copy kernel and ramdisk and tar it up
  cp /boot/vmlinuz-${kver} ${bname}-kernel.img
  cp /boot/initrd.img-${kver} ${bname}-ramdisk.${rdnum}.img
  tar -cvzf ${bname}.tar.gz ${bname}*.img

• Upload the kernel and ramdisk to s3 bucket s3://ubuntu-data/hardy-kernels/ so they show up in http://ubuntu-data.s3.amazonaws.com/hardy-kernels/index.html .

• publish the kernels and ramdisks to EC2:

  XC2_DEF_CFG=~/data/aws-canonical-smoser/info \
    PATH=$HOME/src/ec2-publishing-scripts/trunk:$PATH \
    sh -c 'for f in "$@"; do 
       t=kernel; [ "${f#*ramdisk}" = "${f}" ] || t=ramdisk ;
       a=amd64; [ "${f#*amd64}" = "${f}" ] || a=i386;
       publish-image --bucket-prefix ubuntu-kernels-testing \
          --type $t --output registered.txt --verbose --add-launch all \
          $a $f || exit;
       done' arg0 *-kernel.img *-ramdisk*.img

Length of Operations

The following is relevant as of 2010-10-05, obviously many things can affect how long operations take, so read this with a grain of salt. That said, its very useful to have a general idea about what to expect.