• Launchpad Entry: server-lucid-eucalyptus-merging-and-packaging

• Created: 2009-11-23

• Contributors: DustinKirkland, ThierryCarrez

• Packages affected: eucalyptus, libvirt, qemu-kvm

Summary

Eucalyptus is a massive upstream project, that aims to work under multiple distributions. Ubuntu has worked extensively to package and enhance Eucalyptus in order to deliver the Ubuntu Enterprise Cloud UEC). Because of this, there is now some "drift" between the upstream branch and the ubuntu branch:

• lp:eucalyptus
• lp:~ubuntu-core-dev/eucalyptus/ubuntu

This specification describes the work items related to the merging and packaging of the upstream Eucalyptus project, for the Ubuntu Enterprise Cloud in Lucid.

Release Note

Ubuntu 10.04 LTS builds its Ubuntu Enterprise Cloud on an updated, bug-fix-only Eucalyptus release.

Rationale

It is in the interest of Ubuntu Server developers, Eucalyptus upstream developers, and all UEC/Eucalyptus users to have a well-maintained Ubuntu package, with minimal delta from the upstream code to ensure that:

1. Ubuntu's merging of new Eucalyptus versions happens with as few conflicts as possible
2. Eucalyptus upstream and other users get the benefit of some of Ubuntu's changes
3. Code that each Eucalyptus user is running is as close to upstream mainline as possible, in the interest of shared testing, bug triaging, and support

User stories

• As an Ubuntu developer, I need to work with the Eucalyptus package (perhaps merging, fixing bugs or cherry-picking some fixes from upstream). When the Ubuntu/Eucalyptus packaging is free of conflicts, and any remaining complexities between the Ubuntu and upstream Eucalyptus code are clearly documented, my work is more straightforward.
• As a Eucalyptus developer, I am trying to track down a problem, but I am not able to reproduce the problem on the Ubuntu version of Eucalyptus, while I am experiencing it in the upstream build. When the drift between the upstream Eucalyptus code and the downstream Ubuntu code is minimized, my job tracking down problems is much better.
• As an enterprise user of Ubuntu Server Edition, I want to deploy UEC in production. I do so with confidence, knowing that the Eucalyptus packages in 10.04 have been through an extensive stabilization cycle, and contain no known regressions from 9.10.

Assumptions

1. Ubuntu will merge for Lucid exclusively from a Eucalyptus bug-fix-only branch (either 1.6.x or 1.7)
2. Eucalyptus and Ubuntu will continue using Launchpad/Bazaar for collaboration

Design/Implementation

Given that we're both hosting our code in Launchpad bzr, this should be a reasonable task.

• WSDL stubs generation, 487270

  • Need a better way to generate wsdl
    • When are we supposed to auto-generate the stub?
  • Options:
    • package a subset of axis2 to be able to generate the stub during the build process (proposal)
    • generate the stub as part of the upstream source release process

• Upstart scripts for eucalyptus-nc, 438631

  • needs to be done, should be straight-forward
  • get this into an early lucid merge

• eucalyptus.conf documentation, 458211

  • good euca_conf manpage needed (more so than inline conffile comments)

• /etc/eucalyptus/eucalyptus.conf should NOT be a (dpkg-)conffile, 487275

  • scripts change it, OTOH the sysadmin can customize it - a package upgrade should handle that case as well.
  • we should move cloud "state" information away from eucalyptus.conf (NC list)

• Do not use dpkg-statoverrides, 437012

  • Fixing this will likely require a review by Soren

• Purging fixes, 461202

  • metadata ip (via avahi-publish/upstart job?)
  • image store breakage
• euca_rootwrap revamping

  • Talked to KeesCook, he feels that our current euca_rootwrap is "good enough"

    • If anything, he suggested that we might work on enhancing our argument scrubbing, though this may require substantial effort
    • Eventually, teach Eucalyptus to use capabilities
      • But this is currently blocked on caps not extending through exec's
• Upstream Eucalyptus
  • Make smaller commits

  • Create a packaging branch and move the debian/ directory in it, 487282

  • Eucalyptus will not publish Ubuntu packages directly on their website
• Java dependencies
  • Get information on any extra libraries needed for Eucalyptus 1.7 and package them (none expected)
  • Validate eucalyptus 1.7 against Lucid java stack, apply fixes where necessary
• Downstream Ubuntu
  • Merge/release
  • Seemed to work well from bzr commit bug-fix-only revisions
  • Each commit tied to a LP bug for tracking

  • Help upstream to create a packaging branch (MathiasGug), 487282

  • Keep debian/patches for ubuntu-specific non-upstreamable patches, merge all others
  • Set the package branch to the ubuntu-core-dev branch.
  • Most of changes should be pushed to upstream for review
    • Switch to a branch only submission model so that branches can be reviewed/pushed to both upstream and ubuntu branch
  • Package (more or less) weekly bzr snapshots of Eucalyptus
    • Should reserve a couple of minutes on the weekly Canonical/Eucalyptus call to synchronize and ensure that the tree is in acceptable shape for snapshot/merge/upload to Lucid
  • Clean up the debian/patches directory, either sending these upstream, merging directly into the code, or keeping (with a thorough explanation as to their necessity):

    • kirkland@x200:/local/source/eucalyptus/ubuntu/ubuntu/debian/patches$ ls -halF
      total 14M
      14M  01-wsdl-stubs.patch
      495  02-rely-on-libvirt-defaults.patch
      19K  03-DESTDIR.patch
      25K  04-axis2c-1.6.0-rampart-1.3.0.patch
      6.6K 05-axis-alternative-repository.patch
      1.7K 06-symlinked-jars.patch
      1.5K axis2c_home_init.diff
      781  euca_conf-error-output.diff
      21K  var_lib_eucalyptus.diff

Test/Demo Plan

Testing of UEC/Eucalyptus is a separate item. This specification covers merging/packaging of bug-fix only code. Testing of this spec should be covered by general UEC/Eucalyptus testing needed for Lucid.

Unresolved issues

• Move the database away from hsql to something more stable, 487280

  • ie, postgres, mysql, or ideally, a directory server
  • hsql has reliability issues (e.g. db deadlock issues)
  • hsql scalability
  • upstream says: in principle, there's no problem with hsql
    • other components in there (c3p0, hibernate, etc..)
  • Bug filed, no commitment to fix, just wishlist tracking for now

BoF agenda and discussion

This specification was discussed at UDS Dallas in November 2009.