KarmicEucalyptusRetrospective

Summary

During UDS-L in Dallas we took 2 hours to do a retrospective of how Eucalyptus packaging went during the 9.10 cycle. The moderator(s) recommended several actions to improve the process going forward, this specification tracks those recommendations to make sure they are implemented during the Lucid development cycle.

Release Note

Not applicable, this is a process improvement project which does not directly impact any product

Rationale

During the karmic development cycle, several issues led to a very difficult situation on the Eucalyptus packaging project. This led to a retrospective analysis between the Eucalyptus and Ubuntu Server team during the UDS-L in Dallas, which in turn resulted in several recommendations from the session moderator. This spec ensures that those recommendations are taken into account for the Lucid development cycle, by tracking the implementation of all those actions.

User stories

As an Ubuntu Server team member (or Eucalyptus upstream member), I want to undertake specific actions to make sure future cycles will go smoothly. I implement those recommendations, and it results in a smoother development cycle for Lucid.

As an Ubuntu platform manager, I want to make sure that we learn from our past difficulties with 9.10 and that everything is done to prevent such issues in the future. I asked for a retrospective and recommendations, we implement them, and I feel confident that we'll not repeat the past mistakes.

Assumptions

The Eucalyptus and Ubuntu engineering teams should be willing to accept the recommendations from this Retrospective analysis and adapt their internal workflow accordingly.

Design

What went well

  • Actual release occurred
  • Product actually works
  • Kept good relationships in spite of pressures
  • Face-to-face sprints (~6)
  • Tool set and infrastructure (Launchpad, Bazaar)
  • Upstart integration

What went wrong

  • Lack of bug triage
  • Lack of engineers to do dev, testing, QA
  • Engineer burnout
  • Lack of schedule slip escalation to management
  • Unrealistic schedule optimism
  • Even simple bugs not addressed
  • Lack of overall view of status of project
  • Lack of HW in the project team to do testing
  • Level of build quality from Eucalyptus was not sufficient for integration into Ubuntu
  • Lack of clear communication of level of quality of Eucalyptus build
  • Scope of overall project was not well understood
  • Scope of project was not defined and changes to scope were not communicated to project engineers
  • Two communication channels were set up, but project engineers were not included in one
  • Key Canonical engineer had information that was not shared with others
  • Amount of refactoring by Eucalyptus contributed to lower level of quality of builds
  • Deadline for code drop put extra pressure on Eucalyptus to deliver the build with the quality level that it had on that date

Recommendations

  • Management communication
    • Management must communicate the scope of next project to fellow managers and engineers in a commonly accessible area
    • Engineering teams should get information from their own managers instead of getting it from the partner's executives
  • Open development
    • Must have an open development environment instead of a closed one between Eucalyptus and Canonical
    • We acknowledge Eucalyptus team's will to do closed development for a period of time, but it should be as short as possible
    • During "closed development" Eucalyptus should always provide code to Canonical Server team as a private branch
    • During "closed development" Canonical Server team should regularly package the latest in private branches and PPAs
    • Code must be released to public by Alpha2 (LTS) or Alpha3 (other cycles)
  • Resource planning, QA
    • Provide sufficient development resources, including development, QA and HW
    • Provide sufficient QA plan and do it early
    • Affect a primary and a secondary engineer to all tasks, involve at least half of the team in the project
    • Provide access for those engineers to sufficient hardware for:
      • Development (basic topology)
      • Bug reproduction (all supported topologies)
    • Keep eucalyptus and euca2ools bug lists under control by regular triaging
  • Communication between teams
    • Encourage face-to-face sprints
    • Understand partner's preferred form of interrupt-driven communication and adjust accordingly. Eucalyptus engineers did not like using IRC.
  • Follow-up: monthly reviews during a Eucalyptus tech call (first call of the month):
    • Review alignment with these recommendations
    • Overall assessment of status of project
    • Plan any necessary additional sprint

Implementation

Policy changes (not work items):

  • Communicate minutes of the cloud strategy call to the Canonical server team (cloud strategy call minutes writer)
  • Assign a secondary engineer to all Eucalyptus development projects (server team mgr.)
  • Invite one or more Eucalyptus representative to the Lucid Developer Sprint (server team mgr.)

Work items: see blueprint whiteboard.

Remarks:

Test/Demo Plan

Compliance with recommendations will be evaluated monthly.

Unresolved issues

None.

BoF agenda and discussion

UDS discussion agenda

We'd like to do a look at how the Eucalyptus packaging and collaboration worked in the Karmic cycle with an eye to highlighting what worked and observing what didn't so we can adapt and further improve in the Lucid cycle. To do that, we should take a look at:

  • A Karmic timeline
  • What worked
  • What didn't
  • Actions for Lucid

If you were a part of the Karmic UEC and Eucalyptus work, we'd like to hear from you!

CategorySpec

KarmicEucalyptusRetrospective (last edited 2009-12-03 13:13:20 by lns-bzn-48f-81-56-218-246)