Ubuntu Wiki

A series of errors during the update process seriously affected some Ubuntu 6.06 LTS users who installed updates between the evening of Monday 21 August (UTC) and morning of Tuesday 22 August (UTC). For affected users, this caused the graphical interface of Ubuntu to fail.

This report results from a technical and procedural review of the actions that led to the incident, and identifies corrective actions that have been put in place to avoid similar incidents in the future.

Problems Identified and Corrective Actions

1. A patch which was not appropriate for a stable release update was proposed for inclusion in Dapper by an Ubuntu developer, this proposal was implemented by another developer, and a third developer approved the update for publication. This indicates that an understanding of the policy and process for stable release updates is not sufficiently pervasive within the development team.

• Corrective action: Create developer-oriented documentation for the update policy and process, publish in the Ubuntu wiki, announce to Ubuntu developers, and add to list of required reading for new core developers.
• Completion date: Friday, 15 September 2006
• Owner: Matt Zimmerman

2. A hardware-specific regression in the patch was not detected during the testing which took place prior to its release, despite a substantial number of systems being affected. This indicates that more extensive testing is necessary where an update has the potential for hardware-specific regressions.

• Corrective action: Amend and document changes to formal testing process for stable release updates including the extent of required regression testing.
• Completion date: Friday, 15 September 2006
• Owner: Matt Zimmerman

3. The severity of the regression was not communicated quickly and broadly enough within the project to allow for all possible speed in applying corrective measures.

• Corrective action: Create documentation for escalation criteria and procedures, and add to list of required reading for new core developers
• Completion date: Friday, 15 September 2006
• Owner: Matt Zimmerman

Incident Timeline

2006-08-01

• Bug 54880 is filed by an Ubuntu developer to report that Ubuntu does not properly support the graphics hardware on certain SPARC and OpenPower platforms. The bug report is accompanied by a patch to add support for these configurations, and suggests that it would be beneficial to backport the patch to Ubuntu 6.06 LTS.

2006-08-07

• Another Ubuntu developer prepares an update for the development branch of Ubuntu (Edgy) which incorporates a related, but more comprehensive patch from Fedora.

2006-08-10

• The aforementioned update is released to Edgy users.

2006-08-10+

• Acting on the suggestion from in the bug report, the same developer prepares an update for Ubuntu 6.06 LTS using the patch provided in the bug report, which originated upstream at X.org (that is, it was a different patch than the more intrusive one used in Edgy). This update was tested on the SPARC platform to confirm the fix, and regression tested on additional systems. The developer then adds the update to the queue for review and approval by the Ubuntu release team.

2006-08-21

• The developer inquires with a member of the Ubuntu release team about the status of the update. Having just returned from a holiday, the release team member has not examined the update queue for some time. He examines the update and reviews the developer's description of the testing which has been performed. He approves the update and it is made available to users. Later in the evening, the developer installs the update from the update server and again confirms no regression in his environment.

2006-08-22

• By morning, the developer has received several bug reports (on the order of 5-10) indicating a regression in the update, bug #57153. He investigates, identifies the change which caused the problem, and prepares a new update which reverts that change. He discusses this new update with the release team member who approved the original update. He approves the new update in principle and authorizes another member of the Ubuntu archive administration team to review and process the update, with instructions to confirm that the update merely reverts the earlier changes. He does so, and the problematic update is replaced with the fixed version.