Owner: Steve Langasek

Incident Description

When verifying links in the draft announcement mail for Ubuntu 13.04 final beta, it was discovered that the http://www.ubuntu.com/testing/download link had not been updated for the raring cycle. Because this link provides the frontend for load balancing beta downloads across the mirror network, the beta could not be released until this was resolved, due to the adverse impact on Canonical network infrastructure.

Crisis Response Team

• Steve Langasek (slangasek)
• Pete Graner (pgraner)
• Dave Walker (Daviey)
• Darren Spiteri (DarrenS)
• Tanya Edwards

Events

All times are in UTC.

• 2013-04-05 01:15 - Steve Langasek notices the wrong link while reviewing announcement draft
• 2013-04-05 01:45 - Steve escalates to Pete Graner by phone to discuss options
• 2013-04-05 03:04 - Steve discusses with DarrenS (vanguard) on Canonical #is whether it's acceptable to send out the mail with a link directly to releases.ubuntu.com, or if this would unacceptably swamp Canonical's connection
• 2013-04-05 03:33 - Steve and DarrenS concludes that this should wait until London AM when the web team can fix the link
• 2013-04-05 04:35 - Steve emails Dave Walker and asks him to coordinate with the web team to let the announcement out once the website is fixed
• 2013-04-05 04:37 - Pete attempts to notify Peter Mahnke by email that the website changes are needed ASAP in the morning (UTC)
• 2013-04-05 04:49 - Steve sends the announcement mail with the link intact to ubuntu-announce, to be let out of the moderation queue in the morning once the website is updated
• 2013-04-05 09:00 - the announcement mail is moderated by parties unknown, before the web link has been updated
• 2013-04-05 16:30 - Tanya Edwards confirms that the website redirect has been updated.

Successes

Problems

• The MilestoneProcess was not followed completely for the opt-in alpha milestones, resulting in the web redirect not being updated prior to beta.

• The BetaProcess wiki page was not updated to include a check of the download redirect prior to beta.

• The announcement mail draft was not started until after London EOD on the release day; as a result, any action that needed to be taken by the Canonical web team in response was delayed a day.
• The contact used for the web team was on vacation, so the web team did not receive the notification in a timely manner.

Recommendations

• Fix the BetaProcess to address this specific failure

• Ensure that email and release notes drafts are handled sooner
• Ensure that the milestone checklist for opt-in milestones is correct and adhered to
• Establish/identify an appropriate role contact address for the web team