Owner: Marc Deslauriers
Intro
This template is used to track events during a crisis or potential crisis. The goal is not to analyse the entire event, but rather to provide whiteboard-style communications with the key people involved in the reaction plan. If you are not directly involved, do not speculate on pages of this type.
Incident Description
Crisis Response Team
- Marc Deslauriers (security)
- Jamie Strandboge (security, archive)
- Steve Beattie (security)
- tiaz (IS)
- elmo (IS)
Events
All times are in UTC. <Build a chronological list of events as they unfold.>
Oct 18 16:25: http://www.ubuntu.com/usn/usn-1232-1/ published and mdeslaur begins monitoring xorg-server bugs for any regressions
Oct 19 05:11: https://launchpad.net/bugs/877905 filed against the wrong package (xorg, instead of xorg-server). Reported only against Ubuntu 10.04 LTS
- 17:04: mdeslaur notices bug and begins investigating
- 17:19: mdeslaur escalates to manager (jdstrand)
- 17:24: jdstrand informs canonical-support and platform-managers
- 17:32: !regression-alert announced in #ubuntu-devel
- 17:33: skaet acks
- 17:33: inform IS of problem
- 17:37: tiaz responds to 1) block updates from internal machine, 2) rm the files on the mirror master and trigger an update to all other mirrors
- 17:38: incident report started
17:40: elmo informs DealingWithCrisis is incorrect for package of this importance
- 18:10: jdstrand updates bug for notification and provides workaround
- 18:15: tiaz finishes IS tasks (external mirrors have been triggered, but propagation will take longer)
- 18:22: mdeslaur confirms Ubuntu 10.04 LTS is affected
- 18:22: mdeslaur compiles package reverting the patch believed to cause the issue
- 18:29: mdeslaur confirms reverting patch fixes the issue and proceeds to upload packages for 10.04 LTS to the security PPA
- 18:42: sbeattie confirms that Ubuntu 10.10 is not affected
- 19:03: security PPA finishes building i386 and am64, mdeslaur downloads for testing
- 19:12: mdeslaur tests i386 and amd64 successfully, still waiting for other platforms to build
- 20:09: all platforms finish building
- 20:10: packages are unembargoed
- 20:10: mdeslaur notifies tiaz that mirroring may be resumed
- 21:14: publisher finished publishing update and mdeslaur publishes USN-1232-2
Successes
<Identify positive things that happened. What went right in the course of our response?>
- once bug was identified, the response by all teams was decisive and swift
Problems
<Identify problems with the events. What went wrong in the course of our response?>
https://wiki.canonical.com/UbuntuEngineering/DealingWithCrisis documents archive admin procedures which are were inappropriate for this update. This should be updated
- bug filed against the wrong source package which resulted in issue not being caught sooner
- while the xorg-server updates were performed on real hardware for all releases, this testing was performed when using the nvidia glx drivers, which were not affected by the regression.
Recommendations
<Suggest changes to process to minimize problems in the future. These should correspond to the problems identified above.>
DONE: make sure bare-metal installs are available for all supported releases within the team to ensure bug confirmation and timely testing can be performed