<> Owner: Marc Deslauriers === Intro === This template is used to track events during a crisis or potential crisis. The goal is not to analyse the entire event, but rather to provide whiteboard-style communications with the key people involved in the reaction plan. If you are not directly involved, '''do not''' speculate on pages of this type. === Incident Description === === Crisis Response Team === * Marc Deslauriers (security) * Jamie Strandboge (security, archive) * Steve Beattie (security) * tiaz (IS) * elmo (IS) === Events === All times are in UTC. * Oct 18 16:25: http://www.ubuntu.com/usn/usn-1232-1/ published and mdeslaur begins monitoring xorg-server bugs for any regressions * Oct 19 05:11: https://launchpad.net/bugs/877905 filed against the wrong package (xorg, instead of xorg-server). Reported only against Ubuntu 10.04 LTS * 17:04: mdeslaur notices bug and begins investigating * 17:19: mdeslaur escalates to manager (jdstrand) * 17:24: jdstrand informs canonical-support and platform-managers * 17:32: !regression-alert announced in #ubuntu-devel * 17:33: skaet acks * 17:33: inform IS of problem * 17:37: tiaz responds to 1) block updates from internal machine, 2) rm the files on the mirror master and trigger an update to all other mirrors * 17:38: incident report started * 17:40: elmo informs !DealingWithCrisis is incorrect for package of this importance * 18:10: jdstrand updates bug for notification and provides workaround * 18:15: tiaz finishes IS tasks (external mirrors have been triggered, but propagation will take longer) * 18:22: mdeslaur confirms Ubuntu 10.04 LTS is affected * 18:22: mdeslaur compiles package reverting the patch believed to cause the issue * 18:29: mdeslaur confirms reverting patch fixes the issue and proceeds to upload packages for 10.04 LTS to the security PPA * 18:42: sbeattie confirms that Ubuntu 10.10 is not affected * 19:03: security PPA finishes building i386 and am64, mdeslaur downloads for testing * 19:12: mdeslaur tests i386 and amd64 successfully, still waiting for other platforms to build * 20:09: all platforms finish building * 20:10: packages are unembargoed * 20:10: mdeslaur notifies tiaz that mirroring may be resumed * 21:14: publisher finished publishing update and mdeslaur publishes USN-1232-2 === Successes === * once bug was identified, the response by all teams was decisive and swift === Problems === * https://wiki.canonical.com/UbuntuEngineering/DealingWithCrisis documents archive admin procedures which are were inappropriate for this update. This should be updated * bug filed against the wrong source package which resulted in issue not being caught sooner * while the xorg-server updates were performed on real hardware for all releases, this testing was performed when using the nvidia glx drivers, which were not affected by the regression. === Recommendations === * '''DONE''': make sure bare-metal installs are available for all supported releases within the team to ensure bug confirmation and timely testing can be performed