2011-10-19-xorg-server-security-update-breaks-glx

Contents

  1. Intro
  2. Incident Description
  3. Crisis Response Team
  4. Events
  5. Successes
  6. Problems
  7. Recommendations

Owner: Marc Deslauriers

Intro

This template is used to track events during a crisis or potential crisis. The goal is not to analyse the entire event, but rather to provide whiteboard-style communications with the key people involved in the reaction plan. If you are not directly involved, do not speculate on pages of this type.

Incident Description

Crisis Response Team

  • Marc Deslauriers (security)
  • Jamie Strandboge (security, archive)
  • Steve Beattie (security)
  • tiaz (IS)
  • elmo (IS)

Events

All times are in UTC. <Build a chronological list of events as they unfold.>

  • Oct 18 16:25: http://www.ubuntu.com/usn/usn-1232-1/ published and mdeslaur begins monitoring xorg-server bugs for any regressions

  • Oct 19 05:11: https://launchpad.net/bugs/877905 filed against the wrong package (xorg, instead of xorg-server). Reported only against Ubuntu 10.04 LTS

  • 17:04: mdeslaur notices bug and begins investigating
  • 17:19: mdeslaur escalates to manager (jdstrand)
  • 17:24: jdstrand informs canonical-support and platform-managers
  • 17:32: !regression-alert announced in #ubuntu-devel
  • 17:33: skaet acks
  • 17:33: inform IS of problem
  • 17:37: tiaz responds to 1) block updates from internal machine, 2) rm the files on the mirror master and trigger an update to all other mirrors
  • 17:38: incident report started

  • 17:40: elmo informs DealingWithCrisis is incorrect for package of this importance

  • 18:10: jdstrand updates bug for notification and provides workaround
  • 18:15: tiaz finishes IS tasks (external mirrors have been triggered, but propagation will take longer)
  • 18:22: mdeslaur confirms Ubuntu 10.04 LTS is affected
  • 18:22: mdeslaur compiles package reverting the patch believed to cause the issue
  • 18:29: mdeslaur confirms reverting patch fixes the issue and proceeds to upload packages for 10.04 LTS to the security PPA
  • 18:42: sbeattie confirms that Ubuntu 10.10 is not affected
  • 19:03: security PPA finishes building i386 and am64, mdeslaur downloads for testing
  • 19:12: mdeslaur tests i386 and amd64 successfully, still waiting for other platforms to build
  • 20:09: all platforms finish building
  • 20:10: packages are unembargoed
  • 20:10: mdeslaur notifies tiaz that mirroring may be resumed
  • 21:14: publisher finished publishing update and mdeslaur publishes USN-1232-2

Successes

<Identify positive things that happened. What went right in the course of our response?>

  • once bug was identified, the response by all teams was decisive and swift

Problems

<Identify problems with the events. What went wrong in the course of our response?>

  • https://wiki.canonical.com/UbuntuEngineering/DealingWithCrisis documents archive admin procedures which are were inappropriate for this update. This should be updated

  • bug filed against the wrong source package which resulted in issue not being caught sooner
  • while the xorg-server updates were performed on real hardware for all releases, this testing was performed when using the nvidia glx drivers, which were not affected by the regression.

Recommendations

<Suggest changes to process to minimize problems in the future. These should correspond to the problems identified above.>

  • DONE: make sure bare-metal installs are available for all supported releases within the team to ensure bug confirmation and timely testing can be performed

IncidentReports/2011-10-19-xorg-server-security-update-breaks-glx (last edited 2011-10-24 14:59:52 by jdstrand)