2011-10-19-xorg-server-security-update-breaks-glx

Differences between revisions 12 and 13
⇤ ← Revision 12 as of 2011-10-19 19:04:12 →
Size: 2910
Editor: modemcable109
Comment:
← Revision 13 as of 2011-10-19 19:10:44 → ⇥
Size: 2910
Editor: jdstrand
Comment:
Deletions are marked like this. Additions are marked like this.
Line 27: Line 27:
 * 12:32: !regression-alert announced in #ubuntu-devel  * 17:32: !regression-alert announced in #ubuntu-devel

Contents

  1. Intro
  2. Incident Description
  3. Crisis Response Team
  4. Events
  5. Successes
  6. Problems
  7. Recommendations

Owner: Marc Deslauriers

Intro

This template is used to track events during a crisis or potential crisis. The goal is not to analyse the entire event, but rather to provide whiteboard-style communications with the key people involved in the reaction plan. If you are not directly involved, do not speculate on pages of this type.

Incident Description

Crisis Response Team

  • Marc Deslauriers (security)
  • Jamie Strandboge (security, archive)
  • Steve Beattie (security)
  • tiaz (IS)
  • elmo (IS)

Events

All times are in UTC. <Build a chronological list of events as they unfold.>

  • Oct 18 16:25: http://www.ubuntu.com/usn/usn-1232-1/ published and mdeslaur begins monitoring xorg-server bugs for any regressions

  • Oct 19 05:11: https://launchpad.net/bugs/877905 filed against the wrong package (xorg, instead of xorg-server). Reported only against Ubuntu 10.04 LTS

  • 17:04: mdeslaur notices bug and begins investigating
  • 17:19: mdeslaur escalates to manager (jdstrand)
  • 17:24: jdstrand informs canonical-support and platform-managers
  • 17:32: !regression-alert announced in #ubuntu-devel
  • 17:33: skaet acks
  • 17:33: inform IS of problem
  • 17:37: tiaz responds to 1) block updates from internal machine, 2) rm the files on the mirror master and trigger an update to all other mirrors
  • 17:38: incident report started

  • 17:40: elmo informs DealingWithCrisis is incorrect for package of this importance

  • 18:10: jdstrand updates bug for notification and provides workaround
  • 18:15: tiaz finishes IS tasks (external mirrors have been triggered, but propagation will take longer)
  • 18:22: mdeslaur confirms Ubuntu 10.04 LTS is affected
  • 18:22: mdeslaur compiles package reverting the patch believed to cause the issue
  • 18:29: mdeslaur confirms reverting patch fixes the issue and proceeds to upload packages for 10.04 LTS to the security PPA
  • 18:42: sbeattie confirms that Ubuntu 10.10 is not affected
  • 19:03: security PPA finishes building i386 and am64, mdeslaur downloads for testing

Successes

<Identify positive things that happened. What went right in the course of our response?>

Problems

<Identify problems with the events. What went wrong in the course of our response?>

  • https://wiki.canonical.com/UbuntuEngineering/DealingWithCrisis documents archive admin procedures which are were inappropriate for this update. This should be updated

  • bug filed against the wrong source package which resulted in issue not being caught sooner
  • while the xorg-server updates were performed on real hardware for all releases, this was on an nvidia chip which used nvidia-glx, which was not affected

Recommendations

<Suggest changes to process to minimize problems in the future. These should correspond to the problems identified above.>

IncidentReports/2011-10-19-xorg-server-security-update-breaks-glx (last edited 2011-10-24 14:59:52 by jdstrand)