2012-07-19-oneiric-kernel-regression-kills-EC2-instances

Differences between revisions 22 and 23
⇤ ← Revision 22 as of 2012-07-19 17:46:27 →
Size: 1667
Editor: brad-figg
Comment:
← Revision 23 as of 2012-07-19 18:06:33 → ⇥
Size: 1793
Editor: 74-95-45-185-Oregon
Comment:
Deletions are marked like this. Additions are marked like this.
Line 18: Line 18:
 * Carlos de Avillez (hggdh)
Line 30: Line 31:
 * 2012-07-19 17:30 - QA identified that the EC2 i386 test was testing the wrong architecture.

Contents

  1. Incident Description
  2. Crisis Response Team
  3. Events
  4. Successes
  5. Problems
  6. Recommendations

Owner: ?

Incident Description

The linux kernel 3.0.0.-23.38 update appears to have exposed a regression that kills off EC2 instances. Any user who updates has a dead EC2 instance. instance-store users will lose all data. EBS users will have a very painful recovery path.

This issue is unique to Oneiric kernels.

Crisis Response Team

  • Ben Howard (utlemming)
  • Brad Figg (bjf)
  • Stefan Bader (smb)
  • Kate Stewart (skaet)
  • Adam Conrad (infinity)
  • Antonio Rosales (arosales)
  • Carlos de Avillez (hggdh)

Events

All times are in UTC.

  • 2012-07-19 16:24 - https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/1026690 opened

  • 2012-07-19 16:25 - on #ubuntu-release Ben Howard comments on critical kernel regression and asks how to back out bad SRU.
  • 2012-07-19 16:31 - Kate Stewart points Ben to Brad Figg to work out best path forward.
  • 2012-07-19 16:33 - in #ubuntu-kernel, Scope of impact starts getting discussed between Ben, Brad, Stefan.
  • 2012-07-19 16:50 - Kate starts off incident report.
  • 2012-07-19 16:57 - Adam Removing packages: linux-image-3.0.0-23-virtual 3.0.0-23.38 in oneiric i386. Comment: Broken SRU; blows up EC2 instances
  • 2012-07-19 16:58 - Kernel team has verified that this issue is unique to Oneiric. The bad commit exists in other kernels however, the commit that fixes the issue also exists in the other kernels.
  • 2012-07-19 17:30 - QA identified that the EC2 i386 test was testing the wrong architecture.
  • 2012-07-19 17:44 - Kernel team believes it has identified the problem commit.
  • 2012-07-19 17:44 - Kernel team building test kernels and preparing for a quick upload.

Successes

Problems

Recommendations

IncidentReports/2012-07-19-oneiric-kernel-regression-kills-EC2-instances (last edited 2012-07-21 06:37:13 by adconrad)