2010-03-19-CDBS-broke-gnome-panel-autostart

Incident Description

CDBS 0.4.62+nmu1ubuntu6 was uploaded to address bug #535650. Unfortunately, the X_Gettext_Domain entry appeared above the [Desktop Entry] line, causing a number of parse failures reading result .desktop files. This was built and installed and affected builds of empathy, evolution, gedit, gnome-disk-utility, gnome-panel, nautilus, rhythmbox, and totem. The parse failures for gnome-panel and nautilus broke the autostart, leaving users who had updated with a nonfunctional desktop on login. Bug #542343 was filed by a user experiencing this issue.

Crisis Response Team

  • Jorge Castro
  • Emmet Hikory
  • Chris Coulson
  • Ken Vandine
  • William Grant (Community)
  • Mike Basinger (Community)
  • Steve Kowalik
  • Sarah Hobbs (Community)

Events

All times are in UTC.

2010-03-19

20:06
CDBS causing misbuilds is published to archive

<Unknown to Emmet Hikory>

2010-03-20

01:30-ish

Mike Basinger's panel and nautilus don't work when he logs in. (Tweet.)

01:48
Laserjock pm's jcastro on IRC asking if someone is working on the GNOME desktop failures
02:00
Jorge updates and upgrades, finds desktop broken, couldn't find anyone awake on -release or -devel. Decides to call Rick, who isn't available, so calls Robbie at 02:05. Soon after Robbie updates the platform twitter account to warn users.
02:05
Jorge asks Laserjock and kklimoda to help him find the cause of the problem; Jorge pings Ken Vandine on IM to see if he can help.
02:10ish

Jorge sees thread on forums with people having the problems, asks Mike to immediately make a sticky to warn users. Thread.

02:20ish
Ken Vandine tests the CD image in a test and confirms that it indeed was an update that caused the problem.
02:25

Chris Coulson finds the problem and triages the bug accordingly.

03:17
Jorge needs to leave, hands off to Emmet
04:33
Emmet Hikory calls for help fixing the bug (notifications from #ubuntu-devel backscroll, #ubuntu-desktop backscroll, jcastro@/query, anmar@#distro)
04:49
William Grant contacts Steve Kowalik out-of-band
04:55
Steve Kowalik begins fixing issue
05:00
Steve Kowalik uploads reverted CDBS
05:21
William Grant publishes list of affected packages
05:34
Rick Spencer assigns #542343 to Canonical Desktop Team
05:58
Emmet Hikory updates CDBS task of #542343 to Fix Released
06:58
William Grant provides notice that fixed CDBS is available for new builds
07:28
Steve Kowalik uploads rebuilds for all affected packages
07:38
Sarah Hobbs completes adjustment of buildd queue priorities to be able to finish i386/amd64 within the publisher run
08:00
William Grant confirms gnome-panel and nautilus are in publication for i386/amd64
08:58
William Grant confirms publication of gnome-panel and nautilus
09:06
Emmet Hikory updates gnome-panel task of #542343 to Fix Committed and notes that the fix may be delayed for some users while mirrors push
09:47
William Grant comments to #542343 that all affected packages have been rebuilt except on sparc/armel
11:11
Sebastien Bacher updates gnome-panel task of #542343 to Fix Released

Successes

  • Jorge escalated rather quickly as soon as it was known that the desktop was broken and that it would take multiple people to fix.
  • Mike and Wgrant did a good job of keeping the affected forum users notified of progress throughout the incident.

Problems

  • Rounding up core devs inbetween the west coasters and the people in Asia turned out to be difficult.

Recommendations

  • Escalation procedures for community members - it was dumb luck that Jorge happened to be around, would they have known the process for escalating if no one responded for hours?

IncidentReports/2010-03-19-CDBS-broke-gnome-panel-autostart (last edited 2010-04-28 09:14:27 by robbie.w)