Contents
Incident Description
CDBS 0.4.62+nmu1ubuntu6 was uploaded to address bug #535650. Unfortunately, the X_Gettext_Domain entry appeared above the [Desktop Entry] line, causing a number of parse failures reading result .desktop files. This was built and installed and affected builds of empathy, evolution, gedit, gnome-disk-utility, gnome-panel, nautilus, rhythmbox, and totem. The parse failures for gnome-panel and nautilus broke the autostart, leaving users who had updated with a nonfunctional desktop on login. Bug #542343 was filed by a user experiencing this issue.
Crisis Response Team
- Jorge Castro
- Emmet Hikory
- Chris Coulson
- Ken Vandine
- William Grant (Community)
- Mike Basinger (Community)
- Steve Kowalik
- Sarah Hobbs (Community)
Events
All times are in UTC.
2010-03-19
- 20:06
- CDBS causing misbuilds is published to archive
<Unknown to Emmet Hikory>
2010-03-20
- 01:30-ish
Mike Basinger's panel and nautilus don't work when he logs in. (Tweet.)
- 01:48
- Laserjock pm's jcastro on IRC asking if someone is working on the GNOME desktop failures
- 02:00
- Jorge updates and upgrades, finds desktop broken, couldn't find anyone awake on -release or -devel. Decides to call Rick, who isn't available, so calls Robbie at 02:05. Soon after Robbie updates the platform twitter account to warn users.
- 02:05
- Jorge asks Laserjock and kklimoda to help him find the cause of the problem; Jorge pings Ken Vandine on IM to see if he can help.
- 02:10ish
Jorge sees thread on forums with people having the problems, asks Mike to immediately make a sticky to warn users. Thread.
- 02:20ish
- Ken Vandine tests the CD image in a test and confirms that it indeed was an update that caused the problem.
- 02:25
Chris Coulson finds the problem and triages the bug accordingly.
- 03:17
- Jorge needs to leave, hands off to Emmet
- 04:33
- Emmet Hikory calls for help fixing the bug (notifications from #ubuntu-devel backscroll, #ubuntu-desktop backscroll, jcastro@/query, anmar@#distro)
- 04:49
- William Grant contacts Steve Kowalik out-of-band
- 04:55
- Steve Kowalik begins fixing issue
- 05:00
- Steve Kowalik uploads reverted CDBS
- 05:21
- William Grant publishes list of affected packages
- 05:34
- Rick Spencer assigns #542343 to Canonical Desktop Team
- 05:58
- Emmet Hikory updates CDBS task of #542343 to Fix Released
- 06:58
- William Grant provides notice that fixed CDBS is available for new builds
- 07:28
- Steve Kowalik uploads rebuilds for all affected packages
- 07:38
- Sarah Hobbs completes adjustment of buildd queue priorities to be able to finish i386/amd64 within the publisher run
- 08:00
- William Grant confirms gnome-panel and nautilus are in publication for i386/amd64
- 08:58
- William Grant confirms publication of gnome-panel and nautilus
- 09:06
- Emmet Hikory updates gnome-panel task of #542343 to Fix Committed and notes that the fix may be delayed for some users while mirrors push
- 09:47
- William Grant comments to #542343 that all affected packages have been rebuilt except on sparc/armel
- 11:11
- Sebastien Bacher updates gnome-panel task of #542343 to Fix Released
Successes
- Jorge escalated rather quickly as soon as it was known that the desktop was broken and that it would take multiple people to fix.
- Mike and Wgrant did a good job of keeping the affected forum users notified of progress throughout the incident.
Problems
- Rounding up core devs inbetween the west coasters and the people in Asia turned out to be difficult.
Recommendations
- Escalation procedures for community members - it was dumb luck that Jorge happened to be around, would they have known the process for escalating if no one responded for hours?