CDBS 0.4.62+nmu1ubuntu6 was uploaded to address bug #535650. Unfortunately, the X_Gettext_Domain entry appeared above the [Desktop Entry] line, causing a number of parse failures reading result .desktop files. This was built and installed and affected builds of empathy, evolution, gedit, gnome-disk-utility, gnome-panel, nautilus, rhythmbox, and totem. The parse failures for gnome-panel and nautilus broke the autostart, leaving users who had updated with a nonfunctional desktop on login. Bug #542343 was filed by a user experiencing this issue.
Crisis Response Team
- Jorge Castro
- Emmet Hikory
- Chris Coulson
- Ken Vandine
- William Grant (Community)
- Mike Basinger (Community)
- Steve Kowalik
- Sarah Hobbs (Community)
All times are in UTC.
- CDBS causing misbuilds is published to archive
<Unknown to Emmet Hikory>
Mike Basinger's panel and nautilus don't work when he logs in. (Tweet.)
- Laserjock pm's jcastro on IRC asking if someone is working on the GNOME desktop failures
- Jorge updates and upgrades, finds desktop broken, couldn't find anyone awake on -release or -devel. Decides to call Rick, who isn't available, so calls Robbie at 02:05. Soon after Robbie updates the platform twitter account to warn users.
- Jorge asks Laserjock and kklimoda to help him find the cause of the problem; Jorge pings Ken Vandine on IM to see if he can help.
Jorge sees thread on forums with people having the problems, asks Mike to immediately make a sticky to warn users. Thread.
- Ken Vandine tests the CD image in a test and confirms that it indeed was an update that caused the problem.
Chris Coulson finds the problem and triages the bug accordingly.
- Jorge needs to leave, hands off to Emmet
- Emmet Hikory calls for help fixing the bug (notifications from #ubuntu-devel backscroll, #ubuntu-desktop backscroll, jcastro@/query, anmar@#distro)
- William Grant contacts Steve Kowalik out-of-band
- Steve Kowalik begins fixing issue
- Steve Kowalik uploads reverted CDBS
- William Grant publishes list of affected packages
- Rick Spencer assigns #542343 to Canonical Desktop Team
- Emmet Hikory updates CDBS task of #542343 to Fix Released
- William Grant provides notice that fixed CDBS is available for new builds
- Steve Kowalik uploads rebuilds for all affected packages
- Sarah Hobbs completes adjustment of buildd queue priorities to be able to finish i386/amd64 within the publisher run
- William Grant confirms gnome-panel and nautilus are in publication for i386/amd64
- William Grant confirms publication of gnome-panel and nautilus
- Emmet Hikory updates gnome-panel task of #542343 to Fix Committed and notes that the fix may be delayed for some users while mirrors push
- William Grant comments to #542343 that all affected packages have been rebuilt except on sparc/armel
- Sebastien Bacher updates gnome-panel task of #542343 to Fix Released
- Jorge escalated rather quickly as soon as it was known that the desktop was broken and that it would take multiple people to fix.
- Mike and Wgrant did a good job of keeping the affected forum users notified of progress throughout the incident.
- Rounding up core devs inbetween the west coasters and the people in Asia turned out to be difficult.
- Escalation procedures for community members - it was dumb luck that Jorge happened to be around, would they have known the process for escalating if no one responded for hours?