<> === Incident Description === CDBS 0.4.62+nmu1ubuntu6 was uploaded to address bug #535650. Unfortunately, the X_Gettext_Domain entry appeared above the [Desktop Entry] line, causing a number of parse failures reading result .desktop files. This was built and installed and affected builds of empathy, evolution, gedit, gnome-disk-utility, gnome-panel, nautilus, rhythmbox, and totem. The parse failures for gnome-panel and nautilus broke the autostart, leaving users who had updated with a nonfunctional desktop on login. Bug #542343 was filed by a user experiencing this issue. === Crisis Response Team === * Jorge Castro * Emmet Hikory * Chris Coulson * Ken Vandine * William Grant (Community) * Mike Basinger (Community) * Steve Kowalik * Sarah Hobbs (Community) === Events === All times are in UTC. ==== 2010-03-19 ==== 20:06:: CDBS causing misbuilds is published to archive ==== 2010-03-20 ==== 01:30-ish:: Mike Basinger's panel and nautilus don't work when he logs in. ([[http://twitter.com/Technoviking/status/10751134966|Tweet]].) 01:48:: Laserjock pm's jcastro on IRC asking if someone is working on the GNOME desktop failures 02:00:: Jorge updates and upgrades, finds desktop broken, couldn't find anyone awake on -release or -devel. Decides to call Rick, who isn't available, so calls Robbie at 02:05. Soon after Robbie updates the platform twitter account to warn users. 02:05:: Jorge asks Laserjock and kklimoda to help him find the cause of the problem; Jorge pings Ken Vandine on IM to see if he can help. 02:10ish:: Jorge sees thread on forums with people having the problems, asks Mike to immediately make a sticky to warn users. [[http://ubuntuforums.org/showthread.php?t=1434160|Thread]]. 02:20ish:: Ken Vandine tests the CD image in a test and confirms that it indeed was an update that caused the problem. 02:25:: Chris Coulson finds the problem and triages [[https://launchpad.net/bugs/542343|the bug]] accordingly. 03:17:: Jorge needs to leave, hands off to Emmet 04:33:: Emmet Hikory calls for help fixing the bug (notifications from #ubuntu-devel backscroll, #ubuntu-desktop backscroll, jcastro@/query, anmar@#distro) 04:49:: William Grant contacts Steve Kowalik out-of-band 04:55:: Steve Kowalik begins fixing issue 05:00:: Steve Kowalik uploads reverted CDBS 05:21:: William Grant publishes list of affected packages 05:34:: Rick Spencer assigns #542343 to Canonical Desktop Team 05:58:: Emmet Hikory updates CDBS task of #542343 to Fix Released 06:58:: William Grant provides notice that fixed CDBS is available for new builds 07:28:: Steve Kowalik uploads rebuilds for all affected packages 07:38:: Sarah Hobbs completes adjustment of buildd queue priorities to be able to finish i386/amd64 within the publisher run 08:00:: William Grant confirms gnome-panel and nautilus are in publication for i386/amd64 08:58:: William Grant confirms publication of gnome-panel and nautilus 09:06:: Emmet Hikory updates gnome-panel task of #542343 to Fix Committed and notes that the fix may be delayed for some users while mirrors push 09:47:: William Grant comments to #542343 that all affected packages have been rebuilt except on sparc/armel 11:11:: Sebastien Bacher updates gnome-panel task of #542343 to Fix Released === Successes === * Jorge escalated rather quickly as soon as it was known that the desktop was broken and that it would take multiple people to fix. * Mike and Wgrant did a good job of keeping the affected forum users notified of progress throughout the incident. === Problems === * Rounding up core devs inbetween the west coasters and the people in Asia turned out to be difficult. === Recommendations === * Escalation procedures for community members - it was dumb luck that Jorge happened to be around, would they have known the process for escalating if no one responded for hours?