2011-06-22-Dist-upgrade-pulls-firefox

Owner: Micah Gersten

Intro

This template is used to track events during a crisis or potential crisis. The goal is not to analyse the entire event, but rather to provide whiteboard-style communications with the key people involved in the reaction plan. If you are not directly involved, do not speculate on pages of this type.

Incident Description

Firefox 5 was the security update for Firefox 4. Generally, language packs are only updated once they have been tested. Sometimes there can be a delay in releasing updated language packs for certain languages if testers are scarce. With the new rapid release cycle we would have needed to update the language packs with each Firefox release. The desktop team planned and implemented the decoupling of Firefox updates from the language packs, with sign off from the security and translations teams. In order to provide users with a translated Firefox, we added a recommends in the language pack for the new Firefox locale binary. This Firefox locale binary then had a Depends on Firefox as it needed Firefox to work. However, this had the unintended side effect of pulling in Firefox on systems that didn't have it (800857).

Being such a large change, we put out a call for testing. We didn't have a lot of lead time on the final binaries due to a security fix that upstream took at the last minute. The transition was tracked in (798484).

Crisis Response Team

  • Micah Gersten
  • Rick Spencer
  • Chris Coulson
  • Marc Deslauriers
  • Kees Cook
  • Clint Byrum

Events

All times are in UTC.

  • 2011-06-21 21:00 - Firefox 5 released w/updated language packs to natty
  • 2011-06-22 10:51 - USN-1157-1 announces Firefox 5 security update
  • 2011-06-22 19:16 - (800857) is filed (not noticed until 22:30)

  • 2011-06-22 20:15 - Chris Coulson and debfx start discussing why the new locales have a dependency on Firefox in #ubuntu-mozillateam
  • 2011-06-22 22:13 - Micah Gersten notes in #ubuntu-mozillateam that the dependency chain pulls in Firefox on systems that don't have it by default (like Kubuntu) and we need to fix this
  • 2011-06-22 22:31 - Micah Gersten finds (800857) which tells us that servers could possible be impacted which possibly raises the possible severity of the issue (and started this process)

  • 2011-06-22 22:39 - Micah Gersten attempts to contact Jamie Strandboge (line manager)
  • 2011-06-22 22:56 - Micah Gersten escalates to Rick Spencer, a brief phone call happens to describe the situation
  • 2011-06-22 23:06 - Micah Gersten and Rick Spencer try to ascertain if a default server install is affected (in #ubuntu-server)
  • 2011-06-22 23:13 - Marc Deslauriers mentions that unattended upgrades might be affected (which would raise the severity to Critical)
  • 2011-06-22 23:14 - Micah Gersten uploads fix to ubuntu-mozilla-security PPA
  • 2011-06-22 23:21 - Clint Byrum tells us that the language packs aren't installed by default on a server
  • 2011-06-22 23:21 - Rick Spencer says discussion should move to #ubuntu-mozillateam
  • 2011-06-22 23:26 - Clint Byrum tells us that there are good reasons to install language packs on a server
  • 2011-06-22 23:26 - Marc Deslauriers starts a test server install to see what the actual impact is
  • 2011-06-22 23:42 - Kees Cook joins in to attempt to reproduce the issue with unattended updates
  • 2011-06-23 00:09 - Kees Cook and Marc Deslauriers agree that unattended updates won't be affected, so we don't have to pull the release from the archive
  • 2011-06-23 00:10 - We decide to release i386/amd64 when ready and do a no change rebuild later to pick up powerpc/armel
  • 2011-06-23 00:20 - Rick Spencer dented the issue on the ubuntustatus account

  • 2011-06-23 01:52 - Jamie Strandboge returns, Micah Gersten briefs him, Jamie offers to help test the final binaries
  • 2011-06-23 03:35 - i386/amd64 builds done
  • 2011-06-23 04:14 - Jamie Strandboge finishes testing amd64, no regressions over previous functionality
  • 2011-06-23 04:31 - Micah Gersten finishes testing a dist-upgrade with a language pack installed and i386, dist-upgrade issue fixed, no regressions over previous functionality found in QRT testing
  • 2011-06-23 04:33 - Binaries copied to natty-security
  • 2011-06-23 05:04 - Binaries published in natty-security
  • 2011-06-23 06:04 - Binaries published in natty-updates
  • 2011-06-23 20:00 - USN-1157-3 sent explaining update

Successes

  • Response was prompt and effective.

Problems

  • The problem was not identified in testing or peer-review.
  • The security team had to provide and publish the fixes for the users.

Recommendations

  • Large changes like this should go through the standard SRU process rather than a security update.

IncidentReports/2011-06-22-Dist-upgrade-pulls-firefox (last edited 2011-06-24 16:36:21 by jdstrand)