2011-02-25-Permissions-build-failures

Revision 4 as of 2011-02-25 22:05:18

Clear message

Owner:

Incident Description

Packages were built with incorrect permissions. Most failed to build, but a few succeeded despite the error.

Crisis Response Team

Events

  • 2011-02-25 16:07 Martin Pitt notices a strange hal build failure, and asks LaMont Jones to investigate. Results are inconclusive, although LaMont suggests possible umask issues.

  • 2011-02-25 17:54 Chris Coulson asks the same question. Results remain inconclusive.
  • 2011-02-25 19:26 Further discussion between Chris Coulson, Ken VanDine, and Colin Watson.

  • 2011-02-25 20:32 Colin Watson diffs build logs and determines that there are no updates to natty which might have been relevant.
  • 2011-02-25 20:36 Colin asks #launchpad-ops for help.
  • 2011-02-25 20:50 Colin realises that some packages are being misbuilt containing files with mode 0600, and rings the IS emergency phone. James Troup answers but is not at a computer, and says he'll track down another sysadmin.
  • 2011-02-25 20:57 Colin escalates up his management chain. Robbie Williamson is on holiday; Rick Spencer answers but is also on holiday, and suggests contacting Elliot Murphy (IS) and Kate Stewart (handover within platform). When Colin gets off the phone, LaMont has already responded, so there was no need to contact Elliot.

  • 2011-02-25 21:10 Colin belatedly disables the Launchpad publisher to try to limit damage.
  • 2011-02-25 21:13 LaMont disables the build daemons.

  • 2011-02-25 21:21 Kate Stewart responds; Colin hands over.
  • 2011-02-25 ??:?? Phone discussion with Rick Spencer(holiday), Elliot Murphy (traveling) - La Mont handling disabling.
  • 2011-02-25 21:45 Rick Spencer calls Pete Graner asking to get involved and help Kate and company.
  • 2011-02-25 21:52 Pete Graner sent out identi.ca message, to stop downloading natty builds.

Successes

Problems

Recommendations