2011-02-25-Permissions-build-failures

Differences between revisions 4 and 5
Revision 4 as of 2011-02-25 22:05:18
Size: 1960
Editor: 65
Comment:
Revision 5 as of 2011-02-25 22:17:23
Size: 2224
Editor: 99-191-111-134
Comment:
Deletions are marked like this. Additions are marked like this.
Line 24: Line 24:
 * 2011-02-25 ??:?? Phone discussion with Rick Spencer(holiday), Elliot Murphy (traveling) - La Mont handling disabling.  * 2011-02-25 ??:?? Kate phone discussion with Rick Spencer(holiday), Elliot Murphy (traveling) - La Mont handling disabling.
Line 27: Line 27:
 * 2011-02-25 22:04 La``Mont and Robert Collins trying to identify root cause and scope of impact on #launchpad-ops
 * 2011-02-25 22:06 La``Mont confirmed ARM and PPC not affected - hardy based buildds to remain on manual now until root cause established

Owner:

Incident Description

Packages were built with incorrect permissions. Most failed to build, but a few succeeded despite the error.

Crisis Response Team

Events

  • 2011-02-25 16:07 Martin Pitt notices a strange hal build failure, and asks LaMont Jones to investigate. Results are inconclusive, although LaMont suggests possible umask issues.

  • 2011-02-25 17:54 Chris Coulson asks the same question. Results remain inconclusive.
  • 2011-02-25 19:26 Further discussion between Chris Coulson, Ken VanDine, and Colin Watson.

  • 2011-02-25 20:32 Colin Watson diffs build logs and determines that there are no updates to natty which might have been relevant.
  • 2011-02-25 20:36 Colin asks #launchpad-ops for help.
  • 2011-02-25 20:50 Colin realises that some packages are being misbuilt containing files with mode 0600, and rings the IS emergency phone. James Troup answers but is not at a computer, and says he'll track down another sysadmin.
  • 2011-02-25 20:57 Colin escalates up his management chain. Robbie Williamson is on holiday; Rick Spencer answers but is also on holiday, and suggests contacting Elliot Murphy (IS) and Kate Stewart (handover within platform). When Colin gets off the phone, LaMont has already responded, so there was no need to contact Elliot.

  • 2011-02-25 21:10 Colin belatedly disables the Launchpad publisher to try to limit damage.
  • 2011-02-25 21:13 LaMont disables the build daemons.

  • 2011-02-25 21:21 Kate Stewart responds; Colin hands over.
  • 2011-02-25 ??:?? Kate phone discussion with Rick Spencer(holiday), Elliot Murphy (traveling) - La Mont handling disabling.
  • 2011-02-25 21:45 Rick Spencer calls Pete Graner asking to get involved and help Kate and company.
  • 2011-02-25 21:52 Pete Graner sent out identi.ca message, to stop downloading natty builds.
  • 2011-02-25 22:04 LaMont and Robert Collins trying to identify root cause and scope of impact on #launchpad-ops

  • 2011-02-25 22:06 LaMont confirmed ARM and PPC not affected - hardy based buildds to remain on manual now until root cause established

Successes

Problems

Recommendations

IncidentReports/2011-02-25-Permissions-build-failures (last edited 2011-03-23 17:36:00 by pool-108-1-168-116)