2011-02-25-Permissions-build-failures
Differences between revisions 7 and 8
|
Size: 2751
Comment:
|
Size: 8
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| <<TableOfContents>> Owner: === Incident Description === Packages were built with incorrect permissions. Most failed to build, but a few succeeded despite the error. === Crisis Response Team === === Events === * 2011-02-25 16:07 Martin Pitt notices a strange hal build failure, and asks La``Mont Jones to investigate. Results are inconclusive, although La``Mont suggests possible umask issues. * 2011-02-25 17:54 Chris Coulson asks the same question. Results remain inconclusive. * 2011-02-25 19:26 Further discussion between Chris Coulson, Ken Van``Dine, and Colin Watson. * 2011-02-25 20:32 Colin Watson diffs build logs and determines that there are no updates to natty which might have been relevant. * 2011-02-25 20:36 Colin asks #launchpad-ops for help. * 2011-02-25 20:50 Colin realises that some packages are being misbuilt containing files with mode 0600, and rings the IS emergency phone. James Troup answers but is not at a computer, and says he'll track down another sysadmin. * 2011-02-25 20:57 Colin escalates up his management chain. Robbie Williamson is on holiday; Rick Spencer answers but is also on holiday, and suggests contacting Elliot Murphy (IS) and Kate Stewart (handover within platform). When Colin gets off the phone, La``Mont has already responded, so there was no need to contact Elliot. * 2011-02-25 21:10 Colin belatedly disables the Launchpad publisher to try to limit damage. * 2011-02-25 21:13 La``Mont disables the build daemons. * 2011-02-25 21:21 Kate Stewart responds; Colin hands over. * 2011-02-25 ??:?? Kate phone discussion with Rick Spencer(holiday), Elliot Murphy (traveling) - La Mont handling disabling. * 2011-02-25 21:45 Rick Spencer calls Pete Graner asking to get involved and help Kate and company. * 2011-02-25 21:52 Pete Graner sent out identi.ca message, to stop downloading natty builds. * 2011-02-25 22:04 La``Mont and Robert Collins trying to identify root cause and scope of impact on #launchpad-ops * 2011-02-25 22:06 La``Mont confirmed ARM and PPC not affected - hardy based buildds to remain on manual now until root cause established * 2011-02-25 22:38 La``Mont identifies upgrading sudo Version: 1.7.2p1-1ubuntu5.3 differs from Version: 1.6.9p10-1ubuntu3.8 at 11:39 as cause. * 2011-02-25 22:40 <lamont> proposed plan from here: lamont to revert the sudoish changes on the buildds, wgrant to find out what build records we need to retry, buildds can go back on auto as sudo is downreved and sudoers fixed * 2011_02-26 00:38 wgrant gets list of broken packages * 2011-02-26 00:50 lamont reuploads the broken packages, switches the publisher back on. === Successes === === Problems === === Recommendations === |
deleted |
deleted
IncidentReports/2011-02-25-Permissions-build-failures (last edited 2011-03-23 17:36:00 by pool-108-1-168-116)