Owner: DaveWalker
Incident Description
The update for https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/978464 appears to have exposed a bug in either grub or apt that is breaking scripts based on the Amazon images from cloud.ubuntu.com. dpkg is giving the upgraded config file prompt upon configuring grub-pc ("A new version of configuration file /etc/default/grub is available, but the version installed currently has been locally modified.")
Crisis Response Team
- Scott Ritchie
- Christopher James Halse Rogers
- Kate Stewart
- Martin Pitt
- Steve Langasek
- Dave Walker
Events
All times are in UTC.
2012-06-06 02:50 - on #ubuntu-devel Scott Ritchie(YokoZar1) commented about regression with grub on the AMI images.
- 2012-06-06 03:05 - Scott Kitterman flagged !regression-alert (note: ubottu list is out of date and needs scrub)
- 2012-06-06 03:08 - Chris started investigation.
- 2012-06-06 03:08 - Kate starts of incident report, and pings to cloud team members (smoser, utlemming)
2012-06-06 03:09 - Scott R opens Bug 1009294 to track issue with https://launchpad.net/ubuntu/+source/grub2/1.99-21ubuntu3.1
- 2012-06-06 03:18 - Chris figured out issue is with ucf managed conf file. Work around is setting the environment variable UCF_FORCE_CONFFNEW (or UCF_FORCE_CONFFOLD) which should bypass, and silently install the new (or leave the old) file.
2012-06-06 03:29 - Chris adds workaround to http://askubuntu.com/questions/146921/how-do-i-apt-get-y-dist-upgrade-without-a-grub-config-prompt
- 2012-06-06 03:55 - Chris pulls Martin into brainstorming on what reasonable solutions might be.
- 2012-06-06 04:04 - Martin recommends reverting the grub SRU and replace it with an SRU that only modifies the grub configuration if it is not locally modified.
- 2012-06-06 04:24 - Kate hands off tracking to Martin.
2012-06-06 04:32 - Chris uploads a reversion of the changes in grub 1.99-21ubuntu3.1 into -proposed
2012-06-06 05:21 - Steve Langasek points out that this is not a bug from 1.99-21ubuntu3.1 in particular, but would have happened with any grub update. Either the AMI has a corrupted ucf database, or there is a bug in ucf; it might be related to the rather unusual invocation of ucf with --sum-file=/usr/share/grub/default/grub.md5sum
- 2012-06-06 05:30 - Martin removes grub 1.99-21ubuntu3.2 source from precise-proposed, and the built binaries from the accepted queue (i. e. the binaries of 3.2 never got published)
2012-06-06 05:43 - Neither Steve nor Chris can reproduce the prompt with manually modifying /etc/default/grub. Suspecting that the AMI build process changes the UCF database.
2012-06-06 05:58 - Steve sends updated analysis, reproducer, and workaround to the bug (https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1009294/comments/4)
2012-06-06 06:20 - Steve confirms bug in AMI build scripts and points out solution how to fix them. (https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1009294/comments/5)
2012-06-06 08:21 - Dave Walker comments that he doesn't believe this is a new incident, and is tracked via LP #759545.
- 2012-06-06 08:23 - Martin hands off tracking to Dave Walker.
- 2012-06-06 10:06 - Dave determines that the image building process should do something to update ucf.
- 2012-06-06 10:38 - Dave discovers that the potential duplicate is infact a different (but similar) issue, but considered by those effected to be the long standing tracking bug.
- 2012-06-06 13:21 - Discussions within the server team how to resolve this.
- 2012-06-06 21:54 - Discussions between Steve and Dave determine that "DEBIAN_FRONTEND=noninteractive dpkg-reconfigure grub-pc'" towards the end of the image build process would avoid this issue.
- 2012-06-06 22:03 - Original reporter requested to try this command on an instance prior to upgrade for cross-validation.
2012-06-06 22:45 - Original reporter confirms that the work around (& trial bug fix) resolves the issue. Scheduled to be added to the next spin of images, pending further testing.
Successes
- Caught early.
- Outlined wider issues in escalation process.
As this isn't really an SRU regression, the IncidentReport wasn't really required; however - it outlined some issues with the current process which proved useful. This will allow ongoing quality to improve, based on recommendations.
Problems
- ubottu notify list for regression-alert is out of date
- This was a known issue tracked throughout the Precise cycle, but was being tracked under an inaccurate bug description causing confusion.
- The person assigned to work on it, could not reproduce the issue as outlined. Whilst frequent discussions were attempted, they didn't prove fruitful; suspected to be a communication of the bug issue and manpower issue.
The original bug had an importance of Medium which is accurate for the package in question (https://wiki.ubuntu.com/Bugs/Importance), but higher than Medium for the Server flavour; meaning it was treated with less importance.
Recommendations
- Determine who should be highlighted with the !regression-alert factoid.
- Raise the importance of the original bug.
- Dedicated discussions between the Server and Foundations team to discuss and escalate issues that are pertinent to release.
- Improved validation of long standing bugs to ensure that the description remains accurate.