Owner:

Incident Description

A wxwidgets2.8 lucid SRU has been breaking other packages depending on it. Those needed to be rebuilt. Despite the fact that this need was identified early, the SRU reached lucid-updates.

Crisis Response Team

• Ubuntu SRU Team

Events

• 2010-07-20 Devid Antonio Filoni uploads wxwidgets2.8 2.8.10.1-0ubuntu1.1 to lucid-proposed, fixing bugs 379573 and 559822

• 2010-07-27 wxwidgets2.8 2.8.10.1-0ubuntu1.1 accepted in lucid-proposed

• 2010-07-28 pgadmin3 reported unusable for those that upgraded to lucid-proposed or maverick in bug 610975

• 2010-08-06 Thierry Carrez mentions the regression on bugs 379573 and 559822

• 2010-08-06 Devid Antonio Filoni mentions we can rebuild pgadmin3 later, after packages acceptance

• 2010-08-06 Devid Antonio Filoni uploads wxwidgets2.8 2.8.10.1-0ubuntu1.2 to lucid-proposed, re-fixing bug 559822

• 2010-08-17 wxwidgets2.8 2.8.10.1-0ubuntu1.2 accepted in lucid-proposed
• 2010-08-20 Jean-Baptiste Lallement marks 2.8.10.1-0ubuntu1.2 verification-failed due to regression
• 2010-08-22 Devid Antonio Filoni changes verification-failed to verification-done, with comment "Fix works fine. In order to fix that regression we have to rebuild pgadmin3 package, that cannot be fixed in wxwidgets2.8."
• 2010-08-24 Jean-Baptiste Lallement mentions SRU regression policy but the package was already pushed to -updates

• 2010-08-24 15:33 (UTC) ScottKitterman Initiates the -updates regression procedure

• 2010-08-24 Martin Pitt pushes quick rebuilds for the affected packages in order to quickly fix the situation without pulling off the wxwidgets2.8 update

Successes

• Regression was identified, and the SRU bug was marked verification-failed
• Quick handling of the issue by jean-Baptiste and Martin ensured a small window of vulnerability to the issue

Problems

• The package should not have reached -updates before the corresponding needed rebuilds were ready
• Ubuntu-sru was not subscribed to the bug where the regression was identified (379573) until after the regression had been announced on #ubuntu-devel
• Rebuilds were only prioritized and pushed to -updates after the regression procedure was initiated by a developer who only knew of it because he gets bugmail for the package in question. The person who was most closely associated with the SRU explicitly declined to report the regression.

Recommendations

• The person who uploads the fix should not be allowed to edit verification-failed -> verification-done ?

• Need to understand how the package got to -proposed and -updates without ubuntu-sru being subscribed to all relevant bug (this is according to pitti, the bug log doesn't show this detail).
• Perhaps email to u-d-a to reinforce the current -updates regression procedure and the importance of following it in all cases.