Stable Kernel Release Cadence

Goals

The Kernel Stable Cadence was arrived at during discussions before and during Natty Narwhal Developer Summit in Orlando. The Blueprint from the UDS sessions may be found here. The new process has a number of goals which are designed to improve some problems with the old process. These goals are:

Reduce the maximum time between valid fixes getting applied and verified and their delivery to the end user.
Assure that every bug fix is verified to solve the problem while the kernel is in -proposed testing.
Increase kernel quality by applying comprehensive regression and certification testing to all stable kernels before release.
Reduce the time until non-critical and non-embargoed security fixes are applied and released.
Reduce the impact that critical and embargoed security updates have on the normal stable release schedule.
Reduce the amount of time it takes for upstream stable kernel updates to reach users.
Avoid kernel updates which contain a very large number of updates.
Provide more predictability of when a fix that's been applied to the source repository will be released.

After the Natty cycle, further discussions happened in Oneiric Ocelot Developer Summit in Budapest (see blueprint), where some changes were proposed and done.

Implementation

Whenever possible, Stable kernels will now be released on a short cadence. The cycle consists of three phases - the preparation phase, the verification phase, and the testing phase. The intended complete cycle length is three weeks, with one week for each phase. The cycle immediately repeats every three weeks, with the exception of holidays, UDS weeks, and other considerations. An attempt will be made to synchronize the schedule with the the Ubuntu platform schedule. The specific kernel SRU cadence dates and schedule is posted to the kernel-sru-announce mailing list, as well as on the front page of https://kernel.ubuntu.com/.

The cycle begins with preparation of a new kernel which contains all accumulated fixes.

Patches will be divided into the following classes:

Non-Security Patches

Stable Upstream Patches
- Stable updates from upstream (GregKH's repos) will be accepted without testing of individual functionality.
Other Patches
- No patch will be accepted for a bug if we don't have the hardware to test it or high confidence that it will be tested by the community.
  These patches are required to be verified during the verification phase, and if not verified, will be reverted before the testing phase begins.

Security (CVE) Patches

With the faster release cycle, most CVEs can be released faster as part of the normal stable kernel cycle than if we created a special security release. Critical security fixes are those which must be released as soon as possible, generally because they involve a trivially exploitable privilege escalation vulnerability or something similar. N.B. embargoed CVEs are not necessarily critical.

Non-Critical CVEs
- Non-critical CVE patches will be applied directly to the master branch of the affected git repository and will be part of the next -proposed upload. An out-of-cycle -security upload will not be required for these patches.
Non-embargoed Critical CVEs
- If it arrives during the verification phase we will:
- Take the last kernel which was released (from -updates or -security), make the security fix on that, then push it through whatever testing that the security team does and certification testing, and release that to -security and -updates.
- Place all the changes which were currently in verification back onto this new kernel, and accept that any which have been verified are still valid passes for verification, and continue the verification cycle using this kernel with the security fix.
- This may extend the verification phase a little, but not much, and we would then enter testing phase.
- If it arrives during the testing phase we will:
  - Already have reverted all non-verified fixes from the verification phase
  - Apply the security fix
  - Begin the testing phase, restarting it if needed
  - Notify the security team to perform any testing they require.
Embargoed CVEs
- These are handled the same as Non-embargoed Critical CVEs, but we may have to adjust the actual release date. This will be handled on a case-by-case basis.

Testing

There are 3 types of required testing: verification, certification, and regression. Verification testing happens during the verification phase of the cycle, and certification and regression testing occur during the testing phase.

Verification testing is the assurance that a bug is actually fixed by the proposed kernel, and is the responsibility of the bug reporter. Tags used for tracking verification are verification-needed-<releasename>, verification-done-<releasename>, verification-reverted-<releasename> and verification-failed-<releasename>.
Certification is assurance that the kernel passes certification tests on specific hardware, and is the responsibility of the Platform Services, certification QA team. Tags used for tracking certification are certification-testing-passed and certification-testing-failed
Regression testing is testing to assure that no bugs have been re-introduced by the proposed kernel, and is the responsibility of the Platform QA team. Tags used for tracking regression testing are qa-testing-passed and qa-testing-failed
A tracking bug is created and used with each stable kernel in order to track status and completion of various tasks, including testing.
- See details in 'Tracking Bug Procedure' below
- All related teams which have testing tasks should be subscribed and receive notifications when a new tracking bug is created.
All tracking bugs associated with upstream stable patch sets will be set to verification-done-<releasename> QA when the kernel has passed testing.

Details

Changes from bug reports will only be considered if we have the hardware and resources to test the fix, or a very high confidence that the bug reporter will test the proposed release.
At the beginning of the three-week cycle, bug fixes, upstream stable patches, and non-embargoed security fixes will be applied, and the kernel published to -proposed.
At the end of the second week, any bug fixes which have not been verified as fixed by the reporter or by internal testing, or which failed verification, will have the patches reverted, and the associated bug tasks set to "Incomplete". A comment will be added to the bug describing how to request reapplication of the fix. After the kernel is respinned with the reverted patch, during its verification the tag verification-reverted-<releasename> will be added, replacing any verification-{needed,failed}-<seriesname> tag that might be present.
During the third week, certification and regression testing will be performed to insure that the kernel is functional. Minimum testing will insure that the kernels boots, networking operates, X starts (for workstations), and updates can be performed. Certification tests will not include the entire set of manually executed tests, but will include an automated set of tests. Testing will be a joint effort by the QA and certification teams. The exact testing to be performed is To Be Determined.
Proposed releases will be tracked with a tracking bug opened by the stable kernel team, and linked via an entry in the changelog. This bug is tracked on the kernel team sru-report page. See details in 'Tracking Bug Procedure' below.
Security kernels which are critical or embargoed will also receive the same tests required for proposed kernels.

Tracking Bug Procedure

For tracking and triggering process steps including certification and regression testing, as well as archive admin operations, a tracking bug will be created and associated with each stable kernel in order to track status and completion of testing.

About the tracking bug:

It is created by the Kernel Team _before_ uploading the new kernel to -proposed, using a script the kernel team created (create-release-tracker), which can be found in the kteam-tools git repository. The created bug is tagged with "kernel-release-tracking-bug".
Once the bug is created, be sure that the changelog references the SRU bug number using the 'LP: #NNNNNN' (SRU) convention
The description and title of the tracking bug must contain the exact version of the kernel it is tracking.
The report is nominated for the appropriate Ubuntu release.
All relevant teams must be subscribed to the bug report (usually sru-verification, ubuntu-sru, hardware-certification).
The publication to -proposed will follow the Stable Release Updates procedure.
The tracking bug will contain all tasks needed to perform the kernel SRU Workflow. The details about the kernel SRU Workflow can be seen here.

About Verification Phase in the tracking bug:

If any patches have been reverted, new kernels will be uploaded and built. A new tracking bug will be created, and the previous one will be closed or set as a duplicate of the new one. On the verification phase of new bug, any bugs which had patches reverted will be tagged with verification-reverted-<releasename>.

About Regression testing in the tracking bug

the QA Team can start working when the Regression-testing task is set to 'Confirmed'; when they start testing, they set the same task to 'In Progress'. After the tests are done, they add to the report the result of the test, and add the the tag 'qa-testing-passed' if the tests were ok, or 'qa-testing-failed' if tests failed, and sets the Regression-testing task to 'Fix Released'

About Certification testing in the tracking bug

Certification testing is not done on Hardy, only with Lucid or later.
Certification team don't do testing on backports packages currently.
Certification team can start working when the Certification-testing task is set to 'Confirmed'; when they start testing, they set the same task to 'In Progress'. After the tests are done, they add to the report the result of the test, and add the the tag 'certification-testing-passed' if the tests were ok, or 'certification-testing-failed' if tests failed, and sets the Certification-testing task to 'Fix Released'

When all relevant testing is done and passed on a tracking bug, packages are ready to be copied to -updates/-security. It was decided in the Kernel-Q sprint in Portland, OR that kernels shouldn't be copied to -proposed/-security near or on weekends. John Johanssen raised the issue that, at least for the security team, there are not enough man power on weekends to address any fallout that could happen after things are copied to -updates/security. It was agreed that no publishing to -updates/-security pockets should be done between 18:00 UTC Friday - 21:00 UTC Sunday. Therefore, the Promote-to-updates and Promote-to-security tasks should only be set to Confirmed when not on late Friday/weekend, and Archive Admins should be aware that no copying should be done in this case. Only if tasks are confirmed and we are not near or on a weekend, should the Archive Admin publish the kernel to the -updates/-security pockets. See kernel sru workflow page for more information.

Reconsideration of reverted fixes

When fixes are reverted due to lack of verification, they may be proposed for reconsideration by performing the following:

Update the bug with a firm commitment to verify the fix during the next stable kernel release cycle (usually this will begin one week after the fix is reverted). Change the "verification-reverted-<release>" tag to "verification-needed-<release>" (where <release> is [lucid | maverick . . .]).

Implementation Schedule

All SRU cycles will try adhere as much as possible to the three-week cadence, with adjustments for holidays.

Regressions or delays with other teams may make the schedule slip.

Text for bugs

When a kernel is released to -proposed, the following text will be added as a comment:

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-<releasename>' to 'verification-done-<releasename>'.

If verification is not done by one week from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

When a fix is not verified by the deadline and is reverted, the following text will be added as a comment:

Verification that this bug is fixed has not been completed by the deadline for the current stable kernel release cycle. The change will be reverted and this bug is being set to incomplete.

In order to have this fix considered for reapplication to the kernel, please follow the process documented here:

https://wiki.ubuntu.com/Kernel/StableReleaseCadence

Discussions about the new process tend to take place in #ubuntu-kernel on IRC, so please contribute to the discussion there if you would like.

Thank you!

When a fix is a pre-stable commit (will come later with stable updates) and the kernel is released to -proposed, no verification is needed, and the following text will be added as a comment:

This commit is an early application of a commit that will be coming in via upstream stable. As such it is not subject to the standard bug verification process.

When a fix is a commit which came with stable updates and the kernel is released to -proposed, no verification is needed, and the following text will be added as a comment:

The commit for this issue came in via a stable upstream release. As such it is not subject to the standard bug verification process.

How and where to upload the package

If the update has embargoed CVEs or high priority updates that will supersede the proposed kernel process, use -security in the changelog and hand it off to the security team and you are done
Otherwise, upload to the Kernel PPA with -proposed in the changelog, following the normal kernel SRU workflow. (See "Uploading to the Kernel PPA" for more details.)
Once built the package can be copied out to the relevant -proposed pocket for testing. Once a package is copied to the -proposed pocket, it can be deleted from the PPA. (See "Pocket Copying from the Kernel PPA to -proposed").