Launchpad Entry: hardware-kernel-n-stable-process-review
Created: 2010-10-05
Contributors:
Packages affected:
Summary
The purpose of this spec is to go over the current stable process (including the SRU policy) to see how we are doing and to see which parts can be improved or documented better.
Release Note
The impact on users would be to have a place which well documents what changes can be made to stable releases, how to ask for inclusion and why decision will be made the way they are done. This is less likely to go into the general release notes but should be linked from a prominent place on the kernel wiki page.
Rationale
The stable update process for released kernels has been very non-deterministic, and critical security updates during proposed kernel testing periods have occurred repeatedly, pushing back releases and resulting in released with a very high number of changes. There has been difficulty in getting verification testing completed for bugs, becuase we lack the needed hardware, and because the bug reporters have not tested the proposed kernels in a timely manner.
Assumptions
- Upstream stable updates should get into the master branch as quickly as possible.
- CVEs should be fixed as soon as possible
- Certification testing needs to take place in proposed.
- Regression testing needs to be done in proposed.
- proposed uploads should take place often enough to prevent having a huge number of changes in any one release.
- proposed kernel production should be scheduled as predictably as possible, and the affect of having a critical security release take place should cause minimal delay in releasing.
Design
The stable kernel team will change to a regular two-week release cadence. The cycle is two weeks long, and repeats every two weeks, with the exception of holidays, UDS weeks, and other considerations. An attempt will be made to sync the schedule with the needs of the Ubuntu platform schedule and Linaro release schedule.
Non-CVE Patches
- Stable Upstream Patches
- Stable updates from upstream (GregKH's repos) will be accepted without testing of individual functionality.
- Other Patches
- No patch will be accepted for a bug if we on't have the hardware to test it or high confidence that it will be tested by the community. At the end of the first week of baking in -proposed, if these patches have not been verified as fixing the specific bugs the patch will be reverted.
CVE Patches
- Non-Critical
- Non-critical CVE patches will be applied directly to the master branch of the effected git repository and will be part of the next -proposed upload. A special -security upload will not be required for these patches.
- Critical
- Critical CVE patches will be applied to the master branch of the effected git repository and will be uploaded as a -security release which will be released imediately.
Testing
- There are 3 types of required testing: certification, regression and bug specific. The testing of a -proposed kernel must be completed within 14 working days begining when the upload has been accepted to the -proposed pocket.
- Certification is the responsibility of the Platform Services, certification QA team.
- Regression testing is the responsibility of the Platform QA team.
- Bug specific testing is the responsibility of the originating bug author.
Implementation
- The SRU policy will be amended to indicate that changes will only be considered if we have the hardware and resources to test the fix, or a very high confidence that the bug reporter will test the proposed release.
- Kernel stable updates will follow a two week cycle
- At the beginning of the cycle, bug fixes, upstream stable patches, and non-embargoed security fixes will be applied, and the kernel uploaded to proposed.
- At the end of the first week, any bug fixes which have not been verified as fixed by the reporter or by internal testing will have the patches reverted, and the associated bugs closed as "wontfix".
- During the second week, certification and regression testing will be performed to insure that the kernel is functional. Minimum testing will insure that the kernels boots, networking operates, X starts (for workstations), and updates can be performed. Certification tests will not include the entire set of manually executed tests, but will include an automated set of tests. Testing will be a joint effort by the QA and certification teams.
- Cert and QA will begin testing the kernels currently in proposed next week (1 Nov, 2010) and report on the amount of effort and resources required to perform testing. These results will be used to evaluate the resources required to test all expected variants of released kernels.
- Proposed releases will be tracked with a tracking bug opened by the stable kernel team, and linked via an entry in the changelog. This bug will be assigned to the QA team (or a launchpad team - TBD), and tracked on the kernel bug dashboard. The bug will be set to verified when all QA and Cert testing has passed.
- We need to decide which systems will be tested. Should it be most common, or some subset? It will include at least one virtual environment.
- Security kernels which are critical or embargoed will also receive the same tests required for proposed kernels.
Action Items
- [sconklin] boiler plate text for bug closure on failure to verify
- [canonical-kernel-team] look at arsenal scripts to nag the users
- [skaet][sconklin] workout the sru schedule with respect linaro and platform
- [sconklin] provide a document that describes the process, bug tracking, and which tests are being performed, etc
- [jfo] make sure that the bugs that we are sending for this are on the dashboard
- [pitti] ensure verification bug update indicates that this is going to be pulled
- [sconklin] process to add the bug to the changelog manually during release process.
- [marjo] QA team - define a mechanism to assign the proposed upload tracking bug to them and track results
- [victor] QA team - provide a description of what testing they provide, and an idea of the number of hours required to run those
[leann] provide info from HWDB http://reports.qa.ubuntu.com/reports/ogasawara/hwdb/system-stats-popular-20101019.html
- [action] Victor to investigate which systems are "common" for cert testing
- [cert-team] blueprint for cert-team
Unresolved issues
Actual start date for the new cycles (depends on examination of schedule and resources)
Do we have the capacity to test the required number of kernels?