1. Context

Today, testing a package, a fortiori the kernel, may be slow because of the waiting period that compensate the lack of rigorous test suites to regression-test changes, lack of hardware for QA’ing, we are mainly relying on community for testing, the amount of patches that get rolled can be important,... All of this can lead to a bug with a know fix going unaddressed in Ubuntu for several months.

We need to shorten the delay of validation for Kernel SRUs.

2. Processes to update a Stable Release

The following current processes exist to publish an update to a stable release:

The Update Process

To achieve a high level of quality, the process is broken down into the following stages:

  • Packaging
  • Testing
  • Publication

In the context of Kernel SRUs we are interested by how to achieve steps 2 and 3 and being able to achieve that on short delay.

Testing an SRU

The tasks to be fulfilled are:

  • Review the test case and reproduce the issue (Community + Kernel Team)
  • build in a clean build environment (Integrated to the build process)
  • verify the package still installs
  • verify the package upgrades cleanly
  • verify the package still functions properly
  • verify the bug is fixed using the provided test case
  • verify that it introduces no regression

Today we are relying on the community to achieve theses tasks. The community usually do, to some level, whole or part of tasks 1, 3, 4, 5 and 6, rarely 7. Step 2 (build) is done automatically by the build environment when the package is published to -proposed. In any case, we need a human intervention to test the fix and above all that it introduces no regression.

Test Procedure

We will run, for each version in -proposed, the following:

Each of the above tests will be run on i386, AMD64, KVM, and EC2 environments, for both the -generic and -server kernel flavours.

The HW Cert Lab / Kernel Automated Tests

Using the wide array of hardware in the cert lab to test proposed SRUs is an effective way of addressing the problem of hardware diversity encountered with Kernel SRU. Kernel Automated Tests, ensures that at minimum, the system boots and can be online. From that point the user can update the system and bring it back to normal operational conditions.

Basically, the process of Kernel Automated Tests does the following:

  • We poll -proposed once a day and check if there are changes,
  • If something has changed, we install the system and all everything in -proposed
  • Then the automated tests run against that


In the current SRU process, the patched version of the package will be published to -proposed then to -updates once it is tested and validated.

3. Regular cadence micro-releases


The proposal is to introduce regular cadence “bug fix/HW enablement” releases and a non-resetting bake window.

We would have such a release every 2 weeks (frequency to by defined). During this period, if a security fix is released, it's added to the release in -proposed without resetting the the bake window, ensuring that we can engage on a strict ETA.

Fixes that are not verified at the end of the first week are removed from the update. The author of the report is responsible of the verification. Testing will be performed during the second week.

The main constraint is that most of the regression testing must be automated.

At a minimum we need to ensure that:

  • The system boots
  • X comes up for the desktop or network comes up for the servers
  • Updates can be pulled and installed with update-manager or apt


  • Pros:
    • Strict ETA
    • Ability to use the Automated Kernel Testing
    • Testing limited to the fix but still need regression testing over a large range of setup
    • Keep the work close to the Ubuntu Tree
  • Cons:
    • Do not rely on the community for testing.
    • Testing limited to the internal capacity
    • Strongly rely on the quality and completeness of the QA Regression Test Suite








- Provide a fix
- Nominate for SRU
- Write a test case
- Document any potential regression

Dev team


- Validate the SRU and mandatory information
- Publication to -proposed

SRU Reviewing Team

Testing version in -proposed

- Test that the patch fixes the issue and check for inadvertent side effect

- Reporter
- Community

- Run Kernel Automated Testing in Cert Lab

HW Cert Team / Platform QA


Publication to -updates

SRU Team

QATeam/KernelSRU (last edited 2013-02-22 23:43:03 by javier-lopez)