1. Context

Today, testing a package, a fortiori the kernel, may be slow because of the waiting period that compensate the lack of rigorous test suites to regression-test changes, lack of hardware for QA’ing, we are mainly relying on community for testing, the amount of patches that get rolled can be important,... All of this can lead to a bug with a know fix going unaddressed in Ubuntu for several months.

We need to shorten the delay of validation for Kernel SRUs.

2. Processes to update a Stable Release

The following current processes exist to publish an update to a stable release:

• General SRU process: https://wiki.ubuntu.com/StableReleaseUpdates

• Kernel SRU: https://wiki.ubuntu.com/KernelTeam/KernelUpdates

• Security Updates: https://wiki.ubuntu.com/SecurityTeam/UpdateProcedures

The Update Process

To achieve a high level of quality, the process is broken down into the following stages:

• Packaging
• Testing
• Publication

In the context of Kernel SRUs we are interested by how to achieve steps 2 and 3 and being able to achieve that on short delay.

Testing an SRU

The tasks to be fulfilled are:

• Review the test case and reproduce the issue (Community + Kernel Team)
• build in a clean build environment (Integrated to the build process)
• verify the package still installs
• verify the package upgrades cleanly
• verify the package still functions properly
• verify the bug is fixed using the provided test case
• verify that it introduces no regression

Today we are relying on the community to achieve theses tasks. The community usually do, to some level, whole or part of tasks 1, 3, 4, 5 and 6, rarely 7. Step 2 (build) is done automatically by the build environment when the package is published to -proposed. In any case, we need a human intervention to test the fix and above all that it introduces no regression.

Test Procedure

We will run, for each version in -proposed, the following:

• The QA Regression Testing kernel tests;

• The LTP Lite test suite;

• Some of the kernel autotest modules.

Each of the above tests will be run on i386, AMD64, KVM, and EC2 environments, for both the -generic and -server kernel flavours.

The HW Cert Lab / Kernel Automated Tests

Using the wide array of hardware in the cert lab to test proposed SRUs is an effective way of addressing the problem of hardware diversity encountered with Kernel SRU. Kernel Automated Tests, ensures that at minimum, the system boots and can be online. From that point the user can update the system and bring it back to normal operational conditions.

Basically, the process of Kernel Automated Tests does the following:

• We poll -proposed once a day and check if there are changes,
• If something has changed, we install the system and all everything in -proposed
• Then the automated tests run against that

Publication

In the current SRU process, the patched version of the package will be published to -proposed then to -updates once it is tested and validated.

3. Regular cadence micro-releases

Proposal

The proposal is to introduce regular cadence “bug fix/HW enablement” releases and a non-resetting bake window.

We would have such a release every 2 weeks (frequency to by defined). During this period, if a security fix is released, it's added to the release in -proposed without resetting the the bake window, ensuring that we can engage on a strict ETA.

Fixes that are not verified at the end of the first week are removed from the update. The author of the report is responsible of the verification. Testing will be performed during the second week.

The main constraint is that most of the regression testing must be automated.

At a minimum we need to ensure that:

• The system boots
• X comes up for the desktop or network comes up for the servers
• Updates can be pulled and installed with update-manager or apt

Pros/Cons

• Pros:
  • Strict ETA
  • Ability to use the Automated Kernel Testing
  • Testing limited to the fix but still need regression testing over a large range of setup
  • Keep the work close to the Ubuntu Tree
• Cons:
  • Do not rely on the community for testing.
  • Testing limited to the internal capacity
  • Strongly rely on the quality and completeness of the QA Regression Test Suite

Rollback

TBD

Procedure