Derivative Packages Workflow

This is a rough guide from the stable kernel team for derivative kernel package maintainers, it explains the expected workflow to be used on packages that derive from the master linux branches (that we call derivatives or derivative packages). It assumes the derivative kernel package maintainers contacted the stable kernel team before about his maintainership and everything is settled already. Contact the kernel stable team for any questions.

Preparation phase

• The stable kernel team prepare the main linux packages each 3 weeks (Kernel SRU cadence), taking what is on master-next and putting that into master, doing a new update. We manually open a tracking bug, and have our process for that (what is described on Stable Patch Application and StablePackageBuilding pages). On the tracking bug, we set the prepare-package tasks to In Progress, push master-next to master, tag the release, build the source package, and upload to the canonical kernel team ppa. [1]

• After the build is done on the ppa, the stable bot (stable/sru-workflow-manager from kteam-tools) detects that, and sets the prepare-package tasks to Fix Released, and opens tracking bugs for the derivative packages in that series, so derivative package maintainers like you can start work on updates for the derivatives as well. [2]

• Using as an example the linux-armadaxp derivative package, you then get a tracking bug opened for it, and check if you can or need start working on a new armadaxp SRU at this point:
  • You usually start working on a new update only if a previous update released, that is, if there is not any previous SRU not released to -updates yet due to some delay, otherwise you wait; nothing prevents you to prepare the update in advance, just take care to not upload anything that in the end will overwrite something in -proposed that is still to be released.

  • If a new update doesn't make sense, like you don't need to rebase, or any exception, you must close the tracking bug. Just set all tasks on the tracking bug to invalid, and preferably add a comment to the tracking bug explaining the reason. You also can use "./stable/check-release-tracking-bugs --invalidate --bug=<bug number>" from kteam-tools to close a tracking bug for an update that isn't needed.

• If you need to do the armadaxp SRU, then this is usually the workflow you should be doing:
  • set the prepare-package task in the armadaxp tracking bug to In Progress.
  • you rebase armadaxp, usually against linux master that was pushed by us from which this armadaxp tracking bug derives. The bot always adds the first comment to the tracking bug pointing to the version from which it should be rebased.

  • add/remove/keep anything else on top needed specifically by armadaxp. Any new fix/change specific to armadaxp must conform to the Ubuntu SRU rules, have a BugLink in the changelog, and be verified later in the Verification phase.

  • close the release and tag it, making sure to include manually the armadaxp tracking bug number into the top of the debian package changelog when doing the closing commit. Look at the format looking at the main linux package changelog, should be the same.
  • to verify for any errors, you can run ./maintscripts/verify-release-ready from kteam tools inside the git checkout, before pushing. I'm not sure the script will work with armadaxp or with all derivatives, any problems can be fixed.
  • If everything is ok, builds fine, push it.
  • Update the tracking bug with the package version.
  • If there was no ABI bump, no new armadaxp meta package is needed, and in this case you set prepare-package-meta to Invalid, otherwise you set to In Progress.
  • Set the Prepare-package task to Fix Released.
  • If you have upload rights, set Upload-to-ppa task to In Progress, build/prepare the source packages and upload to the c-k-t ppa, including the -meta with abi bump if needed. If you don't have upload rights, ask someone which has it to do it. It's not the case for armadaxp, but if it was agreed between the stable team and the derivative maintainer that stable team must do the upload, after Prepare-package task is set Fix Released, on next run the stable bot will set Upload-to-ppa task to Confirmed, then we from stable team will look at that and build the source packages needed and upload them.
  • The stable bot monitors the builds on the c-k-t ppa, when they are done, the current In Progress tasks are set to fix released, and promote-to-proposed task is set to confirmed. When this happens, you should run copy-proposed-kernel.py from ubuntu-archive-tools if you have upload rights, and wait an archive admin copy the packages in the queue from the c-k-t ppa to proposed. Enter in contact with an archive admin if the copy doesn't happen. If you don't have upload rights, you may need someone that have to run copy-proposed-kernel.py, or may be enter in contact with an archive admin directly.
  • After the archive admin copied the packages to -proposed, set the promote-to-proposed task to fix released. On next stable bot run, it'll check if everything is ok, like packages landed in correct components. If the bot complains about any problem, you will have to contact an archive admin to fix it, and after that set the promote-to-proposed task to fix released again. If there is no problems, the stable bot will set the Verification-testing task to In Progress.

Verification phase

• When verification-testing is in progress, you must go to all bugs fixed in the update that need verification, and spam them, requesting that reporter or people affected test the -proposed kernel and verify the problem is fixed, this is to be in accordance with Ubuntu SRU rules. A list of bugs is generated for each SRU in progress at the Kernel SRU Report page, all bugs marked with an exclamation mark needs handling (they must be tagged, and may need verification). On each bug needing verification, add a verification-needed-<series> tag and add a comment that must have the request for testing, telling also to change the verification-needed-<series> tag to verification-done-<series> if testing is successful. And it must inform that the testing/verification must be done in 1 week, or the patch for the bug will be reverted. This last thing is important, we give 1 week for people to verify bugs in the SRU cadence, which we call verification week [3]. If no testing is done, you will have to revert the patch for the bug, set the tag to verification-reverted-<series>, set the relevant task in the bug to Incomplete, comment the patch is being reverted because of lack of testing and one must commit on testing for it to be readded again, and restart the SRU update, creating a new tracking bug (and setting the current tracking bug as duplicate of the new one). You can look for bugs we spammed for verification for examples.

• The sru-report page refreshes from time to time, bugs tagged as verification-needed-<series> shows with an '?' icon, and after people test and/or set the tags to verification-done-<series>, they show with a check mark icon.

• Not all untagged bugs listed on sru-report page need verification for the derivative package:
  • If the bug is not related to the derivative, and come from one of the changes on the rebase, it must be verified on the main release package. In this case, you should wait on verification done on the main package, after it's verified, you don't need to verify again that bug for the derivative, and it'll be properly tagged. If the bug only affects the derivative, it must be verified on the derivative, and you must request the testing as usual, adding needed tags etc.

  • If the patch for the bug is queued for a future stable update upstream for that series, it doesn't needed verification, and can automatically tagged as verification-done-<series> (with a comment in the bug added explaining this).

  • If the patch came through an upstream stable update, and a buglink is on it, it'll show on the report as untagged, just tag it as verification-done-<series>, and add a comment about that in the bug. We don't verify bugs that come through upsteam stable updates, think of it as a micro release exception.

  • In rare cases, you may have some bug that doesn't make sense to be tested, or by the nature of the problem can't be verified (like some bug that shows after uptime of more than 1 week, which clashes with the SRU testing limit). In this cases, add the verification-done-<series> tag, and a good comment explaining it on the bug.

• After all bugs are properly tagged and verified (that is, when there aren't any bugs with verification-needed-<series> tag left), you must set Verification-testing task to Fix Released. This will move the update from the Verification phase to the Testing phase.

Testing phase

• When verification-testing is fix released, the stable bot sets the testing tasks to confirmed (certification-testing, security-signoff, regression-testing).
  • certification-testing ensures that systems which were certified before have no problems with the new update, that is, they continue to work, with no new bugs. This is done by the Certification team. If certification testing isn't done for the package (the case for all derivatives at the moment), the task is irrelevant and set to invalid when the tracking bug is created; if the task isn't invalid when it should be, or vice versa, see with the kernel stable team to adjust that.
  • security-signoff is a review/synchronization the security team needs to do to check if there are any security fixes in the update, so they can properly tag the update as needing to go to -updates and prepare their security announce. They handle the task and sets its status as approppriate (Invalid or Fix Released). In case the task stays at confirmed without action for too much time, you should contact the security team for information.
  • regression-testing is the QA of the packages. This is testing done by the QA team, but depending on the package, other team may do the testing. Make sure testing is expected to be done by someone, that should be assigned to the task, and coordinate that with the kernel stable team. Once QA is done and no problems are found, the bug must be tagged as qa-testing-passed, or tagged with qa-testing-failed if there are issues, and the regression-testing task marked as fix released.
• Once all relevant testing tasks are done and passes, the stable bot marks the promote-to-{updates,security} tasks confirmed as necessary, and one archive admin must copy the packages from -proposed to -updates/-security. In case an archive admin misses it, enter in contact with one to have it copied. The archive admin must set the relevant task from confirmed to fix released as well. Once that is done, the stable bot marks the bug as fix released, and the update is done.

[1] New packages usually are done each 3 weeks (Kernel SRU Cadence - 1 in preparation phase, 1 week in verification phase, 1 week in testing phase), but there are exceptions some times, like regressions on current SRU which generates a new tracking bug, packages and a new master branch, so sometimes new updates can come before the next cadence cycle.

[2] You should receive email from launchpad when tracking bugs are opened through subscription or task assignment, if you lose it or don't receive for some reason, you can also use the Kernel SRU Workflow page to see what tracking bugs for the packages you maintain are opened.

[3] Usually, preparation phase ends up before the verification phase week, in this case, you can allow verification to go until the end of the next week which is the verification week, in practice this will be a bit more than 1 week. The same happens with Testing phase, if preparation and verification ends before the 3rd week, you can have more testing time.