IMPORTANT NOTE

Kernel SRU Workflow

The kernel release tracking bug is going to be changed to facilitate better communication between the responsible parties and clearer handoffs as the release progresses.

We are taking advantage of existing Launchpad capabilities. We will be using project series to represent individual workflow tasks. The meaning of status is overloaded to indicate the state of the task.

When a kernel release tracking bug is created, it is created against the relevant kernel source package and nominated for the related Ubuntu series. The new process will target the bug against an additional project, the "Kernel SRU Workflow" project and nominate it for all the series that are defined for that project.

The kernel release tracking bug can be identified on the kernel SRU report page at: https://kernel.ubuntu.com/reports/sru-report/

The "Kernel SRU Workflow" project has a number of custom "series" created for it that represent the different stages of the kernel cadence. A "series" represents a task to be accomplished by a team/person. The different tasks will be assigned to the team/person responsible for that stage. The assignee will set the status of the tasks they are working.

An automated script will run periodically to monitor the current state of the different tasks and change status when necessary. This script will be referred to below as the Workflow Mgr. The kernel team will develop this bot.

The Workflow:

1. The kernel team or Workflow Mgr. creates a tracking bug, all tasks will be set to their initial state (status: New) and be assigned to the appropriate team.

2. If there is an Upload-to-ppa task in the tracking bug:

   1. The assignee of Prepare-package task sets it to the in-progress state (status: In Progress). Then he/she prepares the branch in git repository for the necessary packages, and when release is closed in the branch, updates the version in the tracking bug title and description. If there is no abi change, the assignee must invalidate any Prepare-package-* tasks (status: Invalid). When all this is ready, the assignee changes Prepare-package to completed (status: Fix Released).

   2. Workflow Mgr. detects that the state of the Prepare-package task is now completed and changes the state of the Upload-to-ppa task to its ready-to-start state (status: Confirmed).

   3. The kernel team sets Upload-to-ppa task to the in-progress state (status: In Progress). If build of other dependent packages is needed (Prepare-package-* tasks not invalid), kernel team sets Prepare-package-* tasks to the in-progress state (status: In Progress).

3. If no Upload-to-ppa task is present:

   1. The kernel team sets Prepare-package task to the in-progress state (status: In Progress). If build of other dependent packages is needed, kernel team sets any required Prepare-package-* tasks to the in-progress state, or keeps/sets them to invalid if it isn't needed (status: In Progress or Invalid).

4. The kernel team builds and uploads all the necessary packages. For all Prepare-package-* tasks that are not Invalid and are In Progress, Workflow Mgr. detects when the package successfully builds on c-k-t ppa and is ready to be copied to proposed, then it changes their state to completed (status: Fix Released). When all Prepare-package-* tasks present are finished (status: Fix Released or Invalid), Workflow Mgr. detects when the main source package is built and ready to be copied to proposed, and when it happens then:

   1. If Upload-to-ppa is present, Workflow Mgr. sets it to completed (status: Fix Released).

   2. If no Upload-to-ppa task is present: Workflow Mgr. sets the Prepare-package task to completed (status: Fix Released).

5. When all required packages are built, after Workflow Mgr. changes the state of the Upload-to-ppa to completed, or if it isn't present and it changes the state of Prepare-package to completed, then Workflow Mgr. creates if needed new tracking bugs for derivative (topic branches for this package) and backport packages, and changes the Promote-to-proposed task to its ready-to-start state (status: Confirmed).

6. The kernel team or assignee of Prepare-package runs the script copy-proposed-kernel.py from lp:ubuntu-archive-tools for each source package uploaded, so the built packages are added to the upload queue used by archive admins to sync the packages to proposed.

7. An archive admin sets the Promote-to-proposed task to in-progress (status: In Progress) and syncs the packages from the queue to the proposed pocket in the archive. The task is reassigned to the individual working the task.

8. Once the packages have been copied, the archive admin (or exceptionally an stable release team member) sets the Promote-to-proposed task to completed (status: Fix Released).

9. Workflow Mgr. detects that the state of the Promote-to-proposed task is now completed. It may do additional checks if the packages are really in the archive and in the proper components. If it's completed and everything is ok, it changes the state of the Verification-testing task to the in-progress state (status: In Progress).

10. The kernel team tags all bugs needing verification with verification-needed-<series>, and adds a request for testing/verification.

11. Once all the bugs, requiring verification, listed in the changelog have been marked verification-done-<series>, the kernel team changes the state of the Verification-testing task to completed (status: Fix Released). Workflow Mgr. then detects this and sets the Certification-testing, Regression-testing and Security-signoff tasks to the ready-to-start state (status: Confirmed).

12. When the HW Certification team detects that the Certification-testing task is in the ready-to-start state (status: Confirmed) they may start certification testing. If the HW Certification team will not perform testing on this package, the Certification-testing task may be set to Invalid, but if they are testing then they change the tasks state to in-progress (status: In Progress). The task is reassigned to the individual working the task. (Note - the task may actually be set to Invalid while the task is still in the New state if desired)

13. When the QA team detects that the Regression-testing task is in the ready-to-start state (status: Confirmed) and they start testing, they change the tasks state to in-progress (status: In Progress). The task is reassigned to the individual working the task.

14. When the security team detects that the Security-signoff task is in the ready-to-start state (status: Confirmed), they change the tasks state to in-progress (status: In Progress). The task is reassigned to the individual working the task. They take care of any tasks they deem necessary prior to having an archive admin copy the release to the security pocket. If there are no CVEs, the security team sets the Security-signoff task to the not-needed state (status: Invalid). If there are CVEs in the release and the security team has signed-off on the release being promoted to the security pocket they change the status of the Security-signoff task to completed (status: Fix Released).

15. If certification testing is being performed, once certification testing completes, the HW certification team changes the state of the Certification-testing task to completed (status: Fix Released). If the testing was successful the certification team adds a certification-testing-passed tag otherwise they add a certification-testing-failed tag. These tags are not required if the Certification-testing task is set to Invalid.

16. Once regression testing completes, the QA team changes the state of the Regression-testing task to completed (status:Fix Released). If the testing was successful the QA team adds a qa-testing-passed tag, otherwise they add a qa-testing-failed tag.

17. When the Certification-testing and Regression-testing tasks have been set to completed states (status: Fix Released or optionally Invalid for certification) and both the certification-testing-passed and qa-testing-passed tags have been added by the appropriate team, and when the status of the Security-Signoff task has been set to either Invalid or Fix Released, the Workflow Mgr. is ready to change the state of the promote to update/security tasks to be started (status:Confirmed). However, Workflow Mgr. will only change these tasks if it not runs nearby an weekend (between 18:00 UTC Friday - 21:00 UTC Sunday), as there are less resources on weekends to fix anything if problems happen on packages copied to -updates/-security. Therefore, if promote to updates/security are ready to be started, and Workflow Mgr. isn't running near or on an weekend, it'll set Promote-to-updates task to be ready to be started (status:Confirmed), and do the same to the Promote-to-security task if Security-signoff task isn't invalid (status:Invalid). If Security-signoff is invalid, Promote-to-security is set to invalid as well.

18. If the Promote-to-updates task is set to the ready-to-start state (status: Confirmed), an stable release team member copies the packages from proposed to the updates pocket in the archive and sets the Promote-to-updates task to completed (status: Fix Released). The task is reassigned to the individual working the task.

19. If the Promote-to-security task is set to the ready-to-start state (status: Confirmed), an archive admin copies the packages to the security pocket and sets the state of the Promote-to-security task to completed (status: Fix Released). The task is reassigned to the individual working the task.

20. After all tasks are completed, Workflow Mgr. sets the main kernel-sru-workflow project task to completed (status: Fix Released), and sends all desired notifications about the new update.

Note: Some tasks can move from Confirmed straight to Fix Released depending on the amount of time/effort involved in the task.

An Example Tracking Bug

https://bugs.launchpad.net/kernel-sru-workflow/+bug/677021

Tasks

Status

Implementation Notes

In order to track some meta-information about the overall status of the tracking bug, the kernel-sru-workflow task is used by the software bot that automates process steps. The status of this task should never be changed manually unless the implications are well understood.

Status for this task maps as follows: