kernel-sru-workflow

Differences between revisions 65 and 67 (spanning 2 versions)

IMPORTANT NOTE

The workflow processes described on this page are embodied in workflow tools which perform automated updates to process tasks.

No changes to the processes on this page should be made without discussion among the teams affected, including the Release Manager, Stable Kernel Team, Security Team, QA Team, and Certification Team.

Kernel SRU Workflow

The kernel release tracking bug is going to be changed to facilitate better communication between the responsible parties and clearer handoffs as the release progresses.

We are taking advantage of existing Launchpad capabilities. We will be using project series to represent individual workflow tasks. The meaning of status is overloaded to indicate the state of the task.

When a kernel release tracking bug is created, it is created against the relevant kernel source package and nominated for the related Ubuntu series. The new process will target the bug against an additional project, the "Kernel SRU Workflow" project and nominate it for all the series that are defined for that project.

The kernel release tracking bug can be identified on the kernel SRU report page at: http://people.canonical.com/~kernel/reports/sru-report.html

The "Kernel SRU Workflow" project has a number of custom "series" created for it that represent the different stages of the kernel cadence. A "series" represents a task to be accomplished by a team/person. The different tasks will be assigned to the team/person responsible for that stage. The assignee will set the status of the tasks they are working.

An automated script will run periodically to monitor the current state of the different tasks and change status when necessary. This script will be referred to below as the Workflow Mgr. The kernel team will develop this bot.

The Workflow:

The kernel team or Workflow Mgr. creates a tracking bug, all tasks will be set to their initial state (status: New) and be assigned to the appropriate team.
If there is an Upload-to-ppa task in the tracking bug:
1. The assignee of Prepare-package task sets it to the in-progress state (status: In Progress). Then he/she prepares the branch in git repository for the necessary packages, and when release is closed in the branch, updates the version in the tracking bug title and description. If there is no abi change, the assignee must invalidate any Prepare-package-* tasks (status: Invalid). When all this is ready, the assignee changes Prepare-package to completed (status: Fix Released).
2. Workflow Mgr. detects that the state of the Prepare-package task is now completed and changes the state of the Upload-to-ppa task to its ready-to-start state (status: Confirmed).
3. The kernel team sets Upload-to-ppa task to the in-progress state (status: In Progress). If build of other dependent packages is needed (Prepare-package-* tasks not invalid), kernel team sets Prepare-package-* tasks to the in-progress state (status: In Progress).
If no Upload-to-ppa task is present:
1. The kernel team sets Prepare-package task to the in-progress state (status: In Progress). If build of other dependent packages is needed, kernel team sets any required Prepare-package-* tasks to the in-progress state, or keeps/sets them to invalid if it isn't needed (status: In Progress or Invalid).
The kernel team builds and uploads all the necessary packages. For all Prepare-package-* tasks that are not Invalid and are In Progress, Workflow Mgr. detects when the package successfully builds on c-k-t ppa and is ready to be copied to proposed, then it changes their state to completed (status: Fix Released). When all Prepare-package-* tasks present are finished (status: Fix Released or Invalid), Workflow Mgr. detects when the main source package is built and ready to be copied to proposed, and when it happens then:
1. If Upload-to-ppa is present, Workflow Mgr. sets it to completed (status: Fix Released).
2. If no Upload-to-ppa task is present: Workflow Mgr. sets the Prepare-package task is changed to completed (status: Fix Released).
When all required packages are built, after Workflow Mgr. changes the state of the Upload-to-ppa to completed, or if it isn't present and it changes the state of Prepare-package to completed, then Workflow Mgr. creates if needed new tracking bugs for derivative (topic branches for this package) and backport packages, and changes the Promote-to-proposed task to its ready-to-start state (status: Confirmed).
The kernel team or assignee of Prepare-package runs the script copy-proposed-kernel.py from lp:ubuntu-archive-tools for each source package uploaded, so the built packages are added to the upload queue used by archive admins to sync the packages to proposed.
An archive admin sets the Promote-to-proposed task to in-progress (status: In Progress) and syncs the packages from the queue to the proposed pocket in the archive. The task is reassigned to the individual working the task.
Once the packages have been copied a stable release team member sets the Promote-to-proposed task to completed (status: Fix Released).
Workflow Mgr. detects that the state of the Promote-to-proposed task is now completed and changes the state of the Verification-testing task to the in-progress state (status: In Progress). (Affected bugs are marked for verification needed)
Once all the bugs, requiring verification, listed in the changelog have been marked verification-done, the Workflow Mgr. changes the state of the Verification-testing task to completed (status: Fix Released) and sets the Certification-testing, Regression-testing and Security-signoff tasks to the ready-to-start state (status: Confirmed).
When the HW Certification team detects that the Certification-testing task is in the ready-to-start state (status: Confirmed) they may start certification testing. If the HW Certification team will not perform testing on this package, the Certification-testing task may be set to Invalid, but if they are testing then they change the tasks state to in-progress (status: In Progress). The task is reassigned to the individual working the task. (Note - the task may actually be set to Invalid while the task is still in the New state if desired)
When the QA team detects that the Regression-testing task is in the ready-to-start state (status: Confirmed) and they start testing, they change the tasks state to in-progress (status: In Progress). The task is reassigned to the individual working the task.
When the security team detects that the Security-signoff task is in the ready-to-start state (status: Confirmed), they change the tasks state to in-progress (status: In Progress). The task is reassigned to the individual working the task. They take care of any tasks they deem necessary prior to having an archive admin copy the release to the security pocket. If there are no CVEs, the security team sets the Security-signoff task to the not-needed state (status: Invalid). If there are CVEs in the release and the security team has signed-off on the release being promoted to the security pocket (once all testing passes successfully) they change the status of the Security-signoff task to completed (status: Fix Released).
If certification testing is being performed, once certification testing completes, the HW certification team changes the state of the Certification-testing task to completed (status: Fix Released). If the testing was successful the certification team adds a certification-testing-passed tag otherwise they add a certification-testing-failed tag. These tags are not required if the Certification-testing task is set to Invalid.
Once regression testing completes, the QA team changes the state of the Regression-testing task to completed (status:Fix Released). If the testing was successful the QA team adds a qa-testing-passed tag, otherwise they add a qa-testing-failed tag.
When the Certification-testing and Regression-testing tasks have been set to completed states (status: Fix Released or optionally Invalid for certification) and both the certification-testing-passed and qa-testing-passed tags have been added by the appropriate team, and when the status of the Security-Signoff task has been set to either Invalid or Fix Released, the Workflow Mgr. is ready to change the state of the promote to update/security tasks to be started (status:Confirmed). However, Workflow Mgr. will only change these tasks if it not runs nearby an weekend (between 18:00 UTC Friday - 21:00 UTC Sunday), as there are less resources on weekends to fix anything if problems happen on packages copied to -updates/-security. Therefore, if promote to updates/security are ready to be started, and Workflow Mgr. isn't running near or on an weekend, it'll set Promote-to-updates task to be ready to be started (status:Confirmed), and do the same to the Promote-to-security task if Security-signoff task isn't invalid (status:Invalid). If Security-signoff is invalid, Promote-to-security is set invalid as well.
A stable release team member copies the packages from proposed to the updates pocket in the archive and sets the Promote-to-updates task to completed (status: Fix Released). The task is reassigned to the individual working the task.
If the Promote-to-security task is set to the ready-to-start state(status: Confirmed), an archive admin copies the packages to the security pocket and sets the state of the Promote-to-security task to completed (status: Fix Released).

Note: Some tasks can move from Confirmed straight to Fix Released depending on the amount of time/effort involved in the task.

An Example Tracking Bug

https://bugs.launchpad.net/kernel-sru-workflow/+bug/677021

Tasks

Series	Owner	Description
Prepare-package	Assigned maintainer of the package branch	The assignee has prepared or uploaded the source package for the release to the kernel team's ppa.
Upload-to-ppa	Kernel Team	The kernel team has uploaded the source package for the release to the kernel team's ppa.
Prepare-package-lbm	Kernel Team	The kernel team has uploaded the linux-backport-modules source package for the release to the kernel team's ppa.
Prepare-package-lrm	Kernel Team	The kernel team has uploaded the linux-restricted-modules source package for the release to the kernel team's ppa.
Prepare-package-lum	Kernel Team	The kernel team has uploaded the linux-ubuntu-modules source package for the release to the kernel team's ppa.
Prepare-package-meta	Kernel Team	The kernel team has uploaded the linux-*-meta source package for the release to the kernel team's ppa.
Prepare-package-ports-meta	Kernel Team	The kernel team has uploaded the linux-*-ports-meta source package for the release to the kernel team's ppa.
Promote-to-proposed	SRU Team	The package in the kernel team's ppa is copied to the proposed pocket in the archive.
Verification-testing	Kernel Team	The bugs related to the release are being verified as having been fixed by the appropriate community member.
Certification-testing	HW Certification Team	The kernel in proposed is tested via the certification tests.
Regression-testing	QA Team	The kernel in proposed is tested for regressions.
Security-signoff	Security Team	The security team does any validation they deem necessary and declares if Promote-to-security is needed
Promote-to-updates	SRU Team	The package is copied from the proposed pocket to the updates pocket in the archive.
Promote-to-security	SRU Team	The package is copied from the proposed pocket to the security pocket in the archive, if needed.

Status

Status	Description
New	The initial state of the task. This is not ready for the assigned team/person to begin working on that task.
Confirmed	This indicates that the prerequisites for the task to begin have been met and the task is ready to be started.
In Progress	The assigned team/person has begun the work associated with the given task.
Invalid	The process state is not appropriate for the given kernel release. The individual that changed the state to Invalid should reassign the task to themselves. Examples of where this would be used are: When there are no CVEs fixed in the release and it is not necessary to copy the release.
Fix Released	The assigned team/person has finished the task.

Implementation Notes

In order to track some meta-information about the overall status of the tracking bug, the kernel-sru-workflow task is used by the software bot that automates process steps. The status of this task should never be changed manually unless the implications are well understood.

Status for this task maps as follows:

Status	Description
New	Just created.
In Progress	Actively being processed by Workflow Mgr. This is the only state in which Workflow Mgr will make changes to the bug.
Incomplete	Error, exception, or invalid condition - Processing by Workflow Mgr is suspended.
Invalid	The bug has been replaced by a new tracking bug.
Fix Released	All good and we're done.

Kernel/kernel-sru-workflow (last edited 2023-12-06 20:01:36 by setuid)

-  ⇤ ← Revision 65 as of 2011-09-29 14:21:48 → 
  Size: 14720
  Editor: 188-223-22-218
  Comment:
+   ← Revision 67 as of 2012-07-30 14:17:52 → ⇥
  Size: 15500
  Editor: 189
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 54:
-. The kernel team builds and uploads all the necessary packages. For all '''Prepare-package-*''' tasks that are not invalid, when the package successfully builds and is ready to be copied to proposed, their state is changed to completed (status: '''Fix Released'''). When all '''Prepare-package-*''' tasks present are finished (status: '''Fix Released''' or '''Invalid'''), and the main source package is built and ready to be copied to proposed, then:
+. The kernel team builds and uploads all the necessary packages. For all '''Prepare-package-*''' tasks that are not Invalid and are In Progress, '''Workflow Mgr.''' detects when the package successfully builds on c-k-t ppa and is ready to be copied to proposed, then it changes their state to completed (status: '''Fix Released'''). When all '''Prepare-package-*''' tasks present are finished (status: '''Fix Released''' or '''Invalid'''), '''Workflow Mgr.''' detects when the main source package is built and ready to be copied to proposed, and when it happens then:
 Line 56:
-. If '''Upload-to-ppa''' is present, it's set to completed (status: '''Fix Released''').
+. If '''Upload-to-ppa''' is present, '''Workflow Mgr.''' sets it to completed (status: '''Fix Released''').
 Line 58:
-. If no '''Upload-to-ppa''' task is present: the main '''Prepare-package''' task is changed to completed (status: '''Fix Released''').
+. If no '''Upload-to-ppa''' task is present: '''Workflow Mgr.''' sets the '''Prepare-package''' task is changed to completed (status: '''Fix Released''').
 Line 60:
-. '''Workflow Mgr.''' detects that the state of the '''Upload-to-ppa''' has completed. If '''Upload-to-ppa''' task isn't present, then it detects that the state of '''Prepare-package''' has completed. If the required task is completed, it creates if needed new tracking bugs for derivative (topic branches for this package) and backport packages, and changes the '''Promote-to-proposed''' task to its ready-to-start state (status: '''Confirmed''').
+. When all required packages are built, after '''Workflow Mgr.''' changes the state of the '''Upload-to-ppa''' to completed, or if it isn't present and it changes the state of '''Prepare-package''' to completed, then '''Workflow Mgr.''' creates if needed new tracking bugs for derivative (topic branches for this package) and backport packages, and changes the '''Promote-to-proposed''' task to its ready-to-start state (status: '''Confirmed''').
 Line 62:
-. An archive admin sets the '''Promote-to-proposed''' task to in-progress (status: '''In Progress''') and copies the packages to the '''proposed''' pocket in the archive. The task is reassigned to the individual working the task.
+. The kernel team or assignee of '''Prepare-package''' runs the script copy-proposed-kernel.py from lp:ubuntu-archive-tools for each source package uploaded, so the built packages are added to the upload queue used by archive admins to sync the packages to proposed.

 1. An archive admin sets the '''Promote-to-proposed''' task to in-progress (status: '''In Progress''') and syncs the packages from the queue to the '''proposed''' pocket in the archive. The task is reassigned to the individual working the task.
-Line 76:
+Line 78:
-. '''Workflow Mgr.''' detects that the state of the '''Security-signoff''' task has been completed and changes the '''Promote-to-security''' task to its ready-to-start state (status: '''Confirmed''') if required ('''Security-signoff''' is '''Fix Released'''), or to '''Invalid''' if not required ('''Security-signoff''' is '''Invalid''').
 Line 82:
-. When the '''Certification-testing''' and '''Regression-testing''' tasks have been set to completed states (status: '''Fix Released''' or optionally '''Invalid''' for certification) and both the '''certification-testing-passed''' and '''qa-testing-passed''' tags have been added by the appropriate team, and when the status of the '''Security-Signoff''' task has been set to either '''Invalid''' or '''Fix Released''', the '''Workflow Mgr.''' changes the state of the '''Promote-to-updates''' task to the ready-to-start state (status:'''Confirmed''').
+. When the '''Certification-testing''' and '''Regression-testing''' tasks have been set to completed states (status: '''Fix Released''' or optionally '''Invalid''' for certification) and both the '''certification-testing-passed''' and '''qa-testing-passed''' tags have been added by the appropriate team, and when the status of the '''Security-Signoff''' task has been set to either '''Invalid''' or '''Fix Released''', the '''Workflow Mgr.''' is ready to change the state of the promote to update/security tasks to be started (status:'''Confirmed'''). However, '''Workflow Mgr.''' will only change these tasks if it not runs nearby an weekend (between 18:00 UTC Friday - 21:00 UTC Sunday), as there are less resources on weekends to fix anything if problems happen on packages copied to -updates/-security. Therefore, if promote to updates/security are ready to be started, and '''Workflow Mgr.''' isn't running near or on an weekend, it'll set '''Promote-to-updates''' task to be ready to be started (status:'''Confirmed'''), and do the same to the '''Promote-to-security''' task if '''Security-signoff''' task isn't invalid (status:'''Invalid'''). If '''Security-signoff''' is invalid, '''Promote-to-security''' is set invalid as well.

Ubuntu Wiki