Ubuntu Wiki

To be carried out by: nominated stable release manager, with support from the stable release updates and release teams

Goals:

• Refresh hardware support in LTS releases for carefully-selected hardware
• Roll up accumulated stable updates into updated images to reduce download requirements for new deployments
• Maintain stability of existing installations
• Snapshot the archive for purposes of source retention

Note that, unlike the big releases, we don't push point release images to the CDN, so there's no need to coordinate this with IS. There is also a general expectation that all the images released as part of the point-release have a consistent set of packages.

Along with regular Ubuntu images, every point release is also the time when we refresh all of our Ubuntu Core images. Those follow a slightly different, although simpler, process documented on the UbuntuCore/ReleaseProcess page.

Between Release minus 6 months and Release minus 2 months:

1. Discuss candidates for new or improved hardware support with affected parties. Some sources for this work should be:
   • the OEM team
   • customers
   • the Ubuntu kernel team
   • the Ubuntu QA team
2. Establish a hit-list of bugs to fix in the point release using a milestone. Milestoning bugs is not a commitment to including the changes in the point release; they may be deferred after further information becomes available.
3. In concert with affected developers, triage the hit-list for feasibility.

Release minus 2 months:

1. Process stable release updates as normal. For hardware-enabling fixes, the package should be tested on the affected hardware prior to submitting to sign-off for -proposed.

2. Discuss the possibility of a Canonical press release for the point release with marketing.
3. Canvas OEM and Support organizations for any bugs important to try to get fixed.

Release minus 6 weeks:

1. Create a release tracking Discourse thread for the release in mention. This should include:
   • On top, a short summary of the current state of the release
   • Sections for release blocker and upgrade blocker bugs
   • Section for bugs to watch for
   • Copy of the release checklist as actual checklist items

2. Coordinate with the kernel team to ensure HWE kernels are updated to the new target release (Kernel/RollingLTSEnablementStack#Update_Schedule-2).

Release minus 1 month:

1. In coordination with QA, verify that all candidate bugs are fixed.

2. If the kernel or associated modules have been changed, upload debian-installer after all the binaries are in place. If the ABI changed, make sure to take account of this throughout debian-installer/build/config/ and in the installer seed for all flavours being built.

   • Before pushing the debian-installer packages to the archive, do a test build in a PPA to make sure no size bumps are needed.

   • This step is no longer needed for series focal and newer as debian-installer is no longer used in any installation media.

3. If this point release includes the switch to enable the HWE stack (usually XX.XX.2):

   • Make sure that the variant list in livecd-rootfs's live-build/ubuntu-server/hooks/033-kernel-bits.binary hook includes both ga and hwe to make sure subiquity offers both kernel flavors. Upload and fast-forward into -updates in case it doesn't.

   • Make sure that the server-ship-live seed includes the new hwe kernel.

   • Check and mark livecd-rootfs auto/config LB_KERNEL_FLAVOURS to the -hwe-XX.XX variant for every desktop flavor participating.

4. Change cdimage/lib/cdimage/config.py and debian-cd/CONF.sh to use the new release version number.

5. Make sure that that we are building the daily images with -proposed enabled.
6. Update the manifest to reflect publishing status of images based on input from product leads.
7. Build CD images (which will be published on cdimage.ubuntu.com) and smoke-test in some convenient environment to check for obvious failures.
8. Notify translations-team to prepare updates for point release.
9. Liaise with QA, community, certification and all involved teams to arrange for testing resources.
   • We need to make sure that we have QA coverage for every flavor (desktop, server, pi, riscv, kubuntu etc.)

Release minus 3 weeks:

1. Include the latest translation updates into the sources for the point release.
2. Contact IS, QA, and/or certification as appropriate to request testing for hardware recertification.
   • Make sure to get early feedback on certification testing from the daily images.
3. Check to see if python-apt has been uploaded recently. If not, upload a new version.
4. Check to see if ubuntu-release-upgrader have been uploaded recently. If not, upload a new version after running pre-build.sh as that generates the updated lists of mirrors.
5. Ensure that the stable/ubuntu-XX.YY.Z branch has been published for the subiquity snap.

Release minus 14 days:

1. Prepare change summary and release announcement

   • Script to use for preparing the change summary: http://people.canonical.com/~vorlon/release-updates.py

   • Sort and/or re-divide updates into rough categories (see previous summaries)
   • Remove administrative-only bugs (e.g. kernel release tracking)
   • Try to reduce each entry to just a description of the change; remove redundant bug references and information about where the change came from (people can go to the raw changelog if they care)
   • This can involve substantial amounts of editing; make sure you have a good editor and/or a clear grasp of regular expressions
   • If this point release includes an HWE stack, communicate the life cycle of this HWE stack
   • If this is the final point release for the distroseries (because a new LTS will be released soon), include a reminder of this and of the support cycle/EOL
2. Make sure all critical package updates are by now done and landed in -updates or -security.

3. Check https://people.canonical.com/~ubuntu-archive/phased-updates.html for any packages in -updates which have stopped phasing, or are still phasing, and decide whether to fully phase them or revert to the previous version of the package.

4. Disable -proposed for the series daily image builds (to build images as close to the final product as possible).
5. Ask Certification and QA to start performing preliminary image testing on the daily images as much as possible from now till release, identifying any release critical bugs that have not been noticed earlier.
6. Coordinate with the SRU team to be more careful on which packages get promoted to -updates, keeping the incoming point-release in mind.
   • Keep in mind that due to the fact that even release-critical bugs require an aging and verification period as per the usual SRU policy, this is the final 'safe' moment whenever release facing updates can be accepted without risking the delay of the point release.

Release minus 7 days:

1. Notify web team of the upcoming point release.
   • Determine who from the team the publishing contact will be.
   • Include summary list of actual file names of ISOs that will make up releases.
   • Include detailed information about which image file names will change on the mirrors for the point release, and which ones will not.
   • Discuss release stability and handoffs on release date.

2. Upload a new base-files package to -proposed to bump the lsb_release description (example for 8.04.4). Do not change the DISTRIB_RELEASE value, which is used programmatically by third-party software. If the etc/os-release file exists, update VERSION and PRETTY_NAME, don't update VERSION_ID.

3. Push base-files through to -updates and ensure that it is fully phased.

4. Once testing is verified to be complete, move packages to -updates. This should vary depending on how much testing of the daily -proposed images has been done - ideally we want to flush the whole -proposed pocket to not invalidate the earlier testing.

   • If anything is moved to -updates and on installation media also ensure to fully phase it.
   • Make sure debian-installer has picked up all the installer related changes and doesn't require a re-build.
5. Analyze the package diffs between last point-release and the current daily image for each participating flavour (stripping versions) to make sure no changes are pulling in a bunch of new/unexplained packages.
6. Turn off cron jobs that will auto update into -updates until final images are tested.
7. Send out an e-mail notifying developers of the -updates pocket freeze for the duration of the point-release.
8. Build candidate release images and populate into iso tracker (double check that those images are NOT building with -proposed enabled). Re-spin when appropriate.

   • Before running builds, make sure cdimage is up-to-date with archive contents and run anonftpsync on ancientminister.

   • Also remember about building source images along with those (e.g. DIST=focal cron.source), re-spinning when appropriate.

   • Remember about building core images as well (as per UbuntuCore/ReleaseProcess).

9. Create a snapshot of the archive:

   ubuntu-archive@snakefruit:~$ point-release-snapshot focal focal.3-security-updates-snapshot

   which will copy the relevant indices to a subdirectory of ~ubuntu-archive/point-releases/. (Packages may be retrieved from the Launchpad librarian if required.)

   • Remember to re-create the snapshot whenever new candidate images are built.

Release minus 3 hours:

1. Check with IS whether the previous point release needs to be moved off before prepublishing due to mirror space constraints.
2. If this point-release is an LTS with an OEM stack, contact the Certification Team for a formal sign off of the images.

3. Pre-publish images: ./publish-image-set --prepublish will print the necessary commands.

4. Copy .manifest to .manifest.full, and prune all images from previous releases from the .manifest file to allow timely mirror probing.

5. Run sync-mirrors on ancientminister to push out the pre-published file structure.

Release:

1. Release images as final, and move the previous images to old-releases.ubuntu.com:

   • Find which previous images on cdimage.ubuntu.com for this release are going to be replaced by this image set, and archive them to old-images (using a similar procedure to that in EndOfLifeProcess for moving images from cdimage.ubuntu.com, but leave /srv/cdimage.ubuntu.com/www/full/netboot/RELEASE/ alone).

   • Publish images: ./publish-image-set.py will print the necessary commands.

   • Update version numbers in cdimage/www/simple/HEADER.html and cdimage/www/simple/.htaccess.

   • Archive releases.ubuntu.com images from the previous point release to old-releases by running archive-point-release.

   • Copy .manifest to .manifest.full again, pruning all images from previous releases from the .manifest file to allow timely mirror probing.

   • Run sync-mirrors on ancientminister to push out the published file structure.

     • If you moved images to old-releases, remember to run sync-old-releases as well.

   • Notify Web team to update the iso URLs on the ubuntu.com website

   • Remember to publish core images as well (as per UbuntuCore/ReleaseProcess).

2. Ping bdmurray to update meta-release file on http://changelogs.ubuntu.com/

3. Regenerate the Raspi Installer JSON file to include the latest images and checksums (ping waveform or sil2100).

4. Work with release and web publishing team to monitor mirror pickup
   • verify download from ubuntu.com
   • check that the torrents are working
   • verify download from one of the mirrors
5. Send the announcement mail (see archives for past examples)
   • ubuntu-announce
   • Update Discourse Tracking bug
6. Notify press release team if needed

Release +1 day:

1. Restore the .manifest.full file on releases.ubuntu.com
2. Deactivate the just released "point release" milestone target
3. Re-enable -proposed for daily builds on cdimage
4. Re-enable daily builds
5. Send out update to people running previous LTS (after migration testing completed).
6. Gather feedback info for future improvements to process

CategoryProcess