Differences between revisions 82 and 83
Revision 82 as of 2019-08-02 11:35:14
Size: 10135
Editor: sil2100
Comment: Add the core release process to our regular point release process
Revision 83 as of 2020-02-13 13:42:16
Size: 10226
Editor: sil2100
Deletions are marked like this. Additions are marked like this.
Line 96: Line 96:
  * Run sync-mirrors on nusakan to push out the published file structure.   * Run `sync-mirrors` on nusakan to push out the published file structure.
    * If you moved images to old-releases, remember to run `sync-old-releases` as well.

To be carried out by: nominated stable release manager, with support from the stable release updates and release teams


  • Refresh hardware support in LTS releases for carefully-selected hardware
  • Roll up accumulated stable updates into updated images to reduce download requirements for new deployments
  • Maintain stability of existing installations
  • Snapshot the archive for purposes of source retention

Note that, unlike the big releases, we don't push point release images to the CDN, so there's no need to coordinate this with IS. There is also a general expectation that all the images released as part of the point-release have a consistent set of packages.

Along with regular Ubuntu images, every point release is also the time when we refresh all of our Ubuntu Core images. Those follow a slightly different, although simpler, process documented on the UbuntuCore/ReleaseProcess page.

Between Release minus 6 months and Release minus 2 months:

  1. Discuss candidates for new or improved hardware support with affected parties. Some sources for this work should be:
    • the Canonical support team (via Steve George)
    • customers
    • the Ubuntu kernel team
    • the Ubuntu QA team
  2. Establish a hit-list of bugs to fix in the point release using a milestone. Milestoning bugs is not a commitment to including the changes in the point release; they may be deferred after further information becomes available.
  3. In concert with affected developers, triage the hit-list for feasibility.

Release minus 2 months:

  1. Process stable release updates as normal. For hardware-enabling fixes, the package should be tested on the affected hardware prior to submitting to sign-off for -proposed.

  2. Discuss the possibility of a Canonical press release for the point release with marketing.
  3. Liaise with IS, QA, community and certification to arrange for testing resources.
  4. Canvas OEM and Support organizations for any bugs important to try to get fixed.

Release minus 6 weeks:

  1. Coordinate with the kernel team to ensure HWE kernels are updated to the new target release (Kernel/RollingLTSEnablementStack#Update_Schedule-2).

Release minus 1 month:

  1. In coordination with QA, verify that all candidate bugs are fixed.
  2. If the kernel or associated modules have been changed, upload debian-installer after all the binaries are in place. If the ABI changed, make sure to take account of this throughout debian-installer/build/config/ and in the installer seed for all flavours being built.

    • Before pushing the debian-installer packages to the archive, do a test build in a PPA to make sure no size bumps are needed.
  3. Change cdimage/lib/cdimage/config.py and debian-cd/CONF.sh to use the new release version number.

  4. Make sure that that we are building the daily images with -proposed enabled.
  5. Update the manifest to reflect publishing status of images based on input from product leads.
  6. Build CD images (which will be published on cdimage.ubuntu.com) and smoke-test in some convenient environment to check for obvious failures.
  7. Notify translations-team to prepare updates for point release.

Release minus 3 weeks:

  1. Include the latest translation updates into the sources for the point release.
  2. Contact IS, QA, and/or certification as appropriate to request testing for hardware recertification.
  3. Iterate CD images as needed based on testing feedback, in coordination with the kernel team.
  4. Check to see if python-apt and ubuntu-release-upgrader have been uploaded recently. If not, upload a new version after running pre-build.sh as that generates updated lists of mirrors.
  5. Ensure that the stable/ubuntu-XX.YY.Z branch has been published for the subiquity snap.

Release minus 14 days:

  1. Prepare change summary and release announcement
    • Script to use for preparing the change summary: http://people.canonical.com/~vorlon/release-updates.py

    • Sort and/or re-divide updates into rough categories (see previous summaries)
    • Remove administrative-only bugs (e.g. kernel release tracking)
    • Try to reduce each entry to just a description of the change; remove redundant bug references and information about where the change came from (people can go to the raw changelog if they care)
    • This can involve substantial amounts of editing; make sure you have a good editor and/or a clear grasp of regular expressions
    • If this point release includes an HWE stack, communicate the life cycle of this HWE stack
    • If this is the final point release for the distroseries (because a new LTS will be released soon), include a reminder of this and of the support cycle/EOL

Release minus 7 days:

  1. Upload a new base-files package to -proposed to bump the lsb_release description (example for 8.04.4). Do not change the DISTRIB_RELEASE value, which is used programmatically by third-party software. If the etc/os-release file exists, update VERSION and PRETTY_NAME, don't update VERSION_ID.

  2. Push base-files through to -updates.

  3. Once testing is verified to be complete, move packages to -updates. This should vary depending on how much testing of the daily -proposed images has been done - ideally we want to flush the whole -proposed pocket to not invalidate the earlier testing.

    • Make sure debian-installer has picked up all the installer related changes and doesn't require a re-build.
  4. Analyze the package diffs between last point-release and the current daily image for each participating flavour (stripping versions) to make sure no changes are pulling in a bunch of new/unexplained packages.
  5. Turn off cron jobs that will auto update into -updates until final images are tested.
  6. Build candidate release images and populate into iso tracker. Make sure that those images are NOT building with -proposed enabled. Re-spin when appropriate.
    • Before running builds, make sure cdimage is up-to-date with archive contents and run anonftpsync on nusakan.

    • Also remember about building source images along with those (e.g. DIST=xenial cron.source), re-spinning when appropriate.

    • Remember about building core images as well (as per UbuntuCore/ReleaseProcess).

  7. Create a snapshot of the archive:

    ubuntu-archive@snakefruit:~$ point-release-snapshot lucid lucid.3-security-updates-snapshot

    which will copy the relevant indices to a subdirectory of ~ubuntu-archive/point-releases/. (Packages may be retrieved from the Launchpad librarian if required.)

    • Remember to re-create the snapshot whenever new candidate images are built.
  8. Notify OEM team (irc: KyleN) to build found license list for main.
  9. Notify web team of the upcoming point release.
    • Determine who from the team the publishing contact will be.
    • Include summary list of actual file names of ISOs that will make up releases.
    • Include detailed information about which image file names will change on the mirrors for the point release, and which ones will not.
    • Discuss release stability and handoffs on release date.

Release minus 3 hours:

  1. Check with James Troup whether the previous point release needs to be moved off before prepublishing due to mirror space constraints.
  2. Pre-publish images: ./publish-image-set --prepublish will print the necessary commands.

  3. Copy .manifest to .manifest.full, and prune all images from previous releases from the .manifest file to allow timely mirror probing.
  4. Run sync-mirrors on nusakan to push out the pre-published file structure.


  1. Release images as final, and move the previous images to old-releases.ubuntu.com:

    • Archive the old copy of wubi.exe to old-releases, and publish a new one (do this first so that its timestamp will be newer than that of the checksum files so that the checksums will be updated properly)
    • Find which previous images on cdimage.ubuntu.com for this release are going to be replaced by this image set, and archive them to old-releases (using a similar procedure to that in EndOfLifeProcess for moving images from cdimage.ubuntu.com to old-images, but leave /srv/cdimage.ubuntu.com/www/full/netboot/RELEASE/ alone).

    • Publish images: ./publish-image-set.py will print the necessary commands.

    • Update version numbers in cdimage/www/simple/HEADER.html and cdimage/www/simple/.htaccess.

    • Archive releases.ubuntu.com images from the previous point release to old-releases (using a similar procedure to that in EndOfLifeProcess, but being more selective about which files to move).

    • Copy .manifest to .manifest.full again, pruning all images from previous releases from the .manifest file to allow timely mirror probing.
    • Run sync-mirrors on nusakan to push out the published file structure.

      • If you moved images to old-releases, remember to run sync-old-releases as well.

    • Notify Web team to update the iso URLs on the ubuntu.com website
    • Remember to publish core images as well (as per UbuntuCore/ReleaseProcess).

  2. Ping bdmurray to update meta-release file on http://changelogs.ubuntu.com/

  3. Work with release and web publishing team to monitor mirror pickup
    • verify download from ubuntu.com
    • check that the torrents are working
    • verify download from one of the mirrors
  4. Send the announcement mail (see archives for past examples)
    • ubuntu-announce
    • ubuntu-release
  5. Notify press release team if needed

Release +1 day:

  1. Restore the .manifest.full file on releases.ubuntu.com
  2. Deactivate the just released "point release" milestone target
  3. Re-enable -proposed for daily builds on cdimage
  4. Re-enable daily builds
  5. Send out update to people running previous LTS (after migration testing completed).
  6. Get final summary of updates for release (see precise-updates.py script), and publish to ???
  7. Gather feedback info for future improvements to process


PointReleaseProcess (last edited 2020-02-13 13:42:16 by sil2100)