UbuntuCore/CoreSnapAutomation

Some of the essential snaps are handled by an automated build and promotion infrastructure. This wiki page is an overview on how this automation works and how it can be managed.

Overview

There is two types of automation for essential snaps: automated builds and automated promotion:

All official gadget snaps have LP recipes that automatically re-build the snap on git branch changes, uploading the resulting snap to edge. The base (core18, core20, core22...) snaps have a bit more sophisticated build automation, with the core-builder scripts handling that automation.
All official essential snaps (base snaps cut core snap and gadgets) use the core-promotion automation to perform automated snap promotions from channel to channel based on testing policy defined for every channel.
core snaps (UC16 bases which include snapd) are special. core-builder and core-promoter are not used for them. Instead, a snap created from https://github.com/snapcore/snapd-testing/tree/master/snaps/core-builder is run daily in a cloud instance to trigger daily builds that are published in the edge channel. These snaps are not promoted, instead snaps in beta are built from https://launchpad.net/~snappy-dev/+snap/core-beta/. From there, they are promoted if beta validation tests pass, and then moved to candidate if certification tests pass.

In overall, the build automation always makes sure that the given snaps are automatically built in the edge channel if required and promotion automation makes sure those snaps are promoted from edge to beta, from beta to candidate etc. whenever the required automated testing is performed.

Both those scripts are running periodically as github actions for https://github.com/canonical/core-snap-automation (accessible only for Canonical employees).

core-builder

The core-builder script is only used for core snaps (core16, core18, core20). The script rebuilds the snap using the LP recipe whenever either the snap git branch changes, any packages included in the ubuntu-base tarball used for building have new versions or some extra snaps change in the archive (or the image PPA).

core-promoter

The core-promoter script checks for new snaps in each managed channel (different per snap). If a new version is present, it then runs selected policy checks per channel and if all of those pass, it promotes it to the next defined channel. Those policy check usually involve checking respective trello boards (snapd or certification) for test results, per the established process.

All gadget snaps, once landed in candidate, now age for a selected period of time and then get auto-promoted to stable (if not disabled for some reason). The core* snaps however require manual intervention once in candidate (manual promotion to stable), because they cannot be released without coordination with the snap store.

Testing boards:

Edge testing (core* snaps): https://github.com/snapcore/snapd-testing-jobs/actions/workflows/edge.yaml
Beta testing (certification, needs VPN): https://test-observer.canonical.com/

Deployment

The build and promotion scripts are running on a PS5 juju instance, hosted on the foundations-bastion-ps5 bastion. The environment is called prod-core-snap-automation. Everyone from the https://launchpad.net/~canonical-foundations/ team have access + a few people from the Ubuntu Core team.

For most changes, no involvement in the scripting or configuration is required. The build and promotion actions usually only prepare the script branch and run the respective script with selected parameters - everything else is done from the actual build/promotion scripting. The build and promotion scripts are executed via a crobtab, defined in /etc/cron.d/core-snap-automation, currently running every 4 hours.