Chromium in Ubuntu maintenance notes

chromium snap

The upstream chromium project has three channels of distribution for its Linux builds: stable, beta and dev. Current versions can be queried at https://omahaproxy.appspot.com/, and a JSON is obtainable at https://omahaproxy.appspot.com/all.json?os=linux.

These channels are mirrored to the stable, beta and edge channels in the snap store. Snaps are built on Launchpad, and the source of the packaging is versioned with git and stored on Launchpad as well.

Snaps targeting the dev and beta channels are published in the respective store channels right away after they're built. Snaps targeting the stable channel are first published in the candidate channel. From there they need to be tested and manually promoted to the stable channel.

Snap builds need to be requested manually, they don’t happen automatically when the branches are updated.

A script sends messages in the Mattermost's Browsers Maintenance channel whenever a new version is available in one of the channels of interest. Nathan runs it as a Cron job — this should be improved, probably with these scripts and github actions that send private telegram messages but whose code can be reused to issue other types of notifications.

Snaps are built for amd64, arm64 and armhf. Until recently they were also built for i386, but the switch from core18 to core20 as a base meant we couldn't do i386 builds any longer. Note that the builds on i386 had been broken for a while anyway, and the upstream project explicitly doesn't support this architecture.

The build process for the snap is intended to follow as closely as possible the upstream build process. In particular, it uses chromium's version of clang instead of an older version in the Ubuntu archive. On amd64, Google provides pre-built binaries that are downloaded during the build process, which makes the build rather fast. On armhf and arm64 there aren't pre-built binaries available unfortunately, so clang has to be built from source, which is time-consuming, hence a total build time often > 24 hours.

We do apply a number of patches to the upstream source tarball. These patches are either fixes to the build system to accomodate for a very specific build environment, or snap-specific changes to make the application behave better when running strictly confined. Most of these patches aren't relevant for the upstream project, so they haven't been submitted, but a few of them might be interesting.

The snap contains some simple tests that are run headless to help catch regressions early on in an automated setup. They cannot replace manual smoke testing to validate that a new major version is good for promotion to the stable channel. See the instructions to run the tests manually.

These tests are also being run automatically in the Canonical QA lab on armhf and arm64, for all new revisions in the beta and candidate channels. Test runs generate a report which is sent by e-mail to a private list of interested recipients.

During 2020, the chromium team at Google expressed some interest in building a Chrome snap, and we worked together with them to enable building snaps in their infrastructure. This was mostly working but the project was put on hold when their product managers withdrew their interest, arguing there was no demand for the snap. As a result there are some snap-specific build files in the upstream git repository, but they aren't up-to-date.

Snap-specific bugs are tracked in Launchpad under the chromium-browser source package, with the “snap” tag, and, by convention, their title is prefixed with “[snap]”.

chromium-ffmpeg snap

Documentation on the snapcraft forum.

Code is versioned on Launchpad and snaps are built on Launchpad as well.

The only known clients as of August 2021 are the opera family of snaps: opera, opera-beta, opera-developer.

Whenever a new major version of chromium with an incompatible version of ffmpeg is released, we need to add a new part (and the corresponding slot). See the history of the master branch for references. Building libffmpeg.so requires a full chromium source tarball, and as a consequence, a large amount of disk space.

It also requires custom patches to allow building against clang 10, which is the newest version available in bionic (because the snap has core18 as a base). Upgrading to core20 as a base would allow dropping some (all?) of these patches, but it would also mean no i386 builds anymore.

As of August 2021, the only known consumers of the chromium-ffmpeg snap were the opera family of snaps (opera, opera-beta, opera-developer), and they are only built for amd64, so maybe it’s acceptable to drop i386 support and upgrade to core20? Opera is still based on core18 as of August 2022, so this needs to be carefully tested to avoid binary incompatibilities.

The only reliable way of knowing the consumers of the chromium-ffmpeg snap is a query to the store database, which needs to be politely asked to the store team. There is unfortunately no faster/automated way.

[Archived] chromium-browser deb packages

Starting with Ubuntu 20.04, chromium is distributed as a snap only, and the chromium-browser package in the Ubuntu archive (universe) is a wrapper that installs the snap. The packaging for this wrapper lives in a bzr branch on Launchpad.

The chromium-browser package in Ubuntu 18.04 needs to be supported until June 2023. The Ubuntu Desktop Team is responsible for preparing and testing these updates for every new release of chromium, and handing them over to the Ubuntu Security Team who publishes the updates to bionic-security and bionic-updates.

Updates are being prepared in several PPAs, one for each upstream channel:

• dev: ppa:chromium-team/dev (code at lp:~chromium-team/chromium-browser/bionic-dev)

• beta: ppa:chromium-team/beta (code at lp:~chromium-team/chromium-browser/bionic-beta)

• stable: ppa:canonical-chromium-builds/stage (code at lp:~chromium-team/chromium-browser/bionic-stable)

Updates for the dev and beta channel are primarily meant for testing purposes, to ensure that we catch build problems with new versions early, before they reach the stable branch (given that we target an old release with an old toolchain, it happens frequently that we have to patch chromium to get it to build on all supported architectures). They are not published anywhere else, and we don’t encourage users to use those PPAs.

There are autopkgtests as well as a manual test plan that need to be run at least for every update in the stable channel, before the packages in the stage PPA are deemed ready and handed over to the security team.

Since the updates are prepared in PPAs, the autopkgtests need to be run against these PPAs.