Updating rustc and cargo in ubuntu

Scope

We maintain rustc and cargo in ubuntu almost entirely to support building firefox. If you're a rust developer looking to work on your own rust programs, you are probably going to have a better experience (for now, at least) using rustup.

Generalities

The maintenance uses git-ubuntu and the git merge workflow both for merging upstream updates from debian and maintaining the stacks of patches required to backport rustc and cargo back to supported series.

As part of writing this up, I wrote a bunch of out of control shell scripts. They might come in useful.

PPAs and pockets and timelines

We upload new versions of rustc and cargo to the devel series like any other package.

For supported releases, the ultimate destination for rustc and cargo is the security pocket. They are only copied there though when the version of firefox that requires them is copied there (All PPAs mentioned here must only depend on the security pocket, not updates, and in other ways configured to be suitable binary-copy-to-the-primary-archive sources e.g. build and publish debug symbols).

The idealised timeline goes something like this:

1. upstream firefox master changes to depend on a new version of rustc (and cargo)

2. these versions of rustc and cargo are uploaded to devel and ppa:ubuntu-mozilla-security/rust-updates. I (mwhudson) use another PPA (ppa:mwhudson/rust-stuff) for my initial test builds but I don't ever copy things out of there.

3. when the version of firefox with the new rustc version requirement hits beta, rustc and cargo are binary copied to ppa:ubuntu-mozilla-security/rust-next. Beta versions of firefox are built in ppa:mozillateam/firefox-next, which depends on this one.

4. when this version of firefox is released, rustc and cargo are binary copied to ppa:ubuntu-mozilla-security/ppa and firefox is built in this PPA.

5. when firefox has built, firefox, rustc, and cargo are all copied to the security pocket.

It doesn't always happen like this of course. In particular, upstream doesn't always leave long enough between master switching to a new version and beta for us to get rust-updates updated before beta.

Updating for the devel series

As above, this uses the git ubuntu merge workflow. I have a repo that has remotes for the package import branch on launchpad, the Debian maintainers's git repo from salsa and lp:~canonical-foundations/ubuntu/+source/rustc . But be sure to check if those repositories are up-to-date - sometimes they can lag behind. If that's the case, things have to be done a bit more manual.

To merge a new version, 1.$X, I do something like this.

1. first note the tag of the version from debian you want to merge, this can be the Debian maintainer's tag (e.g. debian/1.32.0+dfsg1-3) or the package importers tag (e.g. pkg/import/1.30.0+dfsg1-1_exp2 -- sometimes you end up merging from experimental). Let's call this $newdebtag.

2. $ git checkout -b merge-1.$X merge-1.$((X-1))

3. $ git rebase -i $newdebtag --keep-empty.

   1. You might want to reword the first, empty, commit to indicate where the merge is from this time (unstable vs experimental vs NEW...)
4. remove the bottom five or so commits: merge-changelogs, reconstruct-changelog, ubuntu-meta, updating upstream source and "releasing package rust version ..."
5. work through the rebase, fixing conflicts as you go

6. Note the git tag for the last release we merged from debian, let's call this $olddebtag (to be honest I often flail a bit at this step and just try things until it works)

7. $ git-ubuntu.merge-changelogs $olddebtag merge-1.$((X-1)) $newdebtag

8. $ git commit -am 'merge-changelogs'

9. $ dch -v $(dpkg-parsechangelog -SVersion | sed 's/+dfsg1/&+llvm/')ubuntu1 '' --release-heuristic log -D UNRELEASED

10. $ git reconstruct-changelog $newdebtag

11. $ git commit -am 'reconstruct-changelog'

12. $ update-maintainer

13. $ git commit -am 'ubuntu-meta'

14. Import a version of the orig that bundles llvm: $ uscan. Now this has a couple of pitfalls:

    1. The tarball is enormous and will make mk-origtargz from older releases choke. Install devscripts from disco or better (and even then it still takes quite a long time, several minutes on my machine).

    2. The rust download page only lists the current release. If we're a bit behind and are packaging an old release (and we can't skip releases) you have to grab the source tarball directly. Luckily the URL of the old source tarball is very predictable. Then you have to run mk-origtargz by hand, which is the super memorable "mk-origtargz --package rustc --version 1.${X}.0 --repack-suffix +dfsg1+llvm --signature 1 --signature-file ../rustc-1.${X}.0-src.tar.xz.asc --repack --compression xz --directory .. --copyright-file debian/copyright ../rustc-1.${X}.0-src.tar.xz".

15. Import the freshly made orig tarball. (I use my-uupdate from my gist, but that's probably overkill)

16. Check that this update only touches (src/llvm-project in newer releases). If it doesn't something odd is going on, maybe you didn't download the orig that matches the version from debian.

17. At this stage I usually do a test build in a PPA and iterate until it builds: (rustbuild.sh --test from my gist does this)

    1. $ dch -b -v $(dpkg-parsechangelog -SVersion)~ppa1 ""

    2. $ dch -D $(distro-info -d) -r ""

    3. $ dpkg-buildpackage -nc -S --hook-done='dput ppa:mwhudson/rust-stuff ../%p_%s_source.changes'

    4. $ git reset --hard

18. When ready, release: (rustbuild.sh --release)

    1. $ dch -D $(distro-info -d) -r ''

    2. $ debcommit -r -a --sign-tag

    3. $ dpkg-buildpackage -nc -S  --hook-done='dput ../%p_%s_source.changes'

    4. $ git push -u foundations && git push foundations --tags

The process for cargo is broadly the same, although there's no need to repack the tarball.

Backports

This process has fewer steps than the devel update but can be trickier (I guess because devel and Debian unstable are usually more or less in sync). Backporting cargo is usually much harder than backporting rustc, because cargo interacts with the wider world much more.

For reasons that are hopefully obvious, work your way from newer to older series.

To backport version 1.$X to an older series, $SERIES, I do:

1. Create a backport branch: (backportrust.sh from my gist automates this but is still a bit scary tbh)

   1. $ git checkout -b $SERIES-1.$X $SERIES-1.$((X-1))

   2. $ git rebase -i --keep-empty merge-1.$X

      1. stop ("e") on each commit that adds patches to check that the patches apply, and fix those up before continuing.

   3. $ dch -b -v $(dpkg-parsechangelog -SVersion)~$(ubuntu-distro-info --series=$(git rev-parse --abbrev-ref HEAD | cut -d- -f1) --release | awk '{ print $1 }').1 '' --release-heuristic log -D UNRELEASED or something similar

   4. $ git reconstruct-changelog merge-1.$X

   5. $ git commit -am reconstruct-changelog

2. More PPA test builds: (rustbuild.sh --test again)

   1. $ dch -b -v $(dpkg-parsechangelog -SVersion)~ppa1 ""

   2. $ dch -D $(git rev-parse --abbrev-ref HEAD | cut -d- -f1) -r ""

   3. $ dpkg-buildpackage -nc -S --hook-done='dput ppa:mwhudson/rust-stuff ../%p_%s_source.changes'

   4. $ git reset --hard

3. Release when ready: (rustbuild.sh --release)

   1. dch -D $(git rev-parse --abbrev-ref HEAD | cut -d- -f1) -r ''

   2. $ debcommit -a -r --sign-tag

   3. $ dpkg-buildpackage -nc -S  --hook-done='dput ppa:ubuntu-mozilla-security/rust-updates ../%p_%s_source.changes'

   4. $ git push -u foundations && git push foundations --tags

A tip: if you have to make changes to a patch added by a commit at some point, you'll have to make the same changes to that patch for any older series. When doing the interactive part of rebase -i for the older series, you can replace the short sha of the change with the short sha of the new version of the change from the $SERIES-1.$X branch.

Upstream updates

Sometimes we need to run ahead of Debian (most likely when Debian is in freeze). In that case, the cargo and rustc source packages both contain README.source files that explain what to do. Good luck!!

When repacking the rustc tarball for a new upstream version, you should change the +dfsg1 suffix to +dfsg0.1 to indicate that the repack was not done by debian. For cargo we're generating a orig-vendor tarball but such version-mangling should not be necessary, as we very rarely sync from Debian anyway.

For cargo, running debian/make_orig_multi.sh to creating the orig-vendor tarball can be very tricky and can potentially require refreshing a lot of quilt packages. When dropped into the sub-shell, be sure to update the patches in both the respective vendor/<PACKAGE> and debcargo-conf/src/<PACKAGE> directories before exiting.

Once devel Ubuntu has the new upstream version, backporting basically follows the same pattern.