FAQ

My unsorted Ubuntu packaging and maintenance FAQ

(incl. even more - somewhat related - topics)

Help

Terms

Some important terms (with each more info below):

+1 Maintenance

Ubuntu Release Process

* References

* Each Ubuntu release cycle follows a similar pattern, with the phases listed below.

* 8 Phases:

  • Beginning a new Release (toolchain, seeds)
  • Planning (specs, ideas)
  • Merging with Upstream (syncs, merges until Debian Import Freeze)
  • Feature Development (oerlap usually with mergig, spec implementation)
  • Stabilization (Freeze) (greater caution towards release, queued uploads, FFe)
  • Milestones (images for alpha, beta, RC)
  • Finalization (only fixing showstopper/severe bugs, crashes, install issues)
  • Stable Releases (SRU - stable release update process starts)

SRU / Stable Release Update process

SRU Verification

Freeze

FFe

  • References:
  • Example:
    • [FFe] [UBUNTU 22.04] ibmca engine with libica = libica.so.4 - sshd dumps core (openssl-ibmca)

      https://bugs.launchpad.net/bugs/1967141 > openssl-ibmca (2.2.3-0ubuntu1) jammy; urgency=medium

    • New upstream release. LP: #1967141
    • The difference between 2.2.2 and 2.2.3 includes just these two fixes:
      • + "PKEY: Fix usage of ECX keys" + "use correct libica for ibmca_mechaList_test" Rather than adding these as quilt patches, raising the package to the bugfix-only version that incl. them is preferable.
    • For "PKEY: Fix usage of ECX keys" a backport of
      • "Fix compilation for OpenSSL 3.0" was needed: d/p/e59cce5-Fix-compilation-for-OpenSSL-3.0.patch
    • For convenience reasons a generated sample config is now included in
      • the package, but also the optional configuration generator Perl script 'ibmca-engine-opensslconfig'.
    • d/control: add dh-autoreconf to Build-Depends to work around a Lintian
      • regression on missing-build-dependency-for-dh-addon

Sponsoring / Sponsorship / Sponsors

Package Uploads

Proposed migrations

autopkgtests

* it's the uploader's responsibility to make sure the package is in a releasable state and that all the autopkgtests triggered by the upload are either passing or badtested

  • lp-test-ppa from https://git.launchpad.net/~ubuntu-server/+git/ubuntu-helpers offers some help:

    • lp-test-ppa --showurl ppa:kstenerud/postfix-postconf-segfault-1753470 --release bionic
  • The '--showurl' parameter prints out the URL, so that they can be cut-and-paste into email or chat. * using KVM:
  • using LXD:
    • autopkgtest-build-lxd images:ubuntu/impish/amd64
    • autopkgtest -U -s -o dep8-mypackage mypackage/ -- lxd autopkgtest/ubuntu/focal/amd64
  • using a PPA: --setup-commands="sudo add-apt-repository -y -u -s ppa:mylaunchpaduser/focal-mypackage-fixed-something-1234567"
  • on Canonistack (which is closest to real autopkgtests)
    • autopkgtest --no-built-binaries --apt-upgrade --setup-commands setup-testbed --shell-fail systemd_247.3-1ubuntu2.dsc -- ssh -s nova -- --flavor m1.small --image ubuntu/ubuntu-focal-daily-s390x-server-20221023-disk1.img --keyname fheimes_canonistack-bos01
  • by default: est will run against all packages in release plus the new candidate from proposed:

    • '--apt-pocket=proposed=src:<yourpkg>' or '--apt-pocket=proposed=src:srcpkg1,srcpkg2'

  • run against all packages that are in proposed:
    • '--apt-pocket=proposed'
  • in case a test requires a proxy one may add:

    • --env "http_proxy=http://squid.internal:3128" --env "https_proxy=http://squid.internal:3128" --env "no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpad.net,10.24.0.0/24"

Phasing / Staging / phased_updates

Merge proposals for package updates

  • It's recommended to do a quick self review of your merge proposal for a package update or version bump
  • Initial Review of the diff / debdiff
  • Check the changelog:
    • Contains bug pointer, like '(LP: #1234567)'

    • Proper version change: https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging

    • Proper distribution: not unstable, not a Debian release name
    • Lists ideally all files changed
    • quilt patches with 'fuzz' are not allowed and need to be fixed/refreshed
    • refresh of quilt patches to solve 'offsets' should be at done at max if working on '-devel'
      • (so for PATCHES, not for SRUs) or to fix a fuzz

    • Proper author & email

    • make the changelog clear and crisp - ideally "grep-able" with key words, file names, etc.

  • Following the dep-3 http://dep.debian.net/deps/dep3/ for quilt patches:

    • Description: <short description, required>

      • <long description that can span multiple lines, optional>

      Author: <name and email of author, optional> Origin: <upstream|backport|vendor|other>, <URL, required except if Author is present> Bug: <URL to the upstream bug report if any, implies patch has been forwarded, > Bug-<Vendor>: <URL to the vendor bug report if any, optional> Forwarded: <URL|no|not-needed, useless if you have a Bug field, optional> Applied-Upstream: <version|URL|commit, identifies patches merged upstream, option> Reviewed-by: <name and email of a reviewer, optional> Last-Update: 2021-11-19 <YYYY-MM-DD, last update of the meta-information, option> --- This patch header follows DEP-3: http://dep.debian.net/deps/dep3/

  • Check the commits:
    • Patches / changes are logically split into separate git commits (d/p)
    • in case a package was never touched by Canonical for Ubuntu
      • (hence has no 'ubuntu?' in its version), but now needs to be, the tool 'update-maintainer' tool needs ot be called which will modify debian/control which then needs to be committed as well (as separate commit) (d/control)
    • changelog is usually the last commit (d/changelog)
  • Check the files:
    • Only files inside the debian directory must be modified!
  • Review Template

    • [ https://github.com/canonical/ubuntu-maintainers-handbook/blob/main/MergeProposalReview.md ] The following template may come useful when submitting reviews:

      • Changelog:
        • [ ] old content and logical tag match as expected
        • [ ] changelog entry correct version and targeted codename
        • [ ] changelog entries correct
        • [ ] update-maintainer has been run
      • Actual changes:
        • [ ] no upstream changes to consider
        • [ ] no further upstream version to consider
        • [ ] debian changes look safe
      • Old Delta:
        • [ ] dropped changes are ok to be dropped
        • [ ] nothing else to drop
        • [ ] changes forwarded upstream/debian (if appropriate)
      • New Delta:
        • [ ] no new patches added
        • [ ] patches match what was proposed upstream
        • [ ] patches correctly included in debian/patches/series
        • [ ] patches have correct DEP3 metadata / header (incl. working URL for field Origin)
      • Build/Test:
        • [ ] build is ok
        • [ ] verified PPA package installs/uninstalls
        • [ ] autopkgtest against the PPA package passes
        • [ ] sanity checks test fine
      • Other:
        • [ ] LP bug public
        • [ ] LP bug affects the correct package(s)
  • If there is now everything is prepared:
    • change the MP from "work in progress" to "needs review"
    • (on change to 'needs review' you'll get a notification mail)
    • and check (again) that the correct reviewer is set
    • and check (again) that a 'Related bug(s)' is assigned
    • change the LP bug status to "In Progress"
    • a SRU template was added to the LP bug (not needed for development release patches)
    • subscribe the "ubuntu-sponsors" LP group to the LP bugs (since sponsorship is needed)
    • ask someone to review and sponsor (maybe informally)
    • add a tag for the ubuntu release, like 'focal'
    • crossing fingers that everything is okay ...
  • If the reviewer / sponsor is happy and uploaded the changes,
    • he/she usually adds a comment to the ticket, like: " Thanks. I fixed up a small lintian warning about the changelog entry being longer than 80 chars. Other than that, LGTM. Uploaded. "
  • If the updated package has arrived at the ubuntu release unapproved queue:
  • SRU package upload is stuck
    • In case an SRU package upload is stuck in the upload queue for several days (e.g. 1+ week), join the IRC channel #ubuntu-release and ask the SRU team kindly for having a look at this specific package, by using the tag "ubuntu-sru", ideally adding the URL and ask for approval. But better double check the status before using rmadison:
    • Example, write:
    • Don't forget to be thankful:

      • "@<SRU-team member> many thx (just received the notification ...)"

    • So use the IRC channel: #ubuntu-release
      • Highlight with "ubuntu-sru" - the SRU team member should have a notification enabled for this keyword (similar to "ubuntu-archive", "ubuntu-release", "ubuntu-sponsors", ...)
  • Sometimes it might be needed to directly ping the SRU team member that's on duty directly
  • In case of package for the development release that is stuck in the upload queue,
    • use again #ubuntu-release (not necessarily #ubuntu-devel), since what's basically needed is approval from someone from the RT (release-team) or AA (archive-admins) to sign this up, and those hang out in '#ubuntu-release'.

Package versioning

  • Updating a package version:
  • No-change rebuild examples:

    • SRU: 1.43-3build2 ==> 1.43-3build2.1

    • PATCH: 1.43-3build2 ==> 1.43-3build3

      • (since it's 'no-change' do not add an Ubuntu specific version suffix 'XubuntuY')
  • Ubuntu-specific examples:

    • PATCH example: 0.111-1 ==> 0.111-1ubuntu1

      • (In such a case one need to run the 'update-maintainer' script.)

  • BUT most important is to check that the upgrade path of a package is never broken.

    • Means the package versions of the different Ubuntu releases must be in an ascending order, like:
      package version 'a' of Ubuntu release n-1 < package version 'b' of Ubuntu release n
      package version 'b' of Ubuntu release n < package version 'c' of Ubuntu release n+1 -- and so on
      so a < b < c
      In other words: ensure there is always a proper upgrade path
      One may use 'dpkg --compare-versions'.

  • a tilde ('~') in the version string usually indicates a backport, like: 1.2.3-3ubuntu1~20.04.1

Patch / Fix

Packaging

Binary Patch / Fix

  • In case a patch is needed for a package that modifies one or more binary files in a source package, quilt (as used in debian/source/format '3.0 quilt') is not longer helpful, since it does not support binary patching ("git binary diffs are not supported").
  • But there are at the following options to tackle or workaround this:

    • Do not patch the package, rather than lift it to the latest version (version bump),

      • that already incl. the latest binary files.

    • Ignore the commit/patch that modified binary files.

      • (I had one case where the binary was at test/, hence didn't actively fix something,
        rather than added another test case. Here it was save to skip it.)

    • Put updated binaries under debian/ and use debian/source/include-binaries

    • Do not deal with binaries inside debian/ directory at all

      • and encode it in base64 and decode it at build time
        (for example done in case smaller images or icon files are needed).

(It's usually not great - and contradictory to the philosophy of Debian source packages - to have binaries inside of a source package, but it is sometimes just needed and the case; for example in test cases.)

New packages

Autobuilders / Builds

Sync / Merge

  • References:
  • General

    • A sync copies a source package verbatim from an external repository into Ubuntu, overwriting any package of the same name. This is used when a newer version of it is available, and should be included in Ubuntu, and happens automatically during some phases of the release cycle. To request a sync, follow the SyncRequestProcess.

    • A merge is a three-way merge of a package which originated in an external repository. This is used when there is a newer version available from the external repository, but the package has also been modified (branched) in Ubuntu. Merge-o-Matic assists with this work, and the Merging page explains how and when to merge. Packages which are maintained in Bazaar can and should be merged using Bazaar itself.
    • Merge (as in ubuntu-maintainers-handbook)
      • Server team does merges using git-ubuntu - similar to a git rebase, where commits from one point are replayed on top of another.
      • also splitting larger commits into smaller logical units,
        • one commit per logical unit (per bug/fix, d/changelog, d/c) (old-style commits that lumped multiple changes)
        • Start a rebase: git rebase -i lp1234567/old/debian
        • Change the commit(s) you're going to split from pick to edit.
        • git reset to get your changes back: git reset HEAD^
        • Now add the commits of a logical unit each, let's say some patches, d/rules, d/control, d changelog
        • and end the rebase with 'git rebase --continue'
        • add tag split (git ubuntu tag --split --bug 1234567)
      • harmonizing d/changelog commits into two commits: a changelog merge and a reconstruction.
  • Sync (as in ubuntu-maintainers-handbook)
    • "For other sync situations, you can review this. Outside of the server team process the common way is to request for an explicit sync via filling a Launchpad Bug or using the 'requestsync' tool."

      https://manpages.ubuntu.com/manpages/impish/man1/requestsync.1.html in case one has permissions to upload the package to Ubuntu: 'syncpackage' (copy source packages from Debian to Ubuntu)

  • merge: debdiff of NEW Debian package and NEW Ubuntu package
  • some general MERGE/SYNC notes to remember:
    • always to a test build (e.g. in PPA) on all relevant architectures (as usual)
    • if tests are available, run them as well
    • also do an package install and update test

    • and check the package content (dpkg -c <package>)

    • look for upstream bugs that can be fixed at the same time

    • check for any "reverse-depends -a source src:<package>" (esp. with libs)

      • and test build them as well, pointing to the <package> test PPA (or using the same)

    • do the source package build using -v<previous version>, like:

      • dpkg-buildpackage -S -d -v7.6.0-4ubuntu1 --sign-key=... this causes the 'in-between changelog items' to appear in the .changes file (means all changelog entries from previous to new version)
    • make sure the LP bug for the merge is referenced in the changelog
      • (since lot's of things are needed to create and populate the LP bug, I start the work w/o having a LP bug number; so don't forget to add it later!)
    • if working based on the Merge-o-matic files there is usually a proposed changelog file
      • either update this properly or create a new/current entry using 'dch -i'
      • check if the version is correct (MERGES and SYNCS aren't SRUs!)
      • update the version from UNSTABLE (or so) to the current development release (kinetic)
      • update the urgency (should usually be medium, and not low as in the 'mom' template)
      • if reusing the 'mom' template overwrite the mom mail with yours
        • and update the date to the current one! (not needed is using 'dch -i')
  • Examples for MERGE:
    • 'ferret-vis' - MERGE
      • LP#1986565 - "Please merge ferret-viz 7.6.0-5 into kinetic"

        https://bugs.launchpad.net/bugs/1986565 ferret-vis (7.6.0-5ubuntu1) kinetic; urgency=low

      • Merge from Debian unstable, remaining change:
        • + Build without lto. lto-disabled-list can't be used (d/rules), because
          • dpkg-buildflags is called after changing the package base directory.
    • 'libgcrypt20' - MERGE
      • LP#1974277 - "Please merge libgcrypt20 1.10.1-2 (main) from Debian unstable (main)"

        https://bugs.launchpad.net/bugs/1974277 > libgcrypt20 (1.10.1-2ubuntu1) kinetic; urgency=low

      • Merge from Debian unstable. (LP: #1974277) Remaining changes:
        • + d/p/disable_fips_enabled_read.patch
          • Disable the library reading /proc/sys/crypto/fips_enabled file and going into FIPS mode. libgcrypt is not a FIPS certified library.
      • Removed d/p/0001-Always-include-config.h-in-cipher-assembly-codes.patch
        • since it's already included in the new version.
      • Removed d/p/0001-poly1305-fix-building-with-arm-linux-gnueabihf-gcc-1.patch
        • since it's already included in the new version.
      • Refreshed d/p/12_lessdeps_libgcrypt-config.diff
        • and d/p/disable_fips_enabled_read.patch due to offsets.

        https://launchpadlibrarian.net/602209655/debdiff_libgcrypt20_of_Debian_1.10.1-2_and_Ubuntu_1.10.1-2ubuntu1.patch

    • 'atlas' - MERGE
  • Examples for SYNC:

Backports

  • References:
  • Backports work 'roughly' similarly to syncs, but have somewhat different requirements.
  • backports have a tilde ('~') in their version string, like: 1.2.3-3ubuntu1~20.04.1

Transition

  • A transition occurs when changes in a package A require further changes to other packages B, C, ..., that depend on A.
    • To solve this, the packages B, C, ... that use or depend on package A, either need to be updated to a new version or recompiled. (Packages blocking a transition might be postponed from testing so that the transition can complete.)
  • To handle a transitions, all affected (source) packages must migrate together to keep the archive installable (and the dependencies satisfied)

    • <and I would add: 'and to keep the combo of such packages working'>

  • This happens in case of an SONAME bump (libfoo[n] changes to libfoo[n+1]),

    • means if build dependencies (e.g. of a library) change significantly,
      like an updated/new version that comes with API/ABI changes.
      or simply in case of a renamed package.

  • A rebuild of all packages that need that package or lib as build dependency is needed
    • (as well as testing, e.g. autopkgtests).
  • To find all reverse dependencies use: reverse-depends -a source src:libica

  • The currently active transitions can be found here: https://people.canonical.com/~ubuntu-archive/transitions/

  • Examples:

    • small example:

      • reverse-depends -a source src:libica
        Reverse-Build-Depends

        • opencryptoki (for libica-dev)
          openssl-ibmca (for libica-dev)

    • mid-size example:

      • reverse-depends -a source src:pcre2
        - 5 'Reverse-Testsuite-Triggers'
        - 91 'Reverse-Build-Depends'

    • huge example:

      • reverse-depends -a source src:zlib (zlib and htslib)
        - 55 'Reverse-Testsuite-Triggers'
        - 1435 'Reverse-Build-Depends'
        - 9 'Reverse-Build-Depends-Indep'

        • (Same as Build-Depends, but only needed when building arch. independent packages.)

        - 23 'Reverse-Build-Depends-Arch'

        • (Same as Build-Depends, but only needed when building arch. dependent packages.)

  • Transition for renamed package (https://wiki.debian.org/RenamingPackages)

    • define a new binary dummy package with the same name as the old package in the control file of the new package. The new source package takes over the binary dummy package, and the old source package, which is then binaryless, can be cleaned up.
  • Example:
    • LP#1959421 - "[22.04 FEAT] Upgrade libica to latest version (crypto) (v4.0)" incl. transition

      https://bugs.launchpad.net/bugs/1959421 > libica (4.0.1-0ubuntu1) jammy; urgency=medium

    • New upstream release LP: #1959421
    • transition from libica3 to libica4
      • + adjust d/control + migrate d/libica3.* to d/libica4.* + extended symbols file
    • adjusting d/copyright
      • + change tabs to spaces + update relevant copyright years

MIR (main inclusion report)

Package Removal

  • References:
  • This is done upon request
    • create a LP bug request for this
    • subscribe (NOT assign) ubuntu-archive to it
    • remove both the source package and all binary packages?
    • state why the package should be removed
    • which Ubuntu release (series) to remove it from

    • confirm that the binary packages have no rdepends (means no other package depends on them)

      • include the output of 'reverse-depends -a source src:$package'
        and 'reverse-depends src:$package'

    • Example: https://bugs.launchpad.net/bugs/2008240

  • If help if needed on the decision to whether a package should be removed or not,
    • start a discussion on the 'ubuntu-devel' mailing.

NBS

  • References:
  • Binary packages that are not built from any source (anymore)
  • some reasons for such cases:
    • a library gets a new SONAME (libfoo[n+1]) (due to an update),
      • the previous version built libfoo[n]
    • a package gets renamed by upstream and the renaming is reflected in the package name change
  • such packages are not automatically removed from the archive,
    • since they still have reverse dependencies (packages depending on them)
  • Sometimes one NBS package will depend on another, which can confuse a transition.
  • In case a package has an alternate dependency, it will still show (but doesn't always need fixing).
  • Remember that NBS removals are manual, which can confuse the results.

Symbols

Seeds

Bileto / former "CI Train"

Upload rights

Kernel bug fixing / patching

  • References:

  • cherry-pick lines:

    • (cherry picked from commit de5012b41e5c900a8a3875c7a825394c5f624c05 linux-next)
      or just:
      (cherry picked from commit de5012b41e5c900a8a3875c7a825394c5f624c05)

  • backported lines:

    • (backported from commit 1a82f6ab23659aa01a796d9d444ec9cc63ded26c linux-next)
      or just:
      (backported from commit 3d787b392d169d4a2e3aee6ac6dfd6ec39722cf2)
      [Frank Heimes: Resolve minor conflict due to additional includes.]

  • In both cases add:

  • for kernel pull-requests:

  • For a normal (means non PR) submission to the kernel teams mailing list

    • use a subject line like this:
      <Subject: [SRU|PATCH][<affected Ubuntu releases>][PATCH n/m] commit name>
      (Non PR submissions should be done btw. up to about 5 commits <cherry-picks or backports>

  • setting kernel options:

    • Remember the new way of handling kernel options (thx to arighi) via the annotations file and script:
      https://discourse.ubuntu.com/t/kernel-configuration-in-ubuntu
      https://lists.ubuntu.com/archives/kernel-team/2023-June/140230.html
      And notice that 'updateconfigs' must be run on the same target platform (Ubuntu release) your kernel sources are made for, so that the correct tool-chain is used, since some kernel options are depending on the tool-chain. (Means run updateconfigs for jammy kernel sources on a jammy system, e.g. using chroot.) A changing tool-chain between updateconfigs and build may (will) cause issues. Alternatively remove any confusing kernel options by hand before building, but this is a bit error prone.

Bugs / Triage

Packaging-Resources

* Debian Packaging:

Further Reading/References

FrankHeimes/FAQ (last edited 2024-02-27 12:23:21 by fheimes)