BuildEnvironment
Size: 47610
Comment: update for UQT conversion to git
|
Size: 51184
Comment: Remove a bunch of crufty old instructions which are no longer needed
|
Deletions are marked like this. | Additions are marked like this. |
Line 6: | Line 6: |
[[https://help.ubuntu.com/community/SbuildLVMHowto|SbuildLVMHowto]]. Additional information may be found in [[http://bazaar.launchpad.net/~ubuntu-security/ubuntu-security-tools/trunk/annotate/head:/README.sbuild_setup|README.sbuild_setup]]. If you're not a Ubuntu Security Team member, you might want to just read the [[https://wiki.ubuntu.com/SimpleSbuild|simpler, more generic instructions]] for setting up sbuild without bothering with the security team's workflow. | [[https://help.ubuntu.com/community/SbuildLVMHowto|SbuildLVMHowto]]. Additional information may be found in [[SecurityTeam/TestingEnvironment | TestingEnvironment]] and [[https://git.launchpad.net/ubuntu-security-tools/tree/README.sbuild_setup|README.sbuild_setup]]. If you're not a Ubuntu Security Team member, you might want to just read the [[https://wiki.ubuntu.com/SimpleSbuild|simpler, more generic instructions]] for setting up sbuild without bothering with the security team's workflow. |
Line 10: | Line 10: |
This guide does not impose a specific version of Ubuntu; the latter is entirely up to you. |
|
Line 11: | Line 13: |
By default when using schroot, mk-sbuild in Lucid uses 'union-type=aufs' while in Precise and later mk-sbuild uses 'union-type=overlayfs'. Also, mk-sbuild passes --variant=buildd to debootstrap so you don't have to. Due to performance issues with ext4 in Ubuntu 10.04 LTS, it is recommended that you use ext3 for your builds (feel free to use ext4 in later releases). For build and package testing, x86_64 users are strongly encouraged to create chroots for both amd64 and i386, whereas i386 users need only to create i386 chroots. '''Note:''' Precise and later no longer supports aufs so after upgrading to Precise, you will have to adjust your schroots to use 'overlayfs' (see below). '''Note 2:''' 'overlayfs' is an Ubuntu sauce patch and not in upstream kernels. To use upstream kernel's overlayfs support, use 'overlay' instead, which can be used with both Ubuntu and upstream kernels. To convert from overlayfs, use `sudo sed -i 's/union-type=overlayfs/union-type=overlay/' /etc/schroot/chroot.d/*` |
For build and package testing, x86_64 users are strongly encouraged to create chroots for both amd64 and i386, whereas i386 users need only to create i386 chroots. Note, [[https://ubuntu.com/blog/statement-on-32-bit-i386-packages-for-ubuntu-19-10-and-20-04-lts?_ga=2.23313978.1843507387.1581024788-1548617305.1358188997|as of Ubuntu 19.10, i386 support is limited]] '''Note:''' schroot does not trivially work on zfs root file systems. On jammy it might be possible to pass --type=zfs-snapshot to mk-sbuild. |
Line 26: | Line 20: |
$ sudo apt-get install sbuild schroot ubuntu-dev-tools }}} '''Note:''' if your host is based on xenial and you get to fail building for yakkety or zesty, some error messages like: {{{ Merged Build-Depends: build-essential, fakeroot Filtered Build-Depends: build-essential, fakeroot dpkg-deb: building package 'sbuild-build-depends-core-dummy' in '/<<BUILDDIR>>/resolver-Pj2d3h/apt_archive/sbuild-build-depends-core-dummy.deb'. gpg: /<<BUILDDIR>>/resolver-Pj2d3h/gpg/trustdb.gpg: trustdb created gpg: Warning: not using 'Sbuild Signer' as default key: No secret key gpg: all values passed to '--default-key' ignored gpg: no default secret key: No secret key gpg: signing failed: No secret key Failed to sign dummy archive Release file. }}} You may need to update your host sbuild for yakkety version 0.71. See previous reported issue [[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=827315|here]] and [[https://bugs.launchpad.net/ubuntu/+source/sbuild/+bug/1621269|here]] for more information. 0. Make sure you are in the 'sbuild' group:{{{ $ sudo adduser <username> sbuild |
$ sudo apt-get install sbuild schroot ubuntu-dev-tools moreutils }}} '''Note:''' for focal it will probably need to install ca-certificate, since it's not installed by default with schroot. 0. Make sure you are in the `sbuild` group. If you aren't, run the command above and re-login to your account:{{{ $ sudo adduser $USER sbuild |
Line 46: | Line 31: |
$ sudo mkfs.ext3 /dev/sdXN | $ sudo mkfs.ext4 /dev/sdXN |
Line 74: | Line 59: |
0. Setup for mounting $HOME and extracting the ddebs: For schroot < 1.6.1-1 (Ubuntu 12.04 and earlier): * Append to /etc/schroot/mount-defaults ('''IMPORTANT''': this file may be overwritten on upgrades of schroot):{{{ /home/<username>/ubuntu/scratch /scratch none rw,bind 0 0 }}} If you use an encrypted home directory, your $HOME is mounted differently (eg /home/<username>/.Private is mounted on /home/<username>), so you will also have to add to /etc/schroot/mount-defaults:{{{ /home/<username> /home/<username> none rw,bind 0 0 }}} * Create /etc/schroot/script-get-ddebs:{{{ # Do whatever the default script would do . /etc/schroot/default/config # Clean up during 00check if [ "x$0" = "x/etc/schroot/setup.d/00check" ]; then if [ $1 = "setup-stop" ]; then # Pull out .ddebs mv /var/cache/buildtmp/schroot/union/overlay/"$SESSION_ID"/build/*/*.ddeb /scratch/ddebs/ 2>/dev/null || true fi fi # end script }}} * Create the following directory:{{{ $ mkdir -p ~/ubuntu/scratch/ddebs }}} <<Anchor(schrootconf12.10)>>For schroot >= 1.6.1-1 (Ubuntu 12.10 and later): |
0. Setup for mounting $HOME: |
Line 110: | Line 69: |
/home/<username>/ubuntu/scratch /scratch none rw,bind 0 0 | |
Line 127: | Line 85: |
# Clean up during 00check if [ "x$0" = "x/etc/schroot/setup.d/00check" ]; then if [ $1 = "setup-stop" ]; then # Pull out .ddebs mv /var/cache/buildtmp/schroot/union/overlay/"$SESSION_ID"/build/*/*.ddeb /scratch/ddebs/ 2>/dev/null || true fi fi |
|
Line 136: | Line 86: |
}}} * Create the following directory:{{{ $ mkdir -p ~/ubuntu/scratch/ddebs |
|
Line 167: | Line 113: |
Then make the following directories (change if specified something different in ~/.sbuildrc):{{{ $ mkdir -p $HOME/ubuntu/logs # $HOME/ubuntu/build (if used) }}} 0. Create ~/.mk-sbuild.rc: For schroot < 1.4.6-1 (Ubuntu 10.04 LTS and earlier):{{{ # mk-sbuild tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin" SKIP_UPDATES="1" }}} For schroot >= 1.4.6-1 (Ubuntu 10.10 and later):{{{ # mk-sbuild tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin preserve-environment=true" SKIP_UPDATES="1" }}} For ubuntu-dev-tools >= 0.138 (ie, 12.04 and later):{{{ # mk-sbuild build tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin preserve-environment=true" SKIP_UPDATES="1" # Do not use the following line if you want to reproduce Launchpad builds # accurately; devscripts has significant dependencies. DEBOOTSTRAP_INCLUDE="devscripts" }}} <<Anchor(mksbuildrc12.10)>>For schroot >= 1.6.1-1 (Ubuntu 12.10 and later):{{{ # mk-sbuild build tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="profile=umt source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin preserve-environment=true" SKIP_UPDATES="1" # Do not use the following line if you want to reproduce Launchpad builds # accurately; devscripts has significant dependencies. DEBOOTSTRAP_INCLUDE="devscripts" }}} <<Anchor(mksbuildrc13.04)>>For Ubuntu 13.04 and later:<<BR>> '''IMPORTANT''': mk-sbuild in Ubuntu 13.04 forces the schroot profile to be "sbuild" by default. On Ubuntu 13.04, we need to use mk-sbuild from Ubuntu 13.10 (ubuntu-dev-tools version 0.149 and later).{{{ |
Then make the following directories (change if specified something different in `~/.sbuildrc`):{{{ $ mkdir -p $HOME/ubuntu/logs $ mkdir -p $HOME/ubuntu/build # only if the $build_dir variable is set in ~/.sbuildrc }}} 0. Create ~/.mk-sbuild.rc:{{{ |
Line 229: | Line 127: |
# Do not use the following line if you want to reproduce Launchpad builds # accurately; devscripts has significant dependencies. DEBOOTSTRAP_INCLUDE="devscripts" |
# devscripts no longer installs correctly, so install it manually later DEBOOTSTRAP_INCLUDE="" |
Line 235: | Line 132: |
Line 237: | Line 134: |
Using 'SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs"' means that /etc/schroot/script-get-ddebs gets called to configure the chroot. This script sources '/etc/schroot/default/config' so it behaves normally, then pulls the ddebs out and puts them into the bind mounted /scratch directory in the chroot (ie into ~/scratch). '''IMPORTANT:''' If you are using overlayfs, keep in mind that the writable overlay is in `/var/lib/schroot/union/overlay` by default (`man 1 schroot`). This means your build's compile while be on this partition and if there isn't enough space, your builds will fail. To change this to somewhere with more space, create an overlay directory (eg, `sudo mkdir -p /srv/devel/schroot/overlay`), then adjust your existing schroots to have '`union-overlay-directory=/srv/devel/schroot/overlay`'. To use this by default for all schroots, modify `~/.mk-sbuild.rc` to have '`union-overlay-directory=/srv/devel/schroot/overlay`' somewhere in `SCHROOT_CONF_SUFFIX`. |
'''IMPORTANT:''' If you are using overlayfs, keep in mind that the writable overlay is in `/var/lib/schroot/union/overlay` by default (`man 1 schroot`). This means your build's compile while be on this partition and if there isn't enough space, your builds will fail. To change this to somewhere with more space, create an overlay directory (eg, `sudo mkdir -p /srv/devel/schroot/overlay`), then adjust your existing schroots to have '`union-overlay-directory=/srv/devel/schroot/overlay`'. To use this by default for all schroots, modify `~/.mk-sbuild.rc` to have '`union-overlay-directory=/srv/devel/schroot/overlay`' somewhere in `SCHROOT_CONF_SUFFIX`. 0. Install Sendmail: `sudo apt install sendmail` |
Line 242: | Line 140: |
$ for i in precise trusty wily xenial; do | $ for i in noble mantic jammy focal bionic xenial trusty; do |
Line 244: | Line 142: |
mk-sbuild $i --arch=i386 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu | if echo $i | grep -qE '(focal|jammy)'; then # libeatmydata1 not available in focal and jammy on i386 mk-sbuild $i --arch=i386 --skip-updates --skip-eatmydata --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu else mk-sbuild $i --arch=i386 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu fi |
Line 246: | Line 148: |
for i in wheezy jessie stretch buster sid ; do |
$ exit # only needed if you used sg sbuild earlier; sg changes primary group }}} If you want debian schroots: {{{ $ for i in buster bullseye stable testing unstable ; do |
Line 259: | Line 164: |
'''Note 1''': debootstrap in Natty will no longer install Dapper schroots because of the switch to SHA256. As a temporary work-around until Dapper goes EoL, you can downgrade to Maverick's debootstrap, create the Dapper schroots, and upgrade back. '''Note 2''': Oneiric (and higher?) schroots may not have the /var/lock directory which is expected in older versions of sbuild. To fix this:{{{ $ for a in i386 amd64 armel ; do schroot -c oneiric-$a-source -u root -- sh -c 'mkdir /var/lock ; chmod 1777 /var/lock' ; done }}} '''Note 3''': Oneiric and higher require you to run {{{sbuild-update --keygen}}} to generate a local archive key. This only needs to be done once and can be done as your user so long as you are in the 'sbuild' group. See manpage for details. (If you see this error when creating the keys 'gpg: can't create `(null)': Permission denied', then simply remove the existing keys in {{{/var/lib/sbuild/apt-keys/*}}} and try again.) '''Note 4''': Debian schroots pull in exim4-base but Ubuntu systems do not. Due to [[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=565613|Debian bug #565613]] the passwd and group databases are pulled in from the host, overwriting what is in the chroot. Since Ubuntu by default does not have exim4-base installed, the user isn't there and you will end up with errors like this when updating the schroot at a later date: `dpkg: syntax error: unknown group `Debian-exim' in statusoverride file`. You can either create the user/group on the Ubuntu host or remove the exim4-base package from the schroot (this may not work with older Debian releases):{{{ |
'''Note 1''': Debian schroots pull in exim4-base but Ubuntu systems do not. Due to [[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=565613|Debian bug #565613]] the passwd and group databases are pulled in from the host, overwriting what is in the chroot. Since Ubuntu by default does not have exim4-base installed, the user isn't there and you will end up with errors like this when updating the schroot at a later date: `dpkg: syntax error: unknown group `Debian-exim' in statusoverride file`. You can either create the user/group on the Ubuntu host or remove the exim4-base package from the schroot (this may not work with older Debian releases):{{{ |
Line 272: | Line 168: |
'''Note 5''': `mk-sbuild --arch armhf vivid` may fail when running the 'finish.sh' script. If it does, go into the source:<chroot> as root and run `/finish.sh`. Eg: `sudo schroot -c source:<chroot name> -u root /finish.sh` '''Note 6''': apt is configured to download the package translations files by default. These can be a source of build failures if they get corrupted and can cause apt-get update to take longer; therefore it can be useful to configure to disable downloading the translations files. To do so, do e.g.:{{{ |
'''Note 2''': apt is configured to download the package translations files by default. These can be a source of build failures if they get corrupted and can cause apt-get update to take longer; therefore it can be useful to configure to disable downloading the translations files. To do so, do e.g.:{{{ |
Line 279: | Line 174: |
'''Note 7''': yakkety will fail with devscripts dependency errors. When generating the yakkety schroot, change ~/.mk-sbuild.rc to include {{{DEBOOTSTRAP_INCLUDE=""}}} . Afterwards, install devscripts in the chroot and change the ~/.mk-sbuild.rc back to {{{DEBOOTSTRAP_INCLUDE="devscripts"}}}. '''Note 8''': if using foreign architectures in a chroot in a container, be sure that qemu-user-static is installed on the host and in the container so that /proc/sys/fs/binfmt_misc/qemu-arm from the host (which is bind mounted in the guest) correctly points to /usr/bin/qemu-arm-static. If you don't do this you might encoutner cryptic errors like "I: Running command: chroot /home/devel/schroot/xenial-armhf /debootstrap/debootstrap --second-stage" '''Note 9''': Uploading packages for Ubuntu 17.10 (artful) and later from a 17.04 (zesty) host or earlier requires installing devscripts from artful (or later) to properly setting signing .buildinfo files |
'''Note 3''': yakkety and newer may fail with devscripts dependency errors such as {{{ W: Failure while configuring base packages. This will be re-attempted up to five times. W: See /srv/devel/schroot/eoan-i386/debootstrap/debootstrap.log for details (possibly the package libnet-ssleay-perl is at fault) }}} or error in /srv/devel/schroot/eoan-i386/debootstrap/debootstrap.log: {{{ dpkg: error processing package devscripts (--configure): dependency problems - leaving unconfigured Errors were encountered while processing: libnet-ssleay-perl libio-socket-ssl-perl liblwp-protocol-https-perl libwww-perl devscripts }}} If so, then be sure to check this line {{{DEBOOTSTRAP_INCLUDE="devscripts"}}} in your ~/.mk-sbuild.rc is commented out. This then requires that for each schroot you need to go and purposely install devscripts in it, otherwise you'll notice packages fail to build. Eg {{{ $ schroot -c source:bionic-amd64 -u root -- apt-get install devscripts }}} '''Note 4''': if using foreign architectures in a chroot in a container, be sure that qemu-user-static is installed on the host and in the container so that /proc/sys/fs/binfmt_misc/qemu-arm from the host (which is bind mounted in the guest) correctly points to /usr/bin/qemu-arm-static. If you don't do this you might encoutner cryptic errors like "I: Running command: chroot /srv/devel/schroot/xenial-armhf /debootstrap/debootstrap --second-stage" '''Note 5''': Bionic and newer no longer support {{{ union-type=overlayfs }}} . {{{ cd /etc/schroot/chroot.d && sed -i 's/overlayfs/overlay/' sbuild-* }}}. |
Line 290: | Line 204: |
This simply adds additional schroots based on what you already have (eg, if you have a lucid-amd64 schroot, this creates a lucid-amd64-shm definition in /etc/schroot/chroot.d/shm-overlays.conf). The caveat is that each shm schroot will have the same 'directory' as its corresponding non-shm schroot, so you shouldn't try to modify both 'lucid-amd64-source' and 'lucid-amd64-shm-source' at the same time. Once the shm schroot definitions are add, reference them via schroot (or umt) like any other schroot. Eg: {{{ $ schroot -c lucid-amd64-shm $ umt build -c lucid-amd64-shm |
This simply adds additional schroots based on what you already have (eg, if you have a bionic-amd64 schroot, this creates a bionic-amd64-shm definition in /etc/schroot/chroot.d/shm-overlays.conf). The caveat is that each shm schroot will have the same 'directory' as its corresponding non-shm schroot, so you shouldn't try to modify both 'source:bionic-amd64' and 'source:bionic-amd64-shm' at the same time. Once the shm schroot definitions are added, reference them via schroot (or umt) like any other schroot. Eg: {{{ $ schroot -c bionic-amd64-shm $ umt build -c bionic-amd64-shm |
Line 297: | Line 211: |
=== Upgrading to Precise (12.04) === Aufs has been deprecated in 12.04 (but is still available). When upgrading your build host to 12.04, you are encouraged to transition to overlayfs: * Adjust all the aufs schroots to use overlayfs. Eg:{{{ $ sudo sed -i 's/aufs/overlayfs/' /etc/schroot/schroot.conf }}} Note that newer versions of mk-sbuild setup up the chroot configurations in ```/etc/schroot/chroot.d/``` (e.g. ```/etc/schroot/chroot.d/sbuild-precise-amd64```) so you may need to edit files there as well. * Adjust ~/.mk-sbuild.rc to have (devscripts no longer installed by default in 12.04 and later. Specify multiple packages via a comma-separated list):{{{ DEBOOTSTRAP_INCLUDE=devscripts }}} Do not do this if you want to reproduce Launchpad builds accurately. === Upgrading to Quantal (12.10) === The script-config option in schroot.conf has been deprecated in favor of the new profile option. /etc/schroot/ has been shuffled around to move to the new profile based layout. This breaks existing chroots on upgrade to Quantal, so you are encouraged to transition to the new profile layout: * Adjust your schroot configuration as described [[#schrootconf12.10|above]] for schroot-1.6.1-1. * Adjust all existing build schroots to use the new umt profile. Eg:{{{ $ sudo sed -i 's/script-config=script-get-ddebs/profile=umt/' /etc/schroot/schroot.conf $ sudo sed -i 's/script-config=script-get-ddebs/profile=umt/' /etc/schroot/chroot.d/* }}} * Adjust ~/.mk-sbuild.rc as described [[#mksbuildrc12.10|above]] for schroot-1.6.1-1. === Install some helpful packages === There are a few packages that will help you more closely mimic the buildd's so that the results of your local builds will be as close as possible to those used to build Ubuntu packages. Here is a list of helpful things to install. The general format is: |
=== Install some additional packages === There are a few packages that will help you more closely mimic the buildd's so that the results of your local builds will be as close as possible to those used to build Ubuntu packages. Here is a list of helpful things to install. The general format is: |
Line 322: | Line 216: |
$ schroot -c natty-amd64-source -u root $ apt-get install package }}} You'll probably want to at least do both the ``amd64`` and ``i386`` platforms, and of course change the distroseries as appropriate. Here are some helpful packages that are not normally installed by default: * [[https://launchpad.net/pkgbinarymangler|pkgbinarymangler]] - performs a number of [[http://bazaar.launchpad.net/~ubuntu-core-dev/pkgbinarymangler/ubuntu/annotate/head:/pkgsanitychecks.1|sanity checks]] including important ones for Python packages. '''NOTE''' for it to strip translations, you'll need to adjust /etc/pkgbinarymangler/striptranslations.conf in the schroot. * [[https://launchpad.net/devscripts|devscripts]] - includes debuild which is useful on its own and required by UMT for building packages (this was included by default when using mk-sbuild until 12.04); do not use this if you want to reproduce Launchpad builds accurately * For yakkety: [[https://launchpad.net/apt-utils|apt-utils]] - to silence error message "debconf: delaying package configuration, since apt-utils is not installed" The version of mk-sbuild in maverick and newer supports adding additional packages when creating the schroots via either the {{{--debootstrap-include}}} command line argument or by adding a DEBOOTSTRAP_INCLUDE setting to the ~/.mk-sbuild.rc. Both methods take a comma-separated list of packages. Thus to always add pkgbinarymangler and devscripts when setting up the schroots, add:{{{ DEBOOTSTRAP_INCLUDE="pkgbinarymangler,devscripts" }}} to your ~/.mk-sbuild.rc file. |
$ schroot -c source:bionic-amd64 -u root -- apt-get install package }}} Here are some helpful packages that are not normally installed by default: * [[https://launchpad.net/pkgbinarymangler|pkgbinarymangler]] - performs a number of [[https://bazaar.launchpad.net/~ubuntu-core-dev/pkgbinarymangler/ubuntu/view/202/pkgsanitychecks.1|consistency checks]] including important ones for Python packages. '''NOTE''' for it to strip translations, you'll need to adjust /etc/pkgbinarymangler/striptranslations.conf in the schroot. * [[https://launchpad.net/devscripts|devscripts]] - includes debuild which is useful on its own and required by UMT for 'umt source' (this was included by default when using mk-sbuild until 12.04); do not use this if you want to reproduce Launchpad builds accurately ('sbuild' alone does not require debuild) * [[https://launchpad.net/apt-utils|apt-utils]] - to silence error message "debconf: delaying package configuration, since apt-utils is not installed" * [[https://launchpad.net/pkg-create-dbgsym|pkg-create-dbgsym]] - automatically build debug symbol ddeb packages. On Ubuntu 18.04 (bionic) and later, this is provided by 'debhelper' and doesn't need to be installed separately (though you may in While mk-sbuild supports adding additional packages when creating the schroots via either the {{{--debootstrap-include}}} command line argument or by adding a DEBOOTSTRAP_INCLUDE setting to the ~/.mk-sbuild.rc (both methods take a comma-separated list of packages), in practice this can lead to problems. Instead, create the chroot without these options and install packages in a separate step as described above. For typical security team build schroots on a bionic or later host, you may use: {{{ $ for rel in noble mantic jammy focal bionic xenial trusty; do for arch in amd64 i386 ; do echo "# $rel" schroot -c source:$rel-$arch -u root -- apt-get update schroot -c source:$rel-$arch -u root -- apt-get install -y --force-yes pkgbinarymangler apt-utils echo $rel | grep -qE '(precise|trusty|xenial)' && schroot -c source:$rel-$arch -u root -- apt-get install -y --force-yes pkg-create-dbgsym # only on <bionic # devscripts is required by 'umt source', but note devscripts could introduce differences from LP builds schroot -c source:$rel-$arch -u root -- apt-get install -y --force-yes devscripts schroot -c source:$rel-$arch -u root -- apt-get clean done done }}} |
Line 348: | Line 252: |
* In Ubuntu 13.04 (and possibly earlier) - rm the appropriate /etc/schroot/chroot.d/sbuild-* file * In earlier versions, edit /etc/schroot/schroot.conf to remove the stanza for the chroot |
* rm the appropriate /etc/schroot/chroot.d/sbuild-* file |
Line 351: | Line 254: |
$ sudo rm -rf /var/lib/schroot/chroots/lucid-amd64 | $ sudo rm -rf /var/lib/schroot/chroots/bionic-amd64 |
Line 354: | Line 257: |
$ sudo rm -rf /srv/devel/schroot/lucid-amd64 | $ sudo rm -rf /srv/devel/schroot/bionic-amd64 |
Line 362: | Line 265: |
$ schroot -c lucid-amd64 -u root (lucid-amd64)root@foo:/home/user# apt-get install ... (lucid-amd64)root@foo:/home/user# su user (lucid-amd64)user@foo$ exit |
$ schroot -c bionic-amd64 -u root (bionic-amd64)root@foo:/home/user# apt-get install ... (bionic-amd64)root@foo:/home/user# su user (bionic-amd64)user@foo$ exit |
Line 367: | Line 270: |
(lucid-amd64)root@foo:/home/user# exit | (bionic-amd64)root@foo:/home/user# exit |
Line 373: | Line 276: |
$ schroot -c lucid-i386 -u root $ schroot -c lucid-armel -u root |
$ schroot -c bionic-i386 -u root $ schroot -c bionic-armhf -u root |
Line 378: | Line 281: |
$ schroot -c lucid-amd64 | $ schroot -c bionic-amd64 |
Line 388: | Line 291: |
$ sbuild -d lucid ../package_1.2.3-4.1.dsc }}} |
$ sbuild -d bionic ../package_1.2.3-4.1.dsc }}} The updated source file, debs and ddebs (when pkg-create-dbgsym or new enough debhelper (focal+) in the schroot) appear in `../`. |
Line 394: | Line 299: |
dapper-amd64 dapper-amd64-source dapper-i386 dapper-i386-source etch-amd64 etch-amd64-source hardy-amd64 hardy-amd64-source hardy-i386 hardy-i386-source |
chroot:bionic-amd64 chroot:bionic-i386 chroot:focal-amd64 chroot:focal-i386 chroot:precise-esm-amd64 |
Line 408: | Line 308: |
$ schroot -i -c hardy-amd64 ---- Chroot ---- Name hardy-amd64 Description hardy-amd64 |
$ schroot -i -c bionic-amd64 --- Chroot --- Name bionic-amd64 Description bionic-amd64 |
Line 413: | Line 313: |
Priority 3 | Message Verbosity normal |
Line 417: | Line 317: |
Root Groups root sbuild admin | Root Groups sbuild root admin |
Line 419: | Line 319: |
Preserve Environment true Default Shell |
|
Line 421: | Line 323: |
Script Configuration script-get-ddebs | Configuration Profile umt Script Configuration |
Line 425: | Line 328: |
Directory /srv/devel/schroot/hardy-amd64 Personality linux Filesystem union type aufs Filesystem union overlay directory /var/lib/schroot/union/overlay Filesystem union underlay directory /var/lib/schroot/union/underlay |
Directory /srv/devel/schroot/bionic-amd64 Personality undefined User Modifiable Keys Root Modifiable Keys User Data setup.config umt/config setup.copyfiles umt/copyfiles setup.fstab umt/fstab setup.nssdatabases umt/nssdatabases Filesystem Union Type aufs Filesystem Union Overlay Directory /srv/devel/schroot/overlay Filesystem Union Underlay Directory /var/lib/schroot/union/underlay |
Line 436: | Line 346: |
The '-source' chroot is the pristine chroot and you shouldn't go into it unless you want to change something for all future schroots into the chroot. Eg, if you wanted to always have 'vim' installed in your hardy-amd64 chroot, use:{{{ $ schroot -c hardy-amd64-source -u root |
The 'source:' (or '-source' in very old schroot) chroot is the pristine chroot and you shouldn't go into it unless you want to change something for all future schroots into the chroot. Eg, if you wanted to always have 'vim' installed in your bionic-amd64 chroot, use:{{{ $ schroot -c source:bionic-amd64 -u root |
Line 441: | Line 351: |
Now, any time you use the hardy-amd64 (ie, without '-source'), vim will be installed. | Now, any time you use the bionic-amd64 (ie, without 'source:'), vim will be installed. |
Line 446: | Line 356: |
skipped="edgy|feisty|gutsy|intrepid|jaunty" | # '|' separated list of prefixes to not update skipped="lucid|-shm(-|$)" |
Line 450: | Line 361: |
echo "Updating '$d'" schroot -q -c $d -u root -- sh -c 'apt-get -qq update && apt-get -qy dist-upgrade && apt-get clean' echo "" |
echo "Updating '$d'" schroot -q -c $d -u root -- sh -c 'apt-get -qq update && apt-get -qy -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confnew" dist-upgrade && apt-get clean' echo "" |
Line 457: | Line 368: |
15 6 * * * /home/<username>/bin/schroot_update | 15 6 * * * /home/<username>/bin/schroot_update |
Line 465: | Line 376: |
=== apt-cacher-ng within the schroots === If you've chosen to use apt-cacher-ng to provide package caching without a dedicated local mirror, you may wish to create the file /etc/apt/apt.conf.d/acng in all your chroots with the correct Proxy setting:{{{ |
=== Enabling caching within the schroots === ==== apt-cacher-ng ==== If you've chosen to use `apt-cacher-ng` to provide package caching without a dedicated local mirror, you may wish to create the file `/etc/apt/apt.conf.d/acng` in all your chroots with the correct Proxy setting:{{{ |
Line 471: | Line 385: |
cd /home/devel/schroot/ | cd /srv/devel/schroot/ |
Line 475: | Line 389: |
==== squid-deb-proxy ==== After setting up `squid-deb-proxy`, you have a configuration file in `/etc/apt/apt.conf.d/<config>` with `apt`'s proxying setting: {{{ Acquire::http::Proxy "http://127.0.0.1:8000"; }}} As this should be present in schroots too, run the following command: {{{ find /srv/devel/schroot/ -type d -name apt.conf.d -exec cp /etc/apt/apt.conf.d/<config> {} \; }}} Ensure that the proxy is accessible locally, with something like: {{{ sudo ufw allow in on lo to any proto tcp port 8000 comment 'squid-deb-proxy' }}} Also, if the private PPAs for ESM are going to be used as per the instructions below, ensure that access is allowed, using something like: {{{ echo private-ppa.launchpadcontent.net | sudo sh -c 'cat >/etc/squid-deb-proxy/mirror-dstdomain.acl.d/20-esm' }}} |
|
Line 476: | Line 408: |
Extra steps are required for setting up a chroot for Ubuntu 12.04 ESM. 1. Create the chroot:{{{ $ mk-sbuild precise --name=precise-esm --arch=amd64 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu }}} 1. Install the apt-transport-https package that's required to communicate with a private Launchpad PPA:{{{ $ schroot -c source:precise-esm-amd64 -u root -- apt-get install apt-transport-https }}} 1. Take note of the required information for accessing the PPA 1. Adjust https://launchpad.net/~<LP_USER>/+archivesubscriptions with your Launchpad username and go to the page 1. Find the {{{Extended Security Maintenance (ppa:ubuntu-esm/esm)}}} row and click the "view" link to the right |
Extra steps are required for setting up a chroot for ESM releases. Also Note: if your host is xenial the path to the archive keys is "/usr/share/keyrings/ubuntu-archive-keyring.gpg" and doesn't contain "removed" in the path. 1. Create the chroots:{{{ $ for rel in noble jammy focal bionic xenial trusty; do mk-sbuild $rel --name=$rel-esm --arch=amd64 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu $(echo $rel | grep -qE '(trusty|xenial)' && echo --debootstrap-keyring=/usr/share/keyrings/ubuntu-archive-removed-keys.gpg) --distro=ubuntu done }}} 1. (OPTIONAL): If setting up i386 esm chroots (bionic, xenial and trusty only):{{{ $ for rel in bionic xenial trusty; do mk-sbuild $rel --name=$rel-esm --arch=i386 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu $(echo $rel | grep -qE '(trusty|xenial)' && echo --debootstrap-keyring=/usr/share/keyrings/ubuntu-archive-removed-keys.gpg) --distro=ubuntu schroot -c source:$rel-esm-i386 -u root -- apt-get -y install apt-transport-https ca-certificates done }}} 1. Install the apt-transport-https and ca-certificates packages to communicate with a private Launchpad PPA:{{{ $ for rel in noble jammy focal bionic xenial trusty; do schroot -c source:$rel-esm-amd64 -u root -- apt-get -y install apt-transport-https ca-certificates done }}} 1. Take note of the required information for accessing the PPA (for 12.04 this is ppa:ubuntu-esm/esm and for 14.04+ this is ppa:ubuntu-esm/esm-infra-security) 1. Go to https://launchpad.net/people/+me/+archivesubscriptions 1. Find the {{{ESM Infrastructure Security (ppa:ubuntu-esm/esm-infra-security)}}} row and click the "view" link to the right |
Line 490: | Line 435: |
1. Repeat the last 2 steps for {{{ESM Apps Security (ppa:ubuntu-esm/esm-apps-security)}}}. 1. You'll need to configure the following repositories: * '''`trusty`''': `esm-infra-security` only * '''`xenial`''' and '''`bionic`''': both `esm-infra-security` and `esm-apps-security` * '''`focal`''' or newer: `esm-apps-security` only |
|
Line 493: | Line 443: |
$ schroot -c source:precise-esm-amd64 -u root -- touch /etc/apt/sources.list.d/esm-ppa.list $ schroot -c source:precise-esm-amd64 -u root -- chown root:sudo /etc/apt/sources.list.d/esm-ppa.list $ schroot -c source:precise-esm-amd64 -u root -- chmod 640 /etc/apt/sources.list.d/esm-ppa.list }}} 1. Add the apt sources lines to a new sources file (replace the echo'ed strings with your personalized apt sources lines): {{{ $ echo "deb ..." | schroot -c source:precise-esm-amd64 -u root -- tee -a /etc/apt/sources.list.d/esm-ppa.list $ echo "deb-src ..." | schroot -c source:precise-esm-amd64 -u root -- tee -a /etc/apt/sources.list.d/esm-ppa.list }}} |
$ for rel in noble jammy focal bionic xenial trusty; do schroot -c source:$rel-esm-amd64 -u root -- sh -c 'cd /etc/apt/sources.list.d; touch esm-ppa.list; chown root:sudo esm-ppa.list; chmod 640 esm-ppa.list' done }}} 1. Add the apt sources lines to a new sources file (enter your LP user and token from the steps above when prompted): {{{ $ read -p "PPA username: " user ; read -sep "ESM-Infra PPA token (hidden): " itoken ; echo \n ; read -sep "ESM-Apps PPA token (hidden): " atoken ; echo \n for rel in noble jammy focal bionic xenial trusty; do if echo $rel | grep -vqE 'trusty'; then # only trusty doesn't get apps echo "deb https://$user:$atoken@private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu $rel main" echo "deb-src https://$user:$atoken@private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu $rel main" fi | schroot -c source:$rel-esm-amd64 -u root -- tee -a /etc/apt/sources.list.d/esm-ppa.list | sed -e "s/$atoken/HIDDEN/" if echo $rel | grep -qE 'trusty|xenial|bionic'; then # focal and newer don't need infra echo "deb https://$user:$itoken@private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu $rel main" echo "deb-src https://$user:$itoken@private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu $rel main" fi | schroot -c source:$rel-esm-amd64 -u root -- tee -a /etc/apt/sources.list.d/esm-ppa.list | sed -e "s/$itoken/HIDDEN/" done ; unset user itoken atoken }}} |
Line 504: | Line 464: |
$ schroot -c source:precise-esm-amd64 -u root -- apt-key adv --recv-key --keyserver keyserver.ubuntu.com <PPA_ARCHIVE_KEY_ID> }}} |
$ for rel in noble jammy focal bionic xenial trusty; do schroot -c source:$rel-esm-amd64 -u root -- apt-key adv --recv-key --keyserver keyserver.ubuntu.com <PPA_ARCHIVE_KEY_ID> done }}} Note: See 3. 5. above for guidance on which PPA key IDs are needed for each release. |
Line 508: | Line 472: |
$ schroot -c source:precise-esm-amd64 -u root -- apt-get update $ schroot -c source:precise-esm-amd64 -u root -- apt-get dist-upgrade }}} You can now use the precise-esm-amd64 chroot to prepare source packages and perform local test builds using UMT, as documented below. |
$ for rel in noble jammy focal bionic xenial trusty; do schroot -c source:$rel-esm-amd64 -u root -- apt-get update schroot -c source:$rel-esm-amd64 -u root -- apt-get -y dist-upgrade done }}} 1. Install `pkgbinarymangler`, `apt-utils`, `pkg-create-dbgsym` and `devscripts` on esm schroots, too: {{{ $ for rel in noble jammy focal bionic xenial trusty; do echo "# $rel" schroot -c source:$rel-amd64 -u root -- apt-get install -y --force-yes pkgbinarymangler apt-utils echo $rel | grep -qE '(precise|trusty|xenial)' && schroot -c source:$rel-amd64 -u root -- apt-get install -y --force-yes pkg-create-dbgsym # only on <bionic # devscripts is required by 'umt source', but note devscripts could introduce differences from LP builds schroot -c source:$rel-amd64 -u root -- apt-get install -y --force-yes devscripts schroot -c source:$rel-amd64 -u root -- apt-get clean done }}} You can now use the ESM chroots to prepare source packages and perform local test builds using UMT, as documented below. It is recommended you also install the supplementary packages as described in 'Install some additional packages', above. |
Line 515: | Line 493: |
In an effort to both mimic the official buildd's and standardize building of packages, the Ubuntu Security team wrote UMT, a part of [[http://bazaar.launchpad.net/~ubuntu-security/ubuntu-security-tools/trunk|ubuntu-security-tools]]. This branch has helper scripts and tools for performing various tasks. Arguably the most important is UMT, as this is the primary tool the Ubuntu Security team uses for building packages. Much of this can be seen in the [[http://bazaar.launchpad.net/~ubuntu-security/ubuntu-security-tools/trunk/annotate/head:/README|README]] file in the branch. | In an effort to both mimic the official buildd's and standardize building of packages, the Ubuntu Security team wrote UMT (Uncomplicated Massive Tool), a part of [[https://git.launchpad.net/ubuntu-security-tools|ubuntu-security-tools]]. This branch has helper scripts and tools for performing various tasks. Arguably the most important is UMT, as this is the primary tool the Ubuntu Security team uses for building packages. Much of this can be seen in the [[https://git.launchpad.net/ubuntu-security-tools/tree/README|README]]. |
Line 521: | Line 499: |
* ../binary (relative to the toplevel source) will contain your new binary packages | * ../binary (relative to the toplevel source) will contain your new binary packages (and ddebs when pkg-create-dbgsym or a new enough debhelper is installed) |
Line 527: | Line 505: |
$ bzr branch lp:ubuntu-security-tools $ git clone lp:ubuntu-qa-tools }}} Add the UST and UQT environment variables to your startup scripts (eg ~/.bashrc) and have them point to your branches:{{{ export UST="$HOME/bzr-pulls/ubuntu-security-tools" |
$ git clone git+ssh://<USER>@git.launchpad.net/ubuntu-security-tools $ git clone git+ssh://<USER>@git.launchpad.net/ubuntu-qa-tools $ git clone git+ssh://<USER>@git.launchpad.net/ubuntu-cve-tracker }}} Add the UST, UCT and UQT environment variables to your startup scripts (eg ~/.bashrc) and have them point to your branches:{{{ export UST="$HOME/git-pulls/ubuntu-security-tools" |
Line 534: | Line 513: |
export UCT="$HOME/git-pulls/ubuntu-cve-tracker" | |
Line 538: | Line 518: |
}}} Link `$UQT` and `$UCT`.{{{ $ ln -s $UQT/common/lpl_common.py $UCT/scripts/lpl_common.py }}} Finally, to enable basic bash-completion support for ```umt``` add something like the following to your ```~/.bash_completion``` file: {{{ if which umt 1>/dev/null; then source $(dirname $(realpath $(which umt)))/umt-completion.bash fi |
|
Line 547: | Line 538: |
release_list="lucid precise quantal saucy trusty utopic" | release_list="trusty xenial bionic focal jammy mantic noble" |
Line 552: | Line 543: |
release_devel="utopic" # non-Ubuntu releases (used for schroots, VMs, etc) release_extras="unstable testing stretch jessie wheezy" |
release_devel="noble" # non-Ubuntu releases (used for schroots, VMs, etc) from Debian release_extras="buster bullseye stable unstable testing" |
Line 585: | Line 576: |
'''Note''': `umt changelog -r` expects `release_list` to be sorted by release date starting by older releases. In case of setting up the environment to also support ESM, ESM entries must be placed right after their base release. E.g. (release_list="trusty trusty/esm xenial esm-infra/xenial esm-apps/xenial bionic esm-infra/bionic esm-apps/bionic focal esm-apps/focal jammy esm-apps/jammy lunar mantic noble") '''Note 2''': `192.168.122.1` is the default IP address through which the guest OS can access the host one. |
|
Line 586: | Line 581: |
Line 592: | Line 588: |
==== apt Sources for Debian (For reference, do not add these if you ran the build-sources-list tool above.) ==== Add the following Debian repositories in the generated `/etc/apt/sources.list.d/ubuntu-security.list` file: {{{ # Buster deb-src http://deb.debian.org/debian buster main contrib non-free deb-src http://deb.debian.org/debian buster-updates main contrib non-free deb-src http://security.debian.org/ buster/updates main contrib non-free # Stable deb-src http://deb.debian.org/debian stable main contrib non-free non-free-firmware deb-src http://deb.debian.org/debian stable-updates main contrib non-free non-free-firmware deb-src http://security.debian.org/debian-security stable-security main contrib non-free non-free-firmware # Testing deb-src http://deb.debian.org/debian testing main non-free non-free-firmware # Unstable deb-src http://deb.debian.org/debian unstable main non-free non-free-firmware }}} ==== apt Sources for ESM ==== 1. Access your private subscriptions at [[https://launchpad.net/people/+me/+archivesubscriptions|https://launchpad.net/people/+me/+archivesubscriptions]]. 1. Search for the ESM Infrastructure Security PPA and click on `View` on the right. 1. Get your access token from one of the URLs specified in the code block. It should follow the username present in the HTTP basic authentication schema (i.e. between the characters `:` and `@`). 1. Repeat the second step with the ESM Apps Security PPA. 1. Create a `/etc/apt/auth.conf.d/esm-ppas.conf` file with restrictead read access, to protect private PPA credentials: {{{ $ sudo touch /etc/apt/auth.conf.d/esm-ppas.conf $ sudo chmod 600 /etc/apt/auth.conf.d/esm-ppas.conf }}} 1. Populate the contents of the `/etc/apt/auth.conf.d/esm-ppas.conf` file (ensuring its previous permissions are kept): {{{ machine private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu login <username> password <second_token> machine private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu login <username> password <second_token> }}} 1. Create a `/etc/apt/sources.list.d/esm-ppas.list` file with: {{{ deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu jammy main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu bionic main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu focal main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu xenial main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu trusty main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu jammy main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu bionic main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu focal main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu xenial main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu trusty main }}} ==== Securing and Updating apt ==== |
|
Line 594: | Line 648: |
$ sudo apt-key update | |
Line 596: | Line 649: |
$ sudo apt-key add /usr/share/keyrings/debian-archive-keyring.gpg | $ sudo cp /usr/share/keyrings/debian-archive-keyring.gpg /etc/apt/trusted.gpg.d/ $ curl -o ~/esm-ppa.asc 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xdbb1fc89762bf6b96707c4059bc0a1a1622cf918' $ sudo mv ~/esm-ppa.asc /etc/apt/trusted.gpg.d/ |
Line 607: | Line 662: |
# dput security:lucid ./*_source.changes | # dput security:bionic ./*_source.changes |
Line 613: | Line 668: |
# dput security-proposed:lucid ./*_source.changes | # dput security-proposed:bionic ./*_source.changes |
Line 678: | Line 733: |
$ umt download -r lucid <source package> | $ umt download -r bionic <source package> |
Line 710: | Line 765: |
* -s (skip pocket and release sanity checks -- useful when doing non-security builds) | * -s (skip pocket and release consistency checks -- useful when doing non-security builds) |
Line 831: | Line 886: |
In general, if there is a problem with the upload, the uploader will be sent an email. If there is a problem with verifying the signature on the package, an email will not be sent. If this happens, verify that launchpad has the most up to date version of your key and if it doesn't, update the key (ie, `gpg --keyserver keyserver.ubuntu.com --send-key <ID>`). |
|
Line 838: | Line 895: |
$ sudo chown -R <username>:<username> /var/www/html/debs | $ sudo chown -R $USER:$USER /var/www/html/debs |
Line 847: | Line 904: |
Changelog release is: lucid Release is: lucid |
Changelog release is: bionic Release is: bionic |
Line 853: | Line 910: |
Repository: /var/www/debs/testing/lucid | Repository: /var/www/debs/testing/bionic |
Line 861: | Line 918: |
/var/www/debs/testing/lucid updated | /var/www/debs/testing/bionic updated |
Line 864: | Line 921: |
deb http://192.168.122.1/debs/testing/ lucid/ deb-src http://192.168.122.1/debs/testing/ lucid/ |
deb http://192.168.122.1/debs/testing/ bionic/ deb-src http://192.168.122.1/debs/testing/ bionic/ |
Line 872: | Line 929: |
* binaries are copied to the release directory under ```package_tools_repo_base```. Eg, in this example running ```umt repo``` on a package with ```lucid-security``` as the distribution name results in the binaries being copied to /var/www/html/debs/testing/lucid. '''Note''': prior to Ubuntu 14.04 LTS, the default !DocumentRoot is /var/www instead of /var/www/html | * binaries are copied to the release directory under ```package_tools_repo_base```. Eg, in this example running ```umt repo``` on a package with ```bionic-security``` as the distribution name results in the binaries being copied to /var/www/html/debs/testing/bionic. '''Note''': prior to Ubuntu 14.04 LTS, the default !DocumentRoot is /var/www instead of /var/www/html |
Line 895: | Line 952: |
Please note, this requires a [[SecurityTeam/UbuntuCVETracker|configured ubuntu-cve-tracker]] to work. | Please note, this requires a [[SecurityTeam/UbuntuCVETracker|configured ubuntu-cve-tracker]] to work. You will also need to install several python packages:{{{ $ apt install python-progressbar python-apt python-launchpadlib python-configobj }}} |
Line 899: | Line 958: |
$ umt download -r lucid foo $ cd foo/lucid/foo-* # cd into the toplevel source |
$ umt download -r bionic foo $ cd foo/bionic/foo-* # cd into the toplevel source |
In an effort to recreate a build environment that closely mimics the Launchpad buildds, the Ubuntu Security team has defined the following steps. This is in many ways an extension of SbuildLVMHowto. Additional information may be found in TestingEnvironment and README.sbuild_setup. If you're not a Ubuntu Security Team member, you might want to just read the simpler, more generic instructions for setting up sbuild without bothering with the security team's workflow.
Read these instructions if you want to build packages for the next, in-development version of Ubuntu on the current stable release, e.g. you want to build Quantal packages on your Precise system. Hint: you need to install debootstrap from -backports first.
This guide does not impose a specific version of Ubuntu; the latter is entirely up to you.
Setting up and using Sbuild with ddebs
For build and package testing, x86_64 users are strongly encouraged to create chroots for both amd64 and i386, whereas i386 users need only to create i386 chroots. Note, as of Ubuntu 19.10, i386 support is limited
Note: schroot does not trivially work on zfs root file systems. On jammy it might be possible to pass --type=zfs-snapshot to mk-sbuild.
Creating the schroots
Install sbuild and schroot:
# optional: also install gnupg-agent or seahorse-agent $ sudo apt-get install sbuild schroot ubuntu-dev-tools moreutils
Note: for focal it will probably need to install ca-certificate, since it's not installed by default with schroot.
Make sure you are in the sbuild group. If you aren't, run the command above and re-login to your account:
$ sudo adduser $USER sbuild
OPTIONAL: If /var doesn't have enough space, create a big partition for your chroots:
$ sudo mkfs.ext4 /dev/sdXN $ sudo mkdir /srv/devel $ ls -l /dev/disk/by-uuid/|grep sdXN # find the UUID for the disk lrwxrwxrwx 1 root root 10 2010-04-22 12:14 d9300709-7b77-4fcc-8d95-77cecdef9e51 -> ../../sdXN
Add the partition to /etc/fstab (by UUID):
UUID=d9300709-7b77-4fcc-8d95-77cecdef9e51 /srv/devel ext3 defaults,relatime 0 0
Mount it:
$ sudo mount /srv/devel $ df -h|grep devel /dev/sdXN 166G 188M 157G 1% /srv/devel
Create the schroot directory:
$ sudo mkdir /srv/devel/schroot
Create ~/.mk-sbuild.sources.ubuntu (can use simply .mk-sbuild.sources and also .mk-sbuild.sources.debian) substituting a local mirror, a nearby mirror or the official archive for '<mirror>'. Eg:
deb http://<mirror>/ubuntu RELEASE main restricted universe multiverse deb-src http://<mirror>/ubuntu RELEASE main restricted universe multiverse #deb http://<mirror>/ubuntu RELEASE-updates main restricted universe multiverse #deb-src http://<mirror>/ubuntu RELEASE-updates main restricted universe multiverse deb http://<mirror>/ubuntu RELEASE-security main restricted universe multiverse deb-src http://<mirror>/ubuntu RELEASE-security main restricted universe multiverse
- Setup for mounting $HOME:
Create a directory for the umt schroot profile:
$ sudo mkdir /etc/schroot/umt
Copy the default profile's fstab:
$ sudo cp /etc/schroot/default/fstab /etc/schroot/umt/
Append to /etc/schroot/umt/fstab:
tmpfs /run/shm tmpfs defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0
If you use an encrypted home directory, your $HOME is mounted differently (eg /home/<username>/.Private is mounted on /home/<username>), so you will also have to add to /etc/schroot/mount-defaults:
/home/<username> /home/<username> none rw,bind 0 0
Create /etc/schroot/umt/config:
# Reuse copyfiles and nssdatabases from the default profile SETUP_COPYFILES="default/copyfiles" SETUP_NSSDATABASES="default/nssdatabases" # Do whatever the default script would do if [ -f "/etc/schroot/default/config" ]; then . /etc/schroot/default/config fi # Set our own fstab after the default script has been sourced SETUP_FSTAB="umt/fstab" # end script
Create ~/.sbuildrc:
# Mail address where logs are sent to (mandatory, no default!) $mailto = 'username'; # Name to use as override in .changes files for the Maintainer: field # (mandatory, no default!). $maintainer_name='Your Name <user@ubuntu.com>'; # Directory for chroot symlinks and sbuild logs. Defaults to the # current directory if unspecified. (Deprecated.) Leave this unset; # umt compare-bin relies upon this being unset. #$build_dir='/home/<username>/ubuntu/build'; # Directory for writing build logs to $log_dir="/home/<username>/ubuntu/logs"; # Override default sbuild dependency resolver (see 'man sbuild'). The default # resolver (apt) mostly works ok but not always (eg, oneiric libreoffice). # Use 'apt', 'aptitude', 'internal'. Can also use '--build-dep-resolver' with # sbuild or '--sbuild-dep-resolver' with umt. #$build_dep_resolver="apt"; # don't remove this, Perl needs it: 1;
Then make the following directories (change if specified something different in ~/.sbuildrc):
$ mkdir -p $HOME/ubuntu/logs $ mkdir -p $HOME/ubuntu/build # only if the $build_dir variable is set in ~/.sbuildrc
Create ~/.mk-sbuild.rc:
# mk-sbuild build tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_PROFILE="umt" SCHROOT_CONF_SUFFIX="source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin preserve-environment=true" SKIP_UPDATES="1" SKIP_PROPOSED="1" # devscripts no longer installs correctly, so install it manually later DEBOOTSTRAP_INCLUDE=""
What this does is tells schroot that the chroots are in "/srv/devel/schroot", and to append to the configuration of the chroot in /etc/schroot/schroot.conf the contents of SCHROOT_CONF_SUFFIX.IMPORTANT: The Ubuntu Security Team uses umt to build packages. umt automatically enables -proposed when building packages in for the current dev release. If you are not using umt to build packages, you'll need to determine if you should use SKIP_PROPOSED="1" in your ~/.mk-sbuild.rc.
IMPORTANT: If you are using overlayfs, keep in mind that the writable overlay is in /var/lib/schroot/union/overlay by default (man 1 schroot). This means your build's compile while be on this partition and if there isn't enough space, your builds will fail. To change this to somewhere with more space, create an overlay directory (eg, sudo mkdir -p /srv/devel/schroot/overlay), then adjust your existing schroots to have 'union-overlay-directory=/srv/devel/schroot/overlay'. To use this by default for all schroots, modify ~/.mk-sbuild.rc to have 'union-overlay-directory=/srv/devel/schroot/overlay' somewhere in SCHROOT_CONF_SUFFIX.
Install Sendmail: sudo apt install sendmail
Finally, create the schroots (this assumes x86_64 host; for i386, omit amd64):
$ sg sbuild # only needed if you were added to the sbuild group during this session $ for i in noble mantic jammy focal bionic xenial trusty; do mk-sbuild $i --arch=amd64 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu if echo $i | grep -qE '(focal|jammy)'; then # libeatmydata1 not available in focal and jammy on i386 mk-sbuild $i --arch=i386 --skip-updates --skip-eatmydata --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu else mk-sbuild $i --arch=i386 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu fi done $ exit # only needed if you used sg sbuild earlier; sg changes primary group
If you want debian schroots:
$ for i in buster bullseye stable testing unstable ; do mk-sbuild $i --arch=amd64 --skip-updates --distro=debian done
For arm (see ARM/RootfsFromScratch for more details) :
$ sudo apt-get install qemu-user-static $ mk-sbuild xenial --arch=armhf --skip-updates --distro=ubuntu
See man mk-sbuild for details.
Note 1: Debian schroots pull in exim4-base but Ubuntu systems do not. Due to Debian bug #565613 the passwd and group databases are pulled in from the host, overwriting what is in the chroot. Since Ubuntu by default does not have exim4-base installed, the user isn't there and you will end up with errors like this when updating the schroot at a later date: dpkg: syntax error: unknown group Debian-exim' in statusoverride file`. You can either create the user/group on the Ubuntu host or remove the exim4-base package from the schroot (this may not work with older Debian releases):
$ schroot -u root -c source:sid-amd64 -- apt-get remove --purge -y --force-yes exim4-base $ schroot -u root -c source:sid-amd64 -- cp /var/lib/dpkg/statoverride-old /var/lib/dpkg/statoverride # remove Debian-exim line
Note 2: apt is configured to download the package translations files by default. These can be a source of build failures if they get corrupted and can cause apt-get update to take longer; therefore it can be useful to configure to disable downloading the translations files. To do so, do e.g.:
$ schroot -c source:trusty-i386 -u root <<EOM echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/99-no-translations EOM
Note 3: yakkety and newer may fail with devscripts dependency errors such as
W: Failure while configuring base packages. This will be re-attempted up to five times. W: See /srv/devel/schroot/eoan-i386/debootstrap/debootstrap.log for details (possibly the package libnet-ssleay-perl is at fault)
or error in /srv/devel/schroot/eoan-i386/debootstrap/debootstrap.log:
dpkg: error processing package devscripts (--configure): dependency problems - leaving unconfigured Errors were encountered while processing: libnet-ssleay-perl libio-socket-ssl-perl liblwp-protocol-https-perl libwww-perl devscripts
If so, then be sure to check this line DEBOOTSTRAP_INCLUDE="devscripts" in your ~/.mk-sbuild.rc is commented out. This then requires that for each schroot you need to go and purposely install devscripts in it, otherwise you'll notice packages fail to build. Eg
$ schroot -c source:bionic-amd64 -u root -- apt-get install devscripts
Note 4: if using foreign architectures in a chroot in a container, be sure that qemu-user-static is installed on the host and in the container so that /proc/sys/fs/binfmt_misc/qemu-arm from the host (which is bind mounted in the guest) correctly points to /usr/bin/qemu-arm-static. If you don't do this you might encoutner cryptic errors like "I: Running command: chroot /srv/devel/schroot/xenial-armhf /debootstrap/debootstrap --second-stage"
Note 5: Bionic and newer no longer support union-type=overlayfs . cd /etc/schroot/chroot.d && sed -i 's/overlayfs/overlay/' sbuild-* .
Optional (experimental): It can often be useful to perform builds in /dev/shm, which is a tmpfs directory in RAM. While there are limitations on what you can compile and the number of concurrent builds that can be performed (based on how much memory your build machine has), building in /dev/shm can lead to much faster build times and reduce disk I/O. One way to achieve this is to configure additional full schroots for this, and specify the 'union-overlay-directory' directive. Another way to create a 'shared' source schroot like so:
$ sudo sh -c "sed -e 's#]#-shm]#' -e 's#^\(directory=.*\)#\1\nunion-overlay-directory=/dev/shm/schroot/overlay#' /etc/schroot/schroot.conf > /etc/schroot/chroot.d/shm-overlays.conf" $ schroot --list
This simply adds additional schroots based on what you already have (eg, if you have a bionic-amd64 schroot, this creates a bionic-amd64-shm definition in /etc/schroot/chroot.d/shm-overlays.conf). The caveat is that each shm schroot will have the same 'directory' as its corresponding non-shm schroot, so you shouldn't try to modify both 'source:bionic-amd64' and 'source:bionic-amd64-shm' at the same time. Once the shm schroot definitions are added, reference them via schroot (or umt) like any other schroot. Eg:
$ schroot -c bionic-amd64-shm $ umt build -c bionic-amd64-shm
Please note that the directory you specify for 'union-overlay-directory' must exist before using the shm chroot (can add an entry to /etc/rc.local). See man schroot.conf for more details. It might also be useful to remount /dev/shm with more memory than the default, which is 50% of RAM. Can adjust like so: sudo mount -o remount,size=75% /dev/shm. See man mount for details.
Install some additional packages
There are a few packages that will help you more closely mimic the buildd's so that the results of your local builds will be as close as possible to those used to build Ubuntu packages. Here is a list of helpful things to install. The general format is:
$ schroot -c source:bionic-amd64 -u root -- apt-get install package
Here are some helpful packages that are not normally installed by default:
pkgbinarymangler - performs a number of consistency checks including important ones for Python packages. NOTE for it to strip translations, you'll need to adjust /etc/pkgbinarymangler/striptranslations.conf in the schroot.
devscripts - includes debuild which is useful on its own and required by UMT for 'umt source' (this was included by default when using mk-sbuild until 12.04); do not use this if you want to reproduce Launchpad builds accurately ('sbuild' alone does not require debuild)
apt-utils - to silence error message "debconf: delaying package configuration, since apt-utils is not installed"
pkg-create-dbgsym - automatically build debug symbol ddeb packages. On Ubuntu 18.04 (bionic) and later, this is provided by 'debhelper' and doesn't need to be installed separately (though you may in
While mk-sbuild supports adding additional packages when creating the schroots via either the --debootstrap-include command line argument or by adding a DEBOOTSTRAP_INCLUDE setting to the ~/.mk-sbuild.rc (both methods take a comma-separated list of packages), in practice this can lead to problems. Instead, create the chroot without these options and install packages in a separate step as described above.
For typical security team build schroots on a bionic or later host, you may use:
$ for rel in noble mantic jammy focal bionic xenial trusty; do for arch in amd64 i386 ; do echo "# $rel" schroot -c source:$rel-$arch -u root -- apt-get update schroot -c source:$rel-$arch -u root -- apt-get install -y --force-yes pkgbinarymangler apt-utils echo $rel | grep -qE '(precise|trusty|xenial)' && schroot -c source:$rel-$arch -u root -- apt-get install -y --force-yes pkg-create-dbgsym # only on <bionic # devscripts is required by 'umt source', but note devscripts could introduce differences from LP builds schroot -c source:$rel-$arch -u root -- apt-get install -y --force-yes devscripts schroot -c source:$rel-$arch -u root -- apt-get clean done done
Configure devscripts by creating a ~/.devscripts file with the following contents:
DEBSIGN_KEYID=<your keyid> BTS_SMTP_HOST=<smtp provider> BTS_SMTP_AUTH_USERNAME=<smtp user> BTS_SMTP_AUTH_PASSWORD=<password>
Deleting a schroot
- Remove the stanza for the chroot
- rm the appropriate /etc/schroot/chroot.d/sbuild-* file
Remove the chroot from the disk:
$ sudo rm -rf /var/lib/schroot/chroots/bionic-amd64
or if on a separate partition:
$ sudo rm -rf /srv/devel/schroot/bionic-amd64
Using the schroot
Using a schroot is similar to the chroot command but you specify the chroot and user you want to use rather than the directory.
$ schroot -c bionic-amd64 -u root (bionic-amd64)root@foo:/home/user# apt-get install ... (bionic-amd64)root@foo:/home/user# su user (bionic-amd64)user@foo$ exit exit (bionic-amd64)root@foo:/home/user# exit logout $
Chroot to the other architectures with:
$ schroot -c bionic-i386 -u root $ schroot -c bionic-armhf -u root
Or without root with:
$ schroot -c bionic-amd64
Or building via sbuild directly (see UMT, below, for the Ubuntu Security build scripts):
$ apt-get source foo $ cd ./foo-* ... do work ... $ dch -i $ update-maintainer $ debuild -S $ sbuild -d bionic ../package_1.2.3-4.1.dsc
The updated source file, debs and ddebs (when pkg-create-dbgsym or new enough debhelper (focal+) in the schroot) appear in ../.
Maintaining the schroots
Can see a listing of all your chroots with:
$ schroot -l chroot:bionic-amd64 chroot:bionic-i386 chroot:focal-amd64 chroot:focal-i386 chroot:precise-esm-amd64 ...
Can get information on your chroots with schroot -i [-c chroot]. Eg:
$ schroot -i -c bionic-amd64 --- Chroot --- Name bionic-amd64 Description bionic-amd64 Type directory Message Verbosity normal Users Groups sbuild root admin Root Users Root Groups sbuild root admin Aliases Preserve Environment true Default Shell Environment Filter ^(BASH_ENV|CDPATH|ENV|HOSTALIASES|IFS|KRB5_CONFIG|KRBCONFDIR|KRBTKFILE|KRB_CONF|LD_.*|LOCALDOMAIN|NLSPATH|PATH_LOCALE|RES_OPTIONS|TERMINFO|TERMINFO_DIRS|TERMPATH)$ Run Setup Scripts true Configuration Profile umt Script Configuration Session Managed true Session Cloned true Session Purged false Directory /srv/devel/schroot/bionic-amd64 Personality undefined User Modifiable Keys Root Modifiable Keys User Data setup.config umt/config setup.copyfiles umt/copyfiles setup.fstab umt/fstab setup.nssdatabases umt/nssdatabases Filesystem Union Type aufs Filesystem Union Overlay Directory /srv/devel/schroot/overlay Filesystem Union Underlay Directory /var/lib/schroot/union/underlay Source Users Source Groups Source Root Users root sbuild admin Source Root Groups root sbuild admin
The 'source:' (or '-source' in very old schroot) chroot is the pristine chroot and you shouldn't go into it unless you want to change something for all future schroots into the chroot. Eg, if you wanted to always have 'vim' installed in your bionic-amd64 chroot, use:
$ schroot -c source:bionic-amd64 -u root $ apt-get install vim $ exit
Now, any time you use the bionic-amd64 (ie, without 'source:'), vim will be installed.
It is also useful to keep your schroots up to date via cron. This can be done by creating $HOME/bin/schroot_update:
# '|' separated list of prefixes to not update skipped="lucid|-shm(-|$)" for d in `schroot -l | grep -- '\(^source:\|-source$\)' | egrep -v "($skipped)"` do echo "Updating '$d'" schroot -q -c $d -u root -- sh -c 'apt-get -qq update && apt-get -qy -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confnew" dist-upgrade && apt-get clean' echo "" done
Then adding a crontab entry for your user:
15 6 * * * /home/<username>/bin/schroot_update
Expiring active schroot sessions
Sometimes needed if schroots are left hanging around due to a crash or reboot. You can expire all active schroot sessions with (see also /etc/default/schroot):
$ schroot -e --all-sessions # see 'man schroot' for details
Enabling caching within the schroots
apt-cacher-ng
If you've chosen to use apt-cacher-ng to provide package caching without a dedicated local mirror, you may wish to create the file /etc/apt/apt.conf.d/acng in all your chroots with the correct Proxy setting:
Acquire::http { Proxy "http://127.0.0.1:3142"; };
This can be done with a shell script similar to this:
cd /srv/devel/schroot/ for d in * ; do cp /etc/apt/apt.conf.d/acng $d/etc/apt/apt.conf.d/ ; done
squid-deb-proxy
After setting up squid-deb-proxy, you have a configuration file in /etc/apt/apt.conf.d/<config> with apt's proxying setting:
Acquire::http::Proxy "http://127.0.0.1:8000";
As this should be present in schroots too, run the following command:
find /srv/devel/schroot/ -type d -name apt.conf.d -exec cp /etc/apt/apt.conf.d/<config> {} \;
Ensure that the proxy is accessible locally, with something like:
sudo ufw allow in on lo to any proto tcp port 8000 comment 'squid-deb-proxy'
Also, if the private PPAs for ESM are going to be used as per the instructions below, ensure that access is allowed, using something like:
echo private-ppa.launchpadcontent.net | sudo sh -c 'cat >/etc/squid-deb-proxy/mirror-dstdomain.acl.d/20-esm'
Setting up a chroot for Extended Security Maintenance
Extra steps are required for setting up a chroot for ESM releases. Also Note: if your host is xenial the path to the archive keys is "/usr/share/keyrings/ubuntu-archive-keyring.gpg" and doesn't contain "removed" in the path.
Create the chroots:
$ for rel in noble jammy focal bionic xenial trusty; do mk-sbuild $rel --name=$rel-esm --arch=amd64 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu $(echo $rel | grep -qE '(trusty|xenial)' && echo --debootstrap-keyring=/usr/share/keyrings/ubuntu-archive-removed-keys.gpg) --distro=ubuntu done
(OPTIONAL): If setting up i386 esm chroots (bionic, xenial and trusty only):
$ for rel in bionic xenial trusty; do mk-sbuild $rel --name=$rel-esm --arch=i386 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu $(echo $rel | grep -qE '(trusty|xenial)' && echo --debootstrap-keyring=/usr/share/keyrings/ubuntu-archive-removed-keys.gpg) --distro=ubuntu schroot -c source:$rel-esm-i386 -u root -- apt-get -y install apt-transport-https ca-certificates done
Install the apt-transport-https and ca-certificates packages to communicate with a private Launchpad PPA:
$ for rel in noble jammy focal bionic xenial trusty; do schroot -c source:$rel-esm-amd64 -u root -- apt-get -y install apt-transport-https ca-certificates done
- Take note of the required information for accessing the PPA (for 12.04 this is ppa:ubuntu-esm/esm and for 14.04+ this is ppa:ubuntu-esm/esm-infra-security)
Go to https://launchpad.net/people/+me/+archivesubscriptions
Find the ESM Infrastructure Security (ppa:ubuntu-esm/esm-infra-security) row and click the "view" link to the right
- You'll need the apt sources lines as well as the key ID of the PPA archive key
Repeat the last 2 steps for ESM Apps Security (ppa:ubuntu-esm/esm-apps-security).
- You'll need to configure the following repositories:
trusty: esm-infra-security only
xenial and bionic: both esm-infra-security and esm-apps-security
focal or newer: esm-apps-security only
- Restrict file access to the ESM apt sources list file prior to writing your credentials to the file
$ for rel in noble jammy focal bionic xenial trusty; do schroot -c source:$rel-esm-amd64 -u root -- sh -c 'cd /etc/apt/sources.list.d; touch esm-ppa.list; chown root:sudo esm-ppa.list; chmod 640 esm-ppa.list' done
Add the apt sources lines to a new sources file (enter your LP user and token from the steps above when prompted):
$ read -p "PPA username: " user ; read -sep "ESM-Infra PPA token (hidden): " itoken ; echo \n ; read -sep "ESM-Apps PPA token (hidden): " atoken ; echo \n for rel in noble jammy focal bionic xenial trusty; do if echo $rel | grep -vqE 'trusty'; then # only trusty doesn't get apps echo "deb https://$user:$atoken@private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu $rel main" echo "deb-src https://$user:$atoken@private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu $rel main" fi | schroot -c source:$rel-esm-amd64 -u root -- tee -a /etc/apt/sources.list.d/esm-ppa.list | sed -e "s/$atoken/HIDDEN/" if echo $rel | grep -qE 'trusty|xenial|bionic'; then # focal and newer don't need infra echo "deb https://$user:$itoken@private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu $rel main" echo "deb-src https://$user:$itoken@private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu $rel main" fi | schroot -c source:$rel-esm-amd64 -u root -- tee -a /etc/apt/sources.list.d/esm-ppa.list | sed -e "s/$itoken/HIDDEN/" done ; unset user itoken atoken
Import the PPA key:
$ for rel in noble jammy focal bionic xenial trusty; do schroot -c source:$rel-esm-amd64 -u root -- apt-key adv --recv-key --keyserver keyserver.ubuntu.com <PPA_ARCHIVE_KEY_ID> done
- Note: See 3. 5. above for guidance on which PPA key IDs are needed for each release.
Synchronize the package index files and upgrade any outdated packages:
$ for rel in noble jammy focal bionic xenial trusty; do schroot -c source:$rel-esm-amd64 -u root -- apt-get update schroot -c source:$rel-esm-amd64 -u root -- apt-get -y dist-upgrade done
Install pkgbinarymangler, apt-utils, pkg-create-dbgsym and devscripts on esm schroots, too:
$ for rel in noble jammy focal bionic xenial trusty; do echo "# $rel" schroot -c source:$rel-amd64 -u root -- apt-get install -y --force-yes pkgbinarymangler apt-utils echo $rel | grep -qE '(precise|trusty|xenial)' && schroot -c source:$rel-amd64 -u root -- apt-get install -y --force-yes pkg-create-dbgsym # only on <bionic # devscripts is required by 'umt source', but note devscripts could introduce differences from LP builds schroot -c source:$rel-amd64 -u root -- apt-get install -y --force-yes devscripts schroot -c source:$rel-amd64 -u root -- apt-get clean done
You can now use the ESM chroots to prepare source packages and perform local test builds using UMT, as documented below. It is recommended you also install the supplementary packages as described in 'Install some additional packages', above.
Setting up and using UMT
In an effort to both mimic the official buildd's and standardize building of packages, the Ubuntu Security team wrote UMT (Uncomplicated Massive Tool), a part of ubuntu-security-tools. This branch has helper scripts and tools for performing various tasks. Arguably the most important is UMT, as this is the primary tool the Ubuntu Security team uses for building packages. Much of this can be seen in the README.
UMT has a number of commands to help make the build process repeatable and easy to follow. The main things to remember are that:
- all the commands are run in the toplevel source (except 'download')
- ../* (relative to the toplevel source) will contain the original source package
- ../source (relative to the toplevel source) will contain your new source package and debdiff
- ../binary (relative to the toplevel source) will contain your new binary packages (and ddebs when pkg-create-dbgsym or a new enough debhelper is installed)
../reports (relative to the toplevel source) will contain the debcompare output after running umt compare-bin
Setting up UMT
First, download the ubuntu-security-tools and ubuntu-qa-tools branches:
$ git clone git+ssh://<USER>@git.launchpad.net/ubuntu-security-tools $ git clone git+ssh://<USER>@git.launchpad.net/ubuntu-qa-tools $ git clone git+ssh://<USER>@git.launchpad.net/ubuntu-cve-tracker
Add the UST, UCT and UQT environment variables to your startup scripts (eg ~/.bashrc) and have them point to your branches:
export UST="$HOME/git-pulls/ubuntu-security-tools" export UQT="$HOME/git-pulls/ubuntu-qa-tools" export UCT="$HOME/git-pulls/ubuntu-cve-tracker"
Then add umt to your PATH.
$ ln -s $UST/build-tools/umt $HOME/bin/umt
Link $UQT and $UCT.
$ ln -s $UQT/common/lpl_common.py $UCT/scripts/lpl_common.py
Finally, to enable basic bash-completion support for umt add something like the following to your ~/.bash_completion file:
if which umt 1>/dev/null; then source $(dirname $(realpath $(which umt)))/umt-completion.bash fi
Make sure the DEBEMAIL and DEBFULLNAME environment variables are setup in your startup scripts (eg ~/.bashrc):
export DEBEMAIL='user@ubuntu.com' export DEBFULLNAME='Your Name'
Create ~/.ubuntu-security-tools.conf to have:
# list of all active releases (included devel) release_list="trusty xenial bionic focal jammy mantic noble" # name of the current devel release # see https://wiki.ubuntu.com/SecurityTeam/ReleaseCycle for schedule # to modify this variable release_devel="noble" # non-Ubuntu releases (used for schroots, VMs, etc) from Debian release_extras="buster bullseye stable unstable testing" # root of the sbuild log output (i.e. where $log_dir in ~/.sbuildrc is bind mounted in the chroot via /etc/schroot/mount-defaults) pkgbuild_logs="/home/<username>/ubuntu/logs" # path when building with ccache (bind mounted in the chroot via /etc/schroot/mount-defaults) pkgbuild_ccache="/scratch/ccache" # sets 'ulimit -v $pkgbuild_ulimit_v'. Set to "unlimited" for no limit. Some # builds require a limit (eg, gcc) and some prefer a much higher limit (eg # openoffice.org/libreoffice). If not set, umt will default to 'unlimited' # except for packages that are known to need a smaller value. #pkgbuild_ulimit_v="1024000" #pkgbuild_ulimit_v="unlimited" # the URL for the regular Ubuntu archive or mirror build_tools_sbuildmirror="http://archive.ubuntu.com/ubuntu" # currently used by check_source_packages package_tools_name="Your Name" package_tools_email="user@ubuntu.com" package_tools_type="security" package_tools_repo_url="http://192.168.122.1/debs/testing" package_tools_repo_base="/var/www/html/debs/testing" #package_tools_force_rmadison="yes" # currently used by ddput upload_tools_ddput_security="security:RELEASE"
Note: umt changelog -r expects release_list to be sorted by release date starting by older releases. In case of setting up the environment to also support ESM, ESM entries must be placed right after their base release. E.g. (release_list="trusty trusty/esm xenial esm-infra/xenial esm-apps/xenial bionic esm-infra/bionic esm-apps/bionic focal esm-apps/focal jammy esm-apps/jammy lunar mantic noble")
Note 2: 192.168.122.1 is the default IP address through which the guest OS can access the host one.
Setting up your apt sources
Your apt sources need to be adjusted to have deb-src lines for all supported releases (and Debian releases). A helper script in ubuntu-security-tools can be used for this:
$ $UST/build-tools/build-sources-list | sudo sh -c 'cat > /etc/apt/sources.list.d/ubuntu-security.list'
This will look at build_tools_sbuildmirror, release_list, release_devel and release_extras in $HOME/.ubuntu-security-tools.conf and generate the appropriate deb-src lines.
apt Sources for Debian (For reference, do not add these if you ran the build-sources-list tool above.)
Add the following Debian repositories in the generated /etc/apt/sources.list.d/ubuntu-security.list file:
# Buster deb-src http://deb.debian.org/debian buster main contrib non-free deb-src http://deb.debian.org/debian buster-updates main contrib non-free deb-src http://security.debian.org/ buster/updates main contrib non-free # Stable deb-src http://deb.debian.org/debian stable main contrib non-free non-free-firmware deb-src http://deb.debian.org/debian stable-updates main contrib non-free non-free-firmware deb-src http://security.debian.org/debian-security stable-security main contrib non-free non-free-firmware # Testing deb-src http://deb.debian.org/debian testing main non-free non-free-firmware # Unstable deb-src http://deb.debian.org/debian unstable main non-free non-free-firmware
apt Sources for ESM
Access your private subscriptions at https://launchpad.net/people/+me/+archivesubscriptions.
Search for the ESM Infrastructure Security PPA and click on View on the right.
Get your access token from one of the URLs specified in the code block. It should follow the username present in the HTTP basic authentication schema (i.e. between the characters : and @).
- Repeat the second step with the ESM Apps Security PPA.
Create a /etc/apt/auth.conf.d/esm-ppas.conf file with restrictead read access, to protect private PPA credentials:
$ sudo touch /etc/apt/auth.conf.d/esm-ppas.conf $ sudo chmod 600 /etc/apt/auth.conf.d/esm-ppas.conf
Populate the contents of the /etc/apt/auth.conf.d/esm-ppas.conf file (ensuring its previous permissions are kept):
machine private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu login <username> password <second_token> machine private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu login <username> password <second_token>
Create a /etc/apt/sources.list.d/esm-ppas.list file with:
deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu jammy main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu bionic main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu focal main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu xenial main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-apps-security/ubuntu trusty main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu jammy main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu bionic main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu focal main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu xenial main deb-src https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu trusty main
Securing and Updating apt
Then, setup secure apt:
$ sudo apt-get install ubuntu-keyring # may already be installed $ sudo apt-get install debian-archive-keyring $ sudo cp /usr/share/keyrings/debian-archive-keyring.gpg /etc/apt/trusted.gpg.d/ $ curl -o ~/esm-ppa.asc 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xdbb1fc89762bf6b96707c4059bc0a1a1622cf918' $ sudo mv ~/esm-ppa.asc /etc/apt/trusted.gpg.d/ $ sudo apt-get update
Setting up dput
Create/update ~/.dput.cf to have:
[DEFAULT] method = sftp login = <your Launchpad account name> # Soyuz security # dput security:bionic ./*_source.changes [security] fqdn = ppa.launchpad.net incoming = ~ubuntu-security/ppa/ubuntu/%(security)s # Soyuz security-proposed # dput security-proposed:bionic ./*_source.changes [security-proposed] fqdn = ppa.launchpad.net incoming = ~ubuntu-security-proposed/ppa/ubuntu/%(security-proposed)s # Soyuz esm # dput esm:precise ./*_source.changes [esm] fqdn = ppa.launchpad.net incoming = ~ubuntu-security/esm/ubuntu/%(esm)s # Soyuz security-staging # NOTE: ppa is public. can be used for test builds, dev release development # dput security-staging:precise ./*_source.changes [security-staging] fqdn = ppa.launchpad.net incoming = ~ubuntu-security/ubuntu-security-staging/ubuntu/%(security-staging)s # Soyuz security-staging-private # dput security-staging-private:precise ./*_source.changes [security-staging-private] fqdn = ppa.launchpad.net incoming = ~ubuntu-security/ubuntu-security-staging-private/ubuntu/%(security-staging-private)s
Using umt
umt basically works like so:
$ umt <command> <options>
You can see all the commands with umt --help. Eg:
$ umt help ... Uncomplicated Massive Tool (umt) umt COMMAND [OPTIONS] COMMAND: search List best source packages for each release download Get source packages and unpack changelog sdch-like new entry creation or extraction of changelog from prior release (dch-repeat-like) source Produces source package from current directory of unpacked source binary Produces binary packages and other outputs from ../source source build Do both above build-orig Temporarily builds the previous release to get the log file sign Sign the packages check Do a check-source-package compare-log Compare the build log against the prior version's log compare-bin Compare the binary debs against the prior version's binaries repo Copy all built packages into local repository upload Uploads with dput OPTIONS: type "umt COMMAND -h" to get a list of options for each command.
Download a package
To download packages for all releases:
$ umt download <source package>
For each release, this will put the source package in './<source package name>/<release>' and perform dpkg-source -x on it.
To download a package for a single release:
$ umt download -r bionic <source package>
Source build
$ umt source Package is: postgresql-8.1 Version is: 8.1.21-0ubuntu0.6.06 Version with epoch is: 8.1.21-0ubuntu0.6.06 Upstream version is: 8.1.21 Changelog release is: dapper Release is: dapper Pocket is: security Component is: main Need -sa for source build? False DEB_BUILD_OPTIONS: Dependency resolution: auto Chroot: dapper-amd64 Censored sources.list for main build Get:1 http://<mirror> dapper Release.gpg [189B] Hit http://<mirror> dapper Release ... dpkg-buildpackage: source only upload (original source is included) Copying files to '../source' Skipping debdiff (only one dsc found) SUCCESS: source package in '../source'
Some useful options for umt source are:
- --force-orig (forces inclusion of orig.tar.gz via use of -sa)
- -v VERSION (use changelog information from all versions strictly later than VERSION)
- -c CHROOT (specify the chroot to use instead of the default)
- -s (skip pocket and release consistency checks -- useful when doing non-security builds)
Binary build
$ umt binary Package is: postgresql-8.1 Version is: 8.1.21-0ubuntu0.6.06 Version with epoch is: 8.1.21-0ubuntu0.6.06 Upstream version is: 8.1.21 Changelog release is: dapper Release is: dapper Pocket is: security Component is: main Need -sa for source build? False DEB_BUILD_OPTIONS: Chroot: dapper-amd64 core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited sbuild command is: /usr/bin/sbuild --setup-hook /tmp/umt-ws7rLM -d dapper-amd64 -A postgresql-8.1_8.1.21-0ubuntu0.6.06.dsc Sessions still open, not unmounting sbuild (Debian sbuild) 0.59.0 (02 Aug 2009) on ... ... Built successfully /var/lib/schroot/mount/dapper-amd64-a5089ef6-62ff-48e9-9977-9613ca584e3c/build/jamie-postgresql-8.1_8.1.21-0ubuntu0.6.06-amd64-kQ6GiK -- Not removing build depends: cloned chroot in use
You can do a simple no-change rebuild (useful for examining the build log) with:
$ umt build -s [-c <chroot>]
You can also build the previous version (needed for log comparing) with:
$ umt build-orig
Signing a package
To sign your own package:
$ umt sign
To sign a sponsored package:
$ umt sign -k
Verifying a package
$ umt compare-log ... diff of last build log and current build opened in EDITOR ... $ umt compare-bin ... creates reports in ../reports using debcompare (from ubuntu-security-tools) ... $ umt check Running '<absolute path to>/security-tools/package-tools/check-source-package -s -T dapper-security -b ../binary ../source/elinks_0.10.6-1ubuntu3.4.dsc' ... Checking: elinks_0.10.6-1ubuntu3.4.dsc Mode: security Debdiff: elinks_0.10.6-1ubuntu3.4.debdiff Retrieving madison output: pass Binary build: pass PGP (verify dsc): pass PGP (verify source.changes): pass PGP (signature present): pass Distribution: pass Pocket: pass Maintainer: pass Changed-By: pass Source has orig.tar.gz: pass Recent date: pass Output of diffstat: elinks-0.10.6/debian/changelog | 16 ++++++++++++++++ elinks-0.10.6/src/intl/charsets.c | 24 ++++++++++++------------ elinks-0.10.6/src/protocol/smb/smb.c | 7 +++++++ 3 files changed, 35 insertions(+), 12 deletions(-) Version: pass Newest version: pass Current: 0.10.6-1ubuntu3.4 Found: 0.10.6-1ubuntu3.3 Diff chaff: pass Merged changelog: ignored Patch system: patchless? (skipped patch system checks) Patch tagging: pass (no patches) Check existing binaries: found existing: elinks dapper-security found existing: elinks-lite dapper-security/universe Total: 2 (new=0, existing=2) Reverse Debdiff: pass Bad files: pass Vcs files in source: pass ------ PASS
Uploading a package
$ umt upload Source is: elinks Version is: 0.10.6-1ubuntu3.4 Version with epoch is: 0.10.6-1ubuntu3.4 Distribution is: dapper Pocket is: security Destination is: ubuntu Validating elinks version 0.10.6-1ubuntu3.4 ... ...
In general, if there is a problem with the upload, the uploader will be sent an email. If there is a problem with verifying the signature on the package, an email will not be sent. If this happens, verify that launchpad has the most up to date version of your key and if it doesn't, update the key (ie, gpg --keyserver keyserver.ubuntu.com --send-key <ID>).
Local apt repository
It is often useful to test the binaries before uploading to the security queue (after which, those binaries should be tested fully). UMT provides some functionality for copying the newly built binaries into a local repository. To use:
install the necessary software:
$ sudo apt-get install apt-utils moreutils apache2
create a directory in /var/www/html, writable by you:
$ sudo mkdir -p /var/www/html/debs/testing $ sudo chown -R $USER:$USER /var/www/html/debs
Now, after building your binaries you can copy them into the local apt repo:
$ umt repo Package is: hello Version is: 2.4-3ubuntu1 Version with epoch is: 2.4-3ubuntu1 Upstream version is: 2.4 Changelog release is: bionic Release is: bionic Pocket is: security Component is: main Need -sa for source build? False DEB_BUILD_OPTIONS: Repository: /var/www/debs/testing/bionic Copying 'hello_2.4-3ubuntu1_i386.deb'... You need a passphrase to unlock the secret key for user: "Your Name <user@ubuntu.com>" 1024-bit DSA key, ... -- /var/www/debs/testing/bionic updated Use: deb http://192.168.122.1/debs/testing/ bionic/ deb-src http://192.168.122.1/debs/testing/ bionic/ -- SUCCESS
The output shows the progress of the command as well as the appropriate lines to add to a sources.list file to apt-get the packages. Keep in mind:
- this will copy all binaries found in ../binary relative to your toplevel source (be careful if you compiled i386 and amd64 binaries at different times to be sure you are testing what you thing you are testing)
binaries are copied to the release directory under package_tools_repo_base. Eg, in this example running umt repo on a package with bionic-security as the distribution name results in the binaries being copied to /var/www/html/debs/testing/bionic. Note: prior to Ubuntu 14.04 LTS, the default DocumentRoot is /var/www instead of /var/www/html
umt repo is not particularly smart -- it will overwrite packages of the same version and leave everything else in the local repo alone. It is often useful to clean out your local repository with something like:
$ rm -rf /var/www/debs/testing/*/*
The update_repo command from ubuntu-security-tools/repo-tools can also be used to update the local repository information (eg Sources, Packages, etc).
Building sources with packages in your local repository
Say that package bar depends on a particular version of package foo, and they need to be updated together. Build foo like normal, then use umt sign && umt repo to put foo into your local repository. Then when building bar, add --add-repo to the build command (eg, umt build --add-repo) and your local repository will be added to the build and bar can be built with your local foo. (Here is an alternative/related approach.)
For this to work, you should add your GPG key that is used to sign your packages to your schroots. Eg:
for rel in lucid precise quantal saucy trusty utopic ; do for arch in amd64 i386 ; do gpg --armor --export <your key id> | schroot -c source:$rel-$arch -u root apt-key add - done done
copy_sppa_to_repos
The ubuntu-security-tools branch also has a script for copying files from LP into your local apt repository, which is very useful for testing the binaries people will actually install. To use:
$ $UST/repo-tools/copy_sppa_to_repos srcpkg1 srcpkg2
Please note, this requires a configured ubuntu-cve-tracker to work. You will also need to install several python packages:
$ apt install python-progressbar python-apt python-launchpadlib python-configobj
Typical package build procedure
$ umt download -r bionic foo $ cd foo/bionic/foo-* # cd into the toplevel source $ umt changelog # runs 'dch -i' with appropriate arguments for security updates ... perform your patching, etc ... $ umt build # runs 'umt source' followed by 'umt binary' $ umt build-orig # builds the prior version, for comparing with your new version $ umt compare-log $ umt compare-bin ... view compare-bin results in ../reports/ ... $ umt check $ umt repo ... in a VM, apt-get packages from local repository to test the packages ... $ umt sign $ umt upload
ESM package build procedure
$ umt download -r precise/esm foo $ cd foo/precise_esm/foo-* # cd into the toplevel source $ umt changelog # runs 'dch -i' with appropriate arguments for security updates ... perform your patching, etc ... $ umt build -c precise-esm-amd64 # runs 'umt source' followed by 'umt binary' $ umt build-orig -sr precise/esm -c precise-esm-amd64 # builds the prior version, for comparing with your new version $ umt compare-log # if the previous binaries to compare against are in the the Precise archive: $ umt compare-bin # otherwise, if the previous binaries to compare against are in ppa:ubuntu-esm/esm: $ umt compare-bin --prev-ppa=ubuntu-esm/esm ... view compare-bin results in ../reports/ ... $ umt check $ umt repo ... in a VM, apt-get packages from local repository to test the packages ... $ umt sign $ umt upload -d esm:precise
Using UMT with git-buildpackage
UMT can also easily be used with packages that are maintained with git-buildpackage. You can run the commands which build packages using gbp's --git-builder option.
Note: mariadb-5.5 and mariadb-10.0 are examples of packages that are maintained using gbp. Full instructions on sponsoring MariaDB security uploads are available here.
To perform a source build:
$ gbp --git-builder='umt source'
To perform a full source and binary build:
$ gbp --git-builder='umt build'
SecurityTeam/BuildEnvironment (last edited 2024-07-02 14:39:12 by pfsmorigo)