BuildEnvironment
46
Comment:
|
31287
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
Describe SecurityTeam/BuildEnvironment here. | ||<tablestyle="float:right; font-size: 0.9em; width:30%; background:#F1F1ED; background-repeat: no-repeat; background-position: 98% 0.5ex; margin: 0 0 1em 1em; padding: 0.5em;"><<TableOfContents>>|| In an effort to recreate a build environment that closely mimics the Launchpad buildds, the Ubuntu Security team has defined the following steps. This is in many ways an extension of [[https://help.ubuntu.com/community/SbuildLVMHowto|SbuildLVMHowto]]. Additional information may be found in [[http://bazaar.launchpad.net/~ubuntu-security/ubuntu-security-tools/trunk/annotate/head:/README.sbuild_setup|README.sbuild_setup]]. '''Read these [[https://wiki.ubuntu.com/DebootstrapChroot|instructions]]''' if you want to build packages for the next, in-development version of Ubuntu on the current stable release, e.g. you want to build Quantal packages on your Precise system. '''Hint:''' you need to install {{{debootstrap}}} from -backports first. == Setting up and using Sbuild with ddebs == By default when using schroot, mk-sbuild in Lucid uses 'union-type=aufs' while in Precise and later mk-sbuild uses 'union-type=overlayfs'. Also, mk-sbuild passes --variant=buildd to debootstrap so you don't have to. Due to performance issues with ext4 in Ubuntu 10.04 LTS, it is recommended that you use ext3 for your builds (feel free to use ext4 in later releases). For build and package testing, x86_64 users are strongly encouraged to create chroots for both amd64 and i386, whereas i386 users need only to create i386 chroots. '''Note:''' Precise and later no longer supports aufs so after upgrading to Precise, you will have to adjust your schroots to use 'overlayfs' (see below). === Creating the schroots === 0. Install sbuild and schroot:{{{ # optional: also install gnupg-agent or seahorse-agent $ sudo apt-get install sbuild schroot ubuntu-dev-tools }}} 0. Make sure you are in the 'sbuild' group:{{{ $ sudo adduser <username> sbuild }}} 0. '''OPTIONAL''': If /var doesn't have enough space, create a big partition for your chroots:{{{ $ sudo mkfs.ext3 /dev/sdXN $ sudo mkdir /srv/devel $ ls -l /dev/disk/by-uuid/|grep sdXN # find the UUID for the disk lrwxrwxrwx 1 root root 10 2010-04-22 12:14 d9300709-7b77-4fcc-8d95-77cecdef9e51 -> ../../sdXN }}} Add the partition to /etc/fstab (by UUID):{{{ UUID=d9300709-7b77-4fcc-8d95-77cecdef9e51 /srv/devel ext3 defaults,relatime 0 0 }}} Mount it:{{{ $ sudo mount /srv/devel $ df -h|grep devel /dev/sdXN 166G 188M 157G 1% /srv/devel }}} Create the schroot directory:{{{ $ sudo mkdir /srv/devel/schroot }}} 0. Create ~/.mk-sbuild.sources.ubuntu (can use simply .mk-sbuild.sources and also .mk-sbuild.sources.debian) substituting a local mirror, a nearby mirror or the official archive for '<mirror>'. Eg:{{{ deb http://<mirror>/ubuntu RELEASE main restricted universe multiverse deb-src http://<mirror>/ubuntu RELEASE main restricted universe multiverse #deb http://<mirror>/ubuntu RELEASE-updates main restricted universe multiverse #deb-src http://<mirror>/ubuntu RELEASE-updates main restricted universe multiverse deb http://<mirror>/ubuntu RELEASE-security main restricted universe multiverse deb-src http://<mirror>/ubuntu RELEASE-security main restricted universe multiverse }}} 0. Setup for mounting $HOME and extracting the ddebs: * Append to /etc/schroot/mount-defaults ('''IMPORTANT''': this file may be overwritten on upgrades of schroot):{{{ /home/<username>/ubuntu/scratch /scratch none rw,bind 0 0 }}} If you use an encrypted home directory, your $HOME is mounted differently (eg /home/<username>/.Private is mounted on /home/<username>), so you will also have to add to /etc/schroot/mount-defaults:{{{ /home/<username> /home/<username> none rw,bind 0 0 }}} * Create /etc/schroot/script-get-ddebs:{{{ # Do whatever the default script would do . /etc/schroot/default/config # Clean up during 00check if [ "x$0" = "x/etc/schroot/setup.d/00check" ]; then if [ $1 = "setup-stop" ]; then # Pull out .ddebs mv /var/cache/buildtmp/schroot/union/overlay/"$SESSION_ID"/build/*/*.ddeb /scratch/ddebs/ 2>/dev/null || true fi fi # end script }}} * Create the following directory:{{{ $ mkdir -p ~/ubuntu/scratch/ddebs }}} 0. Create ~/.sbuildrc:{{{ # Mail address where logs are sent to (mandatory, no default!) $mailto = 'username'; # Name to use as override in .changes files for the Maintainer: field # (mandatory, no default!). $maintainer_name='Your Name <user@ubuntu.com>'; # Directory for chroot symlinks and sbuild logs. Defaults to the # current directory if unspecified. #$build_dir='/home/<username>/ubuntu/build'; # Directory for writing build logs to $log_dir="/home/<username>/ubuntu/logs"; # Override default sbuild dependency resolver (see 'man sbuild'). The default # resolver (apt) mostly works ok but not always (eg, oneiric libreoffice). # Use 'apt', 'aptitude', 'internal'. Can also use '--build-dep-resolver' with # sbuild or '--sbuild-dep-resolver' with umt. #$build_dep_resolver="apt"; # don't remove this, Perl needs it: 1; }}} Then make the following directories (change if specified something different in ~/.sbuildrc):{{{ $ mkdir -p $HOME/ubuntu/logs # $HOME/ubuntu/build (if used) }}} 0. Create ~/.mk-sbuild.rc: For schroot < 1.4.6-1 (Ubuntu 10.04 LTS and earlier):{{{ # mk-sbuild tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin" SKIP_UPDATES="1" }}} For schroot >= 1.4.6-1 (Ubuntu 10.10 and later):{{{ # mk-sbuild tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin preserve-environment=true" SKIP_UPDATES="1" }}} For ubuntu-dev-tools >= 0.138 (ie, 12.04 and later):{{{ # mk-sbuild build tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin preserve-environment=true" SKIP_UPDATES="1" DEBOOTSTRAP_INCLUDE="devscripts" }}} What this does is tells schroot that the chroots are in "/srv/devel/schroot", and to append to the configuration of the chroot in /etc/schroot/schroot.conf the contents of SCHROOT_CONF_SUFFIX. Using 'SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs"' means that /etc/schroot/script-get-ddebs gets called to configure the chroot. This script sources '/etc/schroot/default/config' so it behaves normally, then pulls the ddebs out and puts them into the bind mounted /scratch directory in the chroot (ie into ~/scratch). 0. Finally, create the schroots (this assumes x86_64 host; for i386, omit amd64):{{{ $ sg sbuild # only needed if you were added to the sbuild group during this session $ for i in hardy lucid natty oneiric precise quantal; do mk-sbuild $i --arch=amd64 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu mk-sbuild $i --arch=i386 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu done for i in etch lenny squeeze sid ; do mk-sbuild $i --skip-updates --distro=debian done }}} For arm (see [[ARM/RootfsFromScratch|ARM/RootfsFromScratch]] for more details) :{{{ $ mk-sbuild --arch armel lucid }}} See ```man mk-sbuild``` for details. '''Note 1''': debootstrap in Natty will no longer install Dapper schroots because of the switch to SHA256. As a temporary work-around until Dapper goes EoL, you can downgrade to Maverick's debootstrap, create the Dapper schroots, and upgrade back. '''Note 2''': Oneiric (and higher?) schroots may not have the /var/lock directory which is expected in older versions of sbuild. To fix this:{{{ $ for a in i386 amd64 armel ; do schroot -c oneiric-$a-source -u root -- sh -c 'mkdir /var/lock ; chmod 1777 /var/lock' ; done }}} '''Note 3''': Oneiric and higher require you to run {{{sbuild-update --keygen}}} to generate a local archive key. This only needs to be done once and can be done as your user so long as you are in the 'sbuild' group. See manpage for details. (If you see this error when creating the keys 'gpg: can't create `(null)': Permission denied', then simply remove the existing keys in {{{/var/lib/sbuild/apt-keys/*}}} and try again.) 0. '''Optional (experimental)''': It can often be useful to perform builds in /dev/shm, which is a tmpfs directory in RAM. While there are limitations on what you can compile and the number of concurrent builds that can be performed (based on how much memory your build machine has), building in /dev/shm can lead to much faster build times and reduce disk I/O. One way to achieve this is to configure additional full schroots for this, and specify the 'union-overlay-directory' directive. Another way to create a 'shared' source schroot like so:{{{ $ sudo sh -c "sed -e 's#]#-shm]#' -e 's#^\(directory=.*\)#\1\nunion-overlay-directory=/dev/shm/schroot/overlay#' /etc/schroot/schroot.conf > /etc/schroot/chroot.d/shm-overlays.conf" $ schroot --list }}} This simply adds additional schroots based on what you already have (eg, if you have a lucid-amd64 schroot, this creates a lucid-amd64-shm definition in /etc/schroot/chroot.d/shm-overlays.conf). The caveat is that each shm schroot will have the same 'directory' as its corresponding non-shm schroot, so you shouldn't try to modify both 'lucid-amd64-source' and 'lucid-amd64-shm-source' at the same time. Once the shm schroot definitions are add, reference them via schroot (or umt) like any other schroot. Eg: {{{ $ schroot -c lucid-amd64-shm $ umt build -c lucid-amd64-shm }}} Please note that the directory you specify for 'union-overlay-directory' must exist before using the shm chroot (can add an entry to /etc/rc.local). See ```man schroot.conf``` for more details. It might also be useful to remount /dev/shm with more memory than the default, which is 50% of RAM. Can adjust like so: ```sudo mount -o remount,size=75% /dev/shm```. See ```man mount``` for details. === Upgrading to Precise (12.04) === Aufs has been deprecated in 12.04 (but is still available). When upgrading your build host to 12.04, you are encouraged to transition to overlayfs: * Adjust all the aufs schroots to use overlayfs. Eg:{{{ $ sudo sed -i 's/aufs/overlayfs/' /etc/schroot/schroot.conf }}} Note that newer versions of mk-sbuild setup up the chroot configurations in ```/etc/schroot/chroot.d/``` (e.g. ```/etc/schroot/chroot.d/sbuild-precise-amd64```) so you may need to edit files there as well. * Adjust ~/.mk-sbuild.rc to have (devscripts no longer installed by default in 12.04 and later. Specify multiple packages via a comma-separated list):{{{ DEBOOTSTRAP_INCLUDE=devscripts }}} === Install some helpful packages === There are a few packages that will help you more closely mimic the buildd's so that the results of your local builds will be as close as possible to those used to build Ubuntu packages. Here are a list of helpful things to install. The general format is: {{{ $ schroot -c natty-amd64-source -u root $ apt-get install package }}} You'll probably want to at least do both the ``amd64`` and ``i386`` platforms, and of course change the distroseries as appropriate. Here are some helpful packages that are not normally installed by default: * [[https://edge.launchpad.net/pkgbinarymangler|pkgbinarymangler]] - performs a number of [[http://bazaar.launchpad.net/~ubuntu-core-dev/pkgbinarymangler/ubuntu/annotate/head:/pkgsanitychecks.1|sanity checks]] including important ones for Python packages. * [[https://launchpad.net/devscripts|devscripts]] - includes debuild which is useful on its own and required by UMT for building packages (this was included by default when using mk-sbuild until 12.04) The version of mk-sbuild in maverick and newer supports adding additional packages when creating the schroots via either the {{{--debootstrap-include}}} command line argument or by adding a DEBOOTSTRAP_INCLUDE setting to the ~/.mk-sbuild.rc. Both methods take a comma-separated list of packages. Thus to always add pkgbinarymangler and devscripts when setting up the schroots, add:{{{ DEBOOTSTRAP_INCLUDE="pkgbinarymangler,devscripts" }}} to your ~/.mk-sbuild.rc file. === Deleting a schroot === 0. Edit /etc/schroot/schroot.conf to remove the stanza for the chroot 0. Remove the chroot from the disk:{{{ $ sudo rm -rf /var/lib/schroot/chroots/lucid-amd64 }}} or if on a separate partition:{{{ $ sudo rm -rf /srv/devel/schroot/lucid-amd64 }}} === Using the schroot === Using a schroot is similar to the chroot command but you specify the chroot and user you want to use rather than the directory. {{{ $ schroot -c lucid-amd64 -u root (lucid-amd64)root@foo:/home/user# apt-get install ... (lucid-amd64)root@foo:/home/user# su user (lucid-amd64)user@foo$ exit exit (lucid-amd64)root@foo:/home/user# exit logout $ }}} Chroot to the other architectures with:{{{ $ schroot -c lucid-i386 -u root $ schroot -c lucid-armel -u root }}} Or without root with:{{{ $ schroot -c lucid-amd64 }}} Or building via ```sbuild``` directly (see UMT, below, for the Ubuntu Security build scripts):{{{ $ apt-get source foo $ cd ./foo-* ... do work ... $ dch -i $ update-maintainer $ debuild -S $ sbuild -d lucid ../package_1.2.3-4.1.dsc }}} === Maintaining the schroots === Can see a listing of all your chroots with:{{{ $ schroot -l dapper-amd64 dapper-amd64-source dapper-i386 dapper-i386-source etch-amd64 etch-amd64-source hardy-amd64 hardy-amd64-source hardy-i386 hardy-i386-source ... }}} Can get information on your chroots with ```schroot -i [-c chroot]```. Eg:{{{ $ schroot -i -c hardy-amd64 ---- Chroot ---- Name hardy-amd64 Description hardy-amd64 Type directory Priority 3 Users Groups sbuild root admin Root Users Root Groups root sbuild admin Aliases Environment Filter ^(BASH_ENV|CDPATH|ENV|HOSTALIASES|IFS|KRB5_CONFIG|KRBCONFDIR|KRBTKFILE|KRB_CONF|LD_.*|LOCALDOMAIN|NLSPATH|PATH_LOCALE|RES_OPTIONS|TERMINFO|TERMINFO_DIRS|TERMPATH)$ Run Setup Scripts true Script Configuration script-get-ddebs Session Managed true Session Cloned true Session Purged false Directory /srv/devel/schroot/hardy-amd64 Personality linux Filesystem union type aufs Filesystem union overlay directory /var/lib/schroot/union/overlay Filesystem union underlay directory /var/lib/schroot/union/underlay Source Users Source Groups Source Root Users root sbuild admin Source Root Groups root sbuild admin }}} The '-source' chroot is the pristine chroot and you shouldn't go into it unless you want to change something for all future schroots into the chroot. Eg, if you wanted to always have 'vim' installed in your hardy-amd64 chroot, use:{{{ $ schroot -c hardy-amd64-source -u root $ apt-get install vim $ exit }}} Now, any time you use the hardy-amd64 (ie, without '-source'), vim will be installed. It is also useful to keep your schroots up to date via cron. This can be done by creating $HOME/bin/schroot_update: {{{ #!/bin/sh skipped="edgy|feisty|gutsy|intrepid|jaunty" for d in `schroot -l | grep -- '-source$' | egrep -v "($skipped)"` do echo "Updating '$d'" schroot -q -c $d -u root -- sh -c 'apt-get -qq update && apt-get -qy dist-upgrade && apt-get clean' echo "" done }}} Then adding a crontab entry for your user:{{{ 15 6 * * * /home/<username>/bin/schroot_update }}} === Expiring active schroot sessions === Sometimes needed if schroots are left hanging around due to a crash or reboot. You can expire all active schroot sessions with (see also /etc/default/schroot):{{{ $ schroot -e --all-sessions # see 'man schroot' for details }}} == Setting up and using UMT == In an effort to both mimic the official buildd's and standardize building of packages, the Ubuntu Security team wrote UMT, a part of [[http://bazaar.launchpad.net/~ubuntu-security/ubuntu-security-tools/trunk|ubuntu-security-tools]]. This branch has helper scripts and tools for performing various tasks. Arguably the most important is UMT, as this is the primary tool the Ubuntu Security team uses for building packages. Much of this can be seen in the [[http://bazaar.launchpad.net/~ubuntu-security/ubuntu-security-tools/trunk/annotate/head:/README|README]] file in the branch. UMT has a number of commands to help make the build process repeatable and easy to follow. The main things to remember are that: * all the commands are run in the toplevel source (except 'download') * ../* (relative to the toplevel source) will contain the original source package * ../source (relative to the toplevel source) will contain your new source package and debdiff * ../binary (relative to the toplevel source) will contain your new binary packages * ../reports (relative to the toplevel source) will contain the debcompare output after running ```umt compare-bin``` === Setting up UMT === First, download the ubuntu-security-tools and ubuntu-qa-tools branches:{{{ $ bzr branch lp:ubuntu-security-tools $ bzr branch lp:ubuntu-qa-tools }}} Add the UST and UQT environment variables to your startup scripts (eg ~/.bashrc) and have them point to your branches:{{{ export UST="$HOME/bzr-pulls/ubuntu-security-tools" export UQT="$HOME/bzr-pulls/ubuntu-qa-tools" }}} Then add ```umt``` to your PATH.{{{ $ ln -s $UST/build-tools/umt $HOME/bin/umt }}} Make sure the DEBEMAIL and DEBFULLNAME environment variables are setup in your startup scripts (eg ~/.bashrc):{{{ export DEBEMAIL='user@ubuntu.com' export DEBFULLNAME='Your Name' }}} Create ~/.ubuntu-security-tools.conf to have:{{{ # list of all active releases (included devel) release_list="hardy lucid maverick natty oneiric precise" # name of the current devel release release_devel="precise" # non-Ubuntu releases (used for schroots, VMs, etc) release_extras="unstable unstable-i386 etch etch-i386" # root of the sbuild log output (i.e. where $log_dir in ~/.sbuildrc is bind mounted in the chroot via /etc/schroot/mount-defaults) pkgbuild_logs="/home/<username>/ubuntu/logs" # path when building with ccache (bind mounted in the chroot via /etc/schroot/mount-defaults) pkgbuild_ccache="/scratch/ccache" # sets 'ulimit -v $pkgbuild_ulimit_v'. Set to "unlimited" for no limit. May # break some builds (like openoffice.org), but is important for other # builds (like gcc). pkgbuild_ulimit_v="1024000" # the URL for the regular Ubuntu archive or mirror build_tools_sbuildmirror="http://archive.ubuntu.com/ubuntu" # currently used by check_source_packages package_tools_name="Your Name" package_tools_email="user@ubuntu.com" package_tools_type="security" package_tools_repo_url="http://192.168.122.1/debs/testing" package_tools_repo_base="/var/www/debs/testing" #package_tools_force_rmadison="yes" # currently used by ddput upload_tools_ddput_security="security:RELEASE" }}} === Setting up your apt sources === Your apt sources need to be adjusted to have deb-src lines for all supported releases (and Debian releases). A helper script in ubuntu-security-tools can be used for this:{{{ $ $UST/build-tools/build-sources-list | sudo sh -c 'cat > /etc/apt/sources.list.d/ubuntu-security.list' }}} This will look at ```build_tools_sbuildmirror```, ```release_list```, ```release_devel``` and ```release_extras``` in $HOME/.ubuntu-security-tools.conf and generate the appropriate deb-src lines. Then, setup secure apt: {{{ $ sudo apt-get install ubuntu-keyring # may already be installed $ sudo apt-key update $ sudo apt-get install debian-archive-keyring $ sudo apt-key add /usr/share/keyrings/debian-archive-keyring.gpg $ sudo apt-get update }}} === Setting up dput === Create/update ~/.dput.cf to have:{{{ # Soyuz security # dput security:lucid ./*_source.changes [security] fqdn = ppa.launchpad.net incoming = ~ubuntu-security/ubuntu/%(security)s login = anonymous # Soyuz security-proposed # dput security-proposed:lucid ./*_source.changes [security-proposed] fqdn = ppa.launchpad.net incoming = ~ubuntu-security-proposed/ubuntu/%(security-proposed)s login = anonymous }}} === Using umt === ```umt``` basically works like so:{{{ $ umt <command> <options> }}} You can see all the commands with ```umt --help```. Eg:{{{ $ umt help ... Uncomplicated Massive Tool (umt) umt COMMAND [OPTIONS] COMMAND: search List best source packages for each release download Get source packages and unpack changelog sdch-like new entry creation or extraction of changelog from prior release (dch-repeat-like) source Produces source package from current directory of unpacked source binary Produces binary packages and other outputs from ../source source build Do both above build-orig Temporarily builds the previous release to get the log file sign Sign the packages check Do a check-source-package compare-log Compare the build log against the prior version's log compare-bin Compare the binary debs against the prior version's binaries repo Copy all built packages into local repository upload Uploads with dput OPTIONS: type "umt COMMAND -h" to get a list of options for each command. }}} ==== Download a package ==== To download packages for all releases:{{{ $ umt download <source package> }}} For each release, this will put the source package in './<source package name>/<release>' and perform ```dpkg-source -x``` on it. To download a package for a single release:{{{ $ umt download -r lucid <source package> }}} ==== Source build ==== {{{ $ umt source Package is: postgresql-8.1 Version is: 8.1.21-0ubuntu0.6.06 Version with epoch is: 8.1.21-0ubuntu0.6.06 Upstream version is: 8.1.21 Changelog release is: dapper Release is: dapper Pocket is: security Component is: main Need -sa for source build? False DEB_BUILD_OPTIONS: Dependency resolution: auto Chroot: dapper-amd64 Censored sources.list for main build Get:1 http://<mirror> dapper Release.gpg [189B] Hit http://<mirror> dapper Release ... dpkg-buildpackage: source only upload (original source is included) Copying files to '../source' Skipping debdiff (only one dsc found) SUCCESS: source package in '../source' }}} Some useful options for ```umt source``` are: * --force-orig (forces inclusion of orig.tar.gz via use of -sa) * -v VERSION (use changelog information from all versions strictly later than VERSION) * -c CHROOT (specify the chroot to use instead of the default) * -s (skip pocket and release sanity checks -- useful when doing non-security builds) ==== Binary build ==== {{{ $ umt binary Package is: postgresql-8.1 Version is: 8.1.21-0ubuntu0.6.06 Version with epoch is: 8.1.21-0ubuntu0.6.06 Upstream version is: 8.1.21 Changelog release is: dapper Release is: dapper Pocket is: security Component is: main Need -sa for source build? False DEB_BUILD_OPTIONS: Chroot: dapper-amd64 core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited sbuild command is: /usr/bin/sbuild --setup-hook /tmp/umt-ws7rLM -d dapper-amd64 -A postgresql-8.1_8.1.21-0ubuntu0.6.06.dsc Sessions still open, not unmounting sbuild (Debian sbuild) 0.59.0 (02 Aug 2009) on ... ... Built successfully /var/lib/schroot/mount/dapper-amd64-a5089ef6-62ff-48e9-9977-9613ca584e3c/build/jamie-postgresql-8.1_8.1.21-0ubuntu0.6.06-amd64-kQ6GiK -- Not removing build depends: cloned chroot in use }}} You can do a simple no-change rebuild (useful for examining the build log) with:{{{ $ umt build -s [-c <chroot>] }}} You can also build the previous version (needed for log comparing) with:{{{ $ umt build-orig }}} ==== Signing a package ==== To sign your own package:{{{ $ umt sign }}} To sign a sponsored package:{{{ $ umt sign -k }}} ==== Verifying a package ==== {{{ $ umt compare-log ... diff of last build log and current build opened in EDITOR ... $ umt compare-bin ... creates reports in ../reports using debcompare (from ubuntu-security-tools) ... $ umt check Running '<absolute path to>/security-tools/package-tools/check-source-package -s -T dapper-security -b ../binary ../source/elinks_0.10.6-1ubuntu3.4.dsc' ... Checking: elinks_0.10.6-1ubuntu3.4.dsc Mode: security Debdiff: elinks_0.10.6-1ubuntu3.4.debdiff Retrieving madison output: pass Binary build: pass PGP (verify dsc): pass PGP (verify source.changes): pass PGP (signature present): pass Distribution: pass Pocket: pass Maintainer: pass Changed-By: pass Source has orig.tar.gz: pass Recent date: pass Output of diffstat: elinks-0.10.6/debian/changelog | 16 ++++++++++++++++ elinks-0.10.6/src/intl/charsets.c | 24 ++++++++++++------------ elinks-0.10.6/src/protocol/smb/smb.c | 7 +++++++ 3 files changed, 35 insertions(+), 12 deletions(-) Version: pass Newest version: pass Current: 0.10.6-1ubuntu3.4 Found: 0.10.6-1ubuntu3.3 Diff chaff: pass Merged changelog: ignored Patch system: patchless? (skipped patch system checks) Patch tagging: pass (no patches) Check existing binaries: found existing: elinks dapper-security found existing: elinks-lite dapper-security/universe Total: 2 (new=0, existing=2) Reverse Debdiff: pass Bad files: pass Vcs files in source: pass ------ PASS }}} ==== Uploading a package ==== {{{ $ umt upload Source is: elinks Version is: 0.10.6-1ubuntu3.4 Version with epoch is: 0.10.6-1ubuntu3.4 Distribution is: dapper Pocket is: security Destination is: ubuntu Validating elinks version 0.10.6-1ubuntu3.4 ... ... }}} === Local apt repository === It is often useful to test the binaries before uploading to the security queue (after which, those binaries should be tested fully). UMT provides some functionality for copying the newly built binaries into a local repository. To use: 0. install the necessary software:{{{ $ sudo apt-get install apt-utils moreutils apache2 }}} 0. create a directory in /var/www, writable by you:{{{ $ sudo mkdir -p /var/www/debs/testing $ sudo chown -R <username>:<username> /var/www/debs }}} Now, after building your binaries you can copy them into the local apt repo:{{{ $ umt repo Package is: hello Version is: 2.4-3ubuntu1 Version with epoch is: 2.4-3ubuntu1 Upstream version is: 2.4 Changelog release is: lucid Release is: lucid Pocket is: security Component is: main Need -sa for source build? False DEB_BUILD_OPTIONS: Repository: /var/www/debs/testing/lucid Copying 'hello_2.4-3ubuntu1_i386.deb'... You need a passphrase to unlock the secret key for user: "Your Name <user@ubuntu.com>" 1024-bit DSA key, ... -- /var/www/debs/testing/lucid updated Use: deb http://192.168.122.1/debs/testing/ lucid/ deb-src http://192.168.122.1/debs/testing/ lucid/ -- SUCCESS }}} The output shows the progress of the command as well as the appropriate lines to add to a sources.list file to apt-get the packages. Keep in mind: * this will copy all binaries found in ../binary relative to your toplevel source (be careful if you compiled i386 and amd64 binaries at different times to be sure you are testing what you thing you are testing) * binaries are copied to the release directory under ```package_tools_repo_base```. Eg, in this example running ```umt repo``` on a package with ```lucid-security``` as the distribution name results in the binaries being copied to /var/www/debs/testing/lucid. * ```umt repo``` is not particularly smart -- it will overwrite packages of the same version and leave everything else in the local repo alone. It is often useful to clean out your local repository with something like:{{{ $ rm -rf /var/www/debs/testing/*/* }}} The ```update_repo``` command from ubuntu-security-tools/repo-tools can also be used to update the local repository information (eg Sources, Packages, etc). ==== copy_sppa_to_repos ==== The ubuntu-security-tools branch also has a script for copying files from LP into your local apt repository, which is very useful for testing the binaries people will actually install. To use:{{{ $ $UST/repo-tools/copy_sppa_to_repos srcpkg1 srcpkg2 }}} Please note, this requires a [[SecurityTeam/UbuntuCVETracker|configured ubuntu-cve-tracker]] to work. === Typical package build procedure === {{{ $ umt download -r lucid foo $ cd foo/lucid/foo-* # cd into the toplevel source $ umt changelog # runs 'dch -i' with appropriate arguments for security updates ... perform your patching, etc ... $ umt build # runs 'umt source' followed by 'umt binary' $ umt build-orig # builds the prior version, for comparing with your new version $ umt compare-log $ umt compare-bin ... view compare-bin results in ../reports/ ... $ umt check $ umt repo ... in a VM, apt-get packages from local repository to test the packages ... $ umt sign $ umt upload }}} |
In an effort to recreate a build environment that closely mimics the Launchpad buildds, the Ubuntu Security team has defined the following steps. This is in many ways an extension of SbuildLVMHowto. Additional information may be found in README.sbuild_setup.
Read these instructions if you want to build packages for the next, in-development version of Ubuntu on the current stable release, e.g. you want to build Quantal packages on your Precise system. Hint: you need to install debootstrap from -backports first.
Setting up and using Sbuild with ddebs
By default when using schroot, mk-sbuild in Lucid uses 'union-type=aufs' while in Precise and later mk-sbuild uses 'union-type=overlayfs'. Also, mk-sbuild passes --variant=buildd to debootstrap so you don't have to. Due to performance issues with ext4 in Ubuntu 10.04 LTS, it is recommended that you use ext3 for your builds (feel free to use ext4 in later releases). For build and package testing, x86_64 users are strongly encouraged to create chroots for both amd64 and i386, whereas i386 users need only to create i386 chroots.
Note: Precise and later no longer supports aufs so after upgrading to Precise, you will have to adjust your schroots to use 'overlayfs' (see below).
Creating the schroots
Install sbuild and schroot:
# optional: also install gnupg-agent or seahorse-agent $ sudo apt-get install sbuild schroot ubuntu-dev-tools
Make sure you are in the 'sbuild' group:
$ sudo adduser <username> sbuild
OPTIONAL: If /var doesn't have enough space, create a big partition for your chroots:
$ sudo mkfs.ext3 /dev/sdXN $ sudo mkdir /srv/devel $ ls -l /dev/disk/by-uuid/|grep sdXN # find the UUID for the disk lrwxrwxrwx 1 root root 10 2010-04-22 12:14 d9300709-7b77-4fcc-8d95-77cecdef9e51 -> ../../sdXN
Add the partition to /etc/fstab (by UUID):
UUID=d9300709-7b77-4fcc-8d95-77cecdef9e51 /srv/devel ext3 defaults,relatime 0 0
Mount it:
$ sudo mount /srv/devel $ df -h|grep devel /dev/sdXN 166G 188M 157G 1% /srv/devel
Create the schroot directory:
$ sudo mkdir /srv/devel/schroot
Create ~/.mk-sbuild.sources.ubuntu (can use simply .mk-sbuild.sources and also .mk-sbuild.sources.debian) substituting a local mirror, a nearby mirror or the official archive for '<mirror>'. Eg:
deb http://<mirror>/ubuntu RELEASE main restricted universe multiverse deb-src http://<mirror>/ubuntu RELEASE main restricted universe multiverse #deb http://<mirror>/ubuntu RELEASE-updates main restricted universe multiverse #deb-src http://<mirror>/ubuntu RELEASE-updates main restricted universe multiverse deb http://<mirror>/ubuntu RELEASE-security main restricted universe multiverse deb-src http://<mirror>/ubuntu RELEASE-security main restricted universe multiverse
- Setup for mounting $HOME and extracting the ddebs:
Append to /etc/schroot/mount-defaults (IMPORTANT: this file may be overwritten on upgrades of schroot):
/home/<username>/ubuntu/scratch /scratch none rw,bind 0 0
If you use an encrypted home directory, your $HOME is mounted differently (eg /home/<username>/.Private is mounted on /home/<username>), so you will also have to add to /etc/schroot/mount-defaults:
/home/<username> /home/<username> none rw,bind 0 0
Create /etc/schroot/script-get-ddebs:
# Do whatever the default script would do . /etc/schroot/default/config # Clean up during 00check if [ "x$0" = "x/etc/schroot/setup.d/00check" ]; then if [ $1 = "setup-stop" ]; then # Pull out .ddebs mv /var/cache/buildtmp/schroot/union/overlay/"$SESSION_ID"/build/*/*.ddeb /scratch/ddebs/ 2>/dev/null || true fi fi # end script
Create the following directory:
$ mkdir -p ~/ubuntu/scratch/ddebs
Create ~/.sbuildrc:
# Mail address where logs are sent to (mandatory, no default!) $mailto = 'username'; # Name to use as override in .changes files for the Maintainer: field # (mandatory, no default!). $maintainer_name='Your Name <user@ubuntu.com>'; # Directory for chroot symlinks and sbuild logs. Defaults to the # current directory if unspecified. #$build_dir='/home/<username>/ubuntu/build'; # Directory for writing build logs to $log_dir="/home/<username>/ubuntu/logs"; # Override default sbuild dependency resolver (see 'man sbuild'). The default # resolver (apt) mostly works ok but not always (eg, oneiric libreoffice). # Use 'apt', 'aptitude', 'internal'. Can also use '--build-dep-resolver' with # sbuild or '--sbuild-dep-resolver' with umt. #$build_dep_resolver="apt"; # don't remove this, Perl needs it: 1;
Then make the following directories (change if specified something different in ~/.sbuildrc):
$ mkdir -p $HOME/ubuntu/logs # $HOME/ubuntu/build (if used)
- Create ~/.mk-sbuild.rc:
For schroot < 1.4.6-1 (Ubuntu 10.04 LTS and earlier):
# mk-sbuild tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin" SKIP_UPDATES="1"
For schroot >= 1.4.6-1 (Ubuntu 10.10 and later):
# mk-sbuild tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin preserve-environment=true" SKIP_UPDATES="1"
For ubuntu-dev-tools >= 0.138 (ie, 12.04 and later):
# mk-sbuild build tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory' SOURCE_CHROOTS_DIR="/srv/devel/schroot" # default: /var/lib/schroot/chroots SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs source-root-users=root,sbuild,admin source-root-groups=root,sbuild,admin preserve-environment=true" SKIP_UPDATES="1" DEBOOTSTRAP_INCLUDE="devscripts"
What this does is tells schroot that the chroots are in "/srv/devel/schroot", and to append to the configuration of the chroot in /etc/schroot/schroot.conf the contents of SCHROOT_CONF_SUFFIX. Using 'SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs"' means that /etc/schroot/script-get-ddebs gets called to configure the chroot. This script sources '/etc/schroot/default/config' so it behaves normally, then pulls the ddebs out and puts them into the bind mounted /scratch directory in the chroot (ie into ~/scratch). Finally, create the schroots (this assumes x86_64 host; for i386, omit amd64):
$ sg sbuild # only needed if you were added to the sbuild group during this session $ for i in hardy lucid natty oneiric precise quantal; do mk-sbuild $i --arch=amd64 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu mk-sbuild $i --arch=i386 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu done for i in etch lenny squeeze sid ; do mk-sbuild $i --skip-updates --distro=debian done
For arm (see ARM/RootfsFromScratch for more details) :
$ mk-sbuild --arch armel lucid
See man mk-sbuild for details.
Note 1: debootstrap in Natty will no longer install Dapper schroots because of the switch to SHA256. As a temporary work-around until Dapper goes EoL, you can downgrade to Maverick's debootstrap, create the Dapper schroots, and upgrade back.
Note 2: Oneiric (and higher?) schroots may not have the /var/lock directory which is expected in older versions of sbuild. To fix this:
$ for a in i386 amd64 armel ; do schroot -c oneiric-$a-source -u root -- sh -c 'mkdir /var/lock ; chmod 1777 /var/lock' ; done
Note 3: Oneiric and higher require you to run sbuild-update --keygen to generate a local archive key. This only needs to be done once and can be done as your user so long as you are in the 'sbuild' group. See manpage for details. (If you see this error when creating the keys 'gpg: can't create `(null)': Permission denied', then simply remove the existing keys in /var/lib/sbuild/apt-keys/* and try again.)
Optional (experimental): It can often be useful to perform builds in /dev/shm, which is a tmpfs directory in RAM. While there are limitations on what you can compile and the number of concurrent builds that can be performed (based on how much memory your build machine has), building in /dev/shm can lead to much faster build times and reduce disk I/O. One way to achieve this is to configure additional full schroots for this, and specify the 'union-overlay-directory' directive. Another way to create a 'shared' source schroot like so:
$ sudo sh -c "sed -e 's#]#-shm]#' -e 's#^\(directory=.*\)#\1\nunion-overlay-directory=/dev/shm/schroot/overlay#' /etc/schroot/schroot.conf > /etc/schroot/chroot.d/shm-overlays.conf" $ schroot --list
This simply adds additional schroots based on what you already have (eg, if you have a lucid-amd64 schroot, this creates a lucid-amd64-shm definition in /etc/schroot/chroot.d/shm-overlays.conf). The caveat is that each shm schroot will have the same 'directory' as its corresponding non-shm schroot, so you shouldn't try to modify both 'lucid-amd64-source' and 'lucid-amd64-shm-source' at the same time. Once the shm schroot definitions are add, reference them via schroot (or umt) like any other schroot. Eg:
$ schroot -c lucid-amd64-shm $ umt build -c lucid-amd64-shm
Please note that the directory you specify for 'union-overlay-directory' must exist before using the shm chroot (can add an entry to /etc/rc.local). See man schroot.conf for more details. It might also be useful to remount /dev/shm with more memory than the default, which is 50% of RAM. Can adjust like so: sudo mount -o remount,size=75% /dev/shm. See man mount for details.
Upgrading to Precise (12.04)
Aufs has been deprecated in 12.04 (but is still available). When upgrading your build host to 12.04, you are encouraged to transition to overlayfs:
Adjust all the aufs schroots to use overlayfs. Eg:
$ sudo sed -i 's/aufs/overlayfs/' /etc/schroot/schroot.conf
Note that newer versions of mk-sbuild setup up the chroot configurations in /etc/schroot/chroot.d/ (e.g. /etc/schroot/chroot.d/sbuild-precise-amd64) so you may need to edit files there as well.
Adjust ~/.mk-sbuild.rc to have (devscripts no longer installed by default in 12.04 and later. Specify multiple packages via a comma-separated list):
DEBOOTSTRAP_INCLUDE=devscripts
Install some helpful packages
There are a few packages that will help you more closely mimic the buildd's so that the results of your local builds will be as close as possible to those used to build Ubuntu packages. Here are a list of helpful things to install. The general format is:
$ schroot -c natty-amd64-source -u root $ apt-get install package
You'll probably want to at least do both the amd64 and i386 platforms, and of course change the distroseries as appropriate. Here are some helpful packages that are not normally installed by default:
pkgbinarymangler - performs a number of sanity checks including important ones for Python packages.
devscripts - includes debuild which is useful on its own and required by UMT for building packages (this was included by default when using mk-sbuild until 12.04)
The version of mk-sbuild in maverick and newer supports adding additional packages when creating the schroots via either the --debootstrap-include command line argument or by adding a DEBOOTSTRAP_INCLUDE setting to the ~/.mk-sbuild.rc. Both methods take a comma-separated list of packages. Thus to always add pkgbinarymangler and devscripts when setting up the schroots, add:
DEBOOTSTRAP_INCLUDE="pkgbinarymangler,devscripts"
to your ~/.mk-sbuild.rc file.
Deleting a schroot
- Edit /etc/schroot/schroot.conf to remove the stanza for the chroot
Remove the chroot from the disk:
$ sudo rm -rf /var/lib/schroot/chroots/lucid-amd64
or if on a separate partition:
$ sudo rm -rf /srv/devel/schroot/lucid-amd64
Using the schroot
Using a schroot is similar to the chroot command but you specify the chroot and user you want to use rather than the directory.
$ schroot -c lucid-amd64 -u root (lucid-amd64)root@foo:/home/user# apt-get install ... (lucid-amd64)root@foo:/home/user# su user (lucid-amd64)user@foo$ exit exit (lucid-amd64)root@foo:/home/user# exit logout $
Chroot to the other architectures with:
$ schroot -c lucid-i386 -u root $ schroot -c lucid-armel -u root
Or without root with:
$ schroot -c lucid-amd64
Or building via sbuild directly (see UMT, below, for the Ubuntu Security build scripts):
$ apt-get source foo $ cd ./foo-* ... do work ... $ dch -i $ update-maintainer $ debuild -S $ sbuild -d lucid ../package_1.2.3-4.1.dsc
Maintaining the schroots
Can see a listing of all your chroots with:
$ schroot -l dapper-amd64 dapper-amd64-source dapper-i386 dapper-i386-source etch-amd64 etch-amd64-source hardy-amd64 hardy-amd64-source hardy-i386 hardy-i386-source ...
Can get information on your chroots with schroot -i [-c chroot]. Eg:
$ schroot -i -c hardy-amd64 ---- Chroot ---- Name hardy-amd64 Description hardy-amd64 Type directory Priority 3 Users Groups sbuild root admin Root Users Root Groups root sbuild admin Aliases Environment Filter ^(BASH_ENV|CDPATH|ENV|HOSTALIASES|IFS|KRB5_CONFIG|KRBCONFDIR|KRBTKFILE|KRB_CONF|LD_.*|LOCALDOMAIN|NLSPATH|PATH_LOCALE|RES_OPTIONS|TERMINFO|TERMINFO_DIRS|TERMPATH)$ Run Setup Scripts true Script Configuration script-get-ddebs Session Managed true Session Cloned true Session Purged false Directory /srv/devel/schroot/hardy-amd64 Personality linux Filesystem union type aufs Filesystem union overlay directory /var/lib/schroot/union/overlay Filesystem union underlay directory /var/lib/schroot/union/underlay Source Users Source Groups Source Root Users root sbuild admin Source Root Groups root sbuild admin
The '-source' chroot is the pristine chroot and you shouldn't go into it unless you want to change something for all future schroots into the chroot. Eg, if you wanted to always have 'vim' installed in your hardy-amd64 chroot, use:
$ schroot -c hardy-amd64-source -u root $ apt-get install vim $ exit
Now, any time you use the hardy-amd64 (ie, without '-source'), vim will be installed.
It is also useful to keep your schroots up to date via cron. This can be done by creating $HOME/bin/schroot_update:
skipped="edgy|feisty|gutsy|intrepid|jaunty" for d in `schroot -l | grep -- '-source$' | egrep -v "($skipped)"` do echo "Updating '$d'" schroot -q -c $d -u root -- sh -c 'apt-get -qq update && apt-get -qy dist-upgrade && apt-get clean' echo "" done
Then adding a crontab entry for your user:
15 6 * * * /home/<username>/bin/schroot_update
Expiring active schroot sessions
Sometimes needed if schroots are left hanging around due to a crash or reboot. You can expire all active schroot sessions with (see also /etc/default/schroot):
$ schroot -e --all-sessions # see 'man schroot' for details
Setting up and using UMT
In an effort to both mimic the official buildd's and standardize building of packages, the Ubuntu Security team wrote UMT, a part of ubuntu-security-tools. This branch has helper scripts and tools for performing various tasks. Arguably the most important is UMT, as this is the primary tool the Ubuntu Security team uses for building packages. Much of this can be seen in the README file in the branch.
UMT has a number of commands to help make the build process repeatable and easy to follow. The main things to remember are that:
- all the commands are run in the toplevel source (except 'download')
- ../* (relative to the toplevel source) will contain the original source package
- ../source (relative to the toplevel source) will contain your new source package and debdiff
- ../binary (relative to the toplevel source) will contain your new binary packages
../reports (relative to the toplevel source) will contain the debcompare output after running umt compare-bin
Setting up UMT
First, download the ubuntu-security-tools and ubuntu-qa-tools branches:
$ bzr branch lp:ubuntu-security-tools $ bzr branch lp:ubuntu-qa-tools
Add the UST and UQT environment variables to your startup scripts (eg ~/.bashrc) and have them point to your branches:
export UST="$HOME/bzr-pulls/ubuntu-security-tools" export UQT="$HOME/bzr-pulls/ubuntu-qa-tools"
Then add umt to your PATH.
$ ln -s $UST/build-tools/umt $HOME/bin/umt
Make sure the DEBEMAIL and DEBFULLNAME environment variables are setup in your startup scripts (eg ~/.bashrc):
export DEBEMAIL='user@ubuntu.com' export DEBFULLNAME='Your Name'
Create ~/.ubuntu-security-tools.conf to have:
# list of all active releases (included devel) release_list="hardy lucid maverick natty oneiric precise" # name of the current devel release release_devel="precise" # non-Ubuntu releases (used for schroots, VMs, etc) release_extras="unstable unstable-i386 etch etch-i386" # root of the sbuild log output (i.e. where $log_dir in ~/.sbuildrc is bind mounted in the chroot via /etc/schroot/mount-defaults) pkgbuild_logs="/home/<username>/ubuntu/logs" # path when building with ccache (bind mounted in the chroot via /etc/schroot/mount-defaults) pkgbuild_ccache="/scratch/ccache" # sets 'ulimit -v $pkgbuild_ulimit_v'. Set to "unlimited" for no limit. May # break some builds (like openoffice.org), but is important for other # builds (like gcc). pkgbuild_ulimit_v="1024000" # the URL for the regular Ubuntu archive or mirror build_tools_sbuildmirror="http://archive.ubuntu.com/ubuntu" # currently used by check_source_packages package_tools_name="Your Name" package_tools_email="user@ubuntu.com" package_tools_type="security" package_tools_repo_url="http://192.168.122.1/debs/testing" package_tools_repo_base="/var/www/debs/testing" #package_tools_force_rmadison="yes" # currently used by ddput upload_tools_ddput_security="security:RELEASE"
Setting up your apt sources
Your apt sources need to be adjusted to have deb-src lines for all supported releases (and Debian releases). A helper script in ubuntu-security-tools can be used for this:
$ $UST/build-tools/build-sources-list | sudo sh -c 'cat > /etc/apt/sources.list.d/ubuntu-security.list'
This will look at build_tools_sbuildmirror, release_list, release_devel and release_extras in $HOME/.ubuntu-security-tools.conf and generate the appropriate deb-src lines.
Then, setup secure apt:
$ sudo apt-get install ubuntu-keyring # may already be installed $ sudo apt-key update $ sudo apt-get install debian-archive-keyring $ sudo apt-key add /usr/share/keyrings/debian-archive-keyring.gpg $ sudo apt-get update
Setting up dput
Create/update ~/.dput.cf to have:
# Soyuz security # dput security:lucid ./*_source.changes [security] fqdn = ppa.launchpad.net incoming = ~ubuntu-security/ubuntu/%(security)s login = anonymous # Soyuz security-proposed # dput security-proposed:lucid ./*_source.changes [security-proposed] fqdn = ppa.launchpad.net incoming = ~ubuntu-security-proposed/ubuntu/%(security-proposed)s login = anonymous
Using umt
umt basically works like so:
$ umt <command> <options>
You can see all the commands with umt --help. Eg:
$ umt help ... Uncomplicated Massive Tool (umt) umt COMMAND [OPTIONS] COMMAND: search List best source packages for each release download Get source packages and unpack changelog sdch-like new entry creation or extraction of changelog from prior release (dch-repeat-like) source Produces source package from current directory of unpacked source binary Produces binary packages and other outputs from ../source source build Do both above build-orig Temporarily builds the previous release to get the log file sign Sign the packages check Do a check-source-package compare-log Compare the build log against the prior version's log compare-bin Compare the binary debs against the prior version's binaries repo Copy all built packages into local repository upload Uploads with dput OPTIONS: type "umt COMMAND -h" to get a list of options for each command.
Download a package
To download packages for all releases:
$ umt download <source package>
For each release, this will put the source package in './<source package name>/<release>' and perform dpkg-source -x on it.
To download a package for a single release:
$ umt download -r lucid <source package>
Source build
$ umt source Package is: postgresql-8.1 Version is: 8.1.21-0ubuntu0.6.06 Version with epoch is: 8.1.21-0ubuntu0.6.06 Upstream version is: 8.1.21 Changelog release is: dapper Release is: dapper Pocket is: security Component is: main Need -sa for source build? False DEB_BUILD_OPTIONS: Dependency resolution: auto Chroot: dapper-amd64 Censored sources.list for main build Get:1 http://<mirror> dapper Release.gpg [189B] Hit http://<mirror> dapper Release ... dpkg-buildpackage: source only upload (original source is included) Copying files to '../source' Skipping debdiff (only one dsc found) SUCCESS: source package in '../source'
Some useful options for umt source are:
- --force-orig (forces inclusion of orig.tar.gz via use of -sa)
- -v VERSION (use changelog information from all versions strictly later than VERSION)
- -c CHROOT (specify the chroot to use instead of the default)
- -s (skip pocket and release sanity checks -- useful when doing non-security builds)
Binary build
$ umt binary Package is: postgresql-8.1 Version is: 8.1.21-0ubuntu0.6.06 Version with epoch is: 8.1.21-0ubuntu0.6.06 Upstream version is: 8.1.21 Changelog release is: dapper Release is: dapper Pocket is: security Component is: main Need -sa for source build? False DEB_BUILD_OPTIONS: Chroot: dapper-amd64 core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited sbuild command is: /usr/bin/sbuild --setup-hook /tmp/umt-ws7rLM -d dapper-amd64 -A postgresql-8.1_8.1.21-0ubuntu0.6.06.dsc Sessions still open, not unmounting sbuild (Debian sbuild) 0.59.0 (02 Aug 2009) on ... ... Built successfully /var/lib/schroot/mount/dapper-amd64-a5089ef6-62ff-48e9-9977-9613ca584e3c/build/jamie-postgresql-8.1_8.1.21-0ubuntu0.6.06-amd64-kQ6GiK -- Not removing build depends: cloned chroot in use
You can do a simple no-change rebuild (useful for examining the build log) with:
$ umt build -s [-c <chroot>]
You can also build the previous version (needed for log comparing) with:
$ umt build-orig
Signing a package
To sign your own package:
$ umt sign
To sign a sponsored package:
$ umt sign -k
Verifying a package
$ umt compare-log ... diff of last build log and current build opened in EDITOR ... $ umt compare-bin ... creates reports in ../reports using debcompare (from ubuntu-security-tools) ... $ umt check Running '<absolute path to>/security-tools/package-tools/check-source-package -s -T dapper-security -b ../binary ../source/elinks_0.10.6-1ubuntu3.4.dsc' ... Checking: elinks_0.10.6-1ubuntu3.4.dsc Mode: security Debdiff: elinks_0.10.6-1ubuntu3.4.debdiff Retrieving madison output: pass Binary build: pass PGP (verify dsc): pass PGP (verify source.changes): pass PGP (signature present): pass Distribution: pass Pocket: pass Maintainer: pass Changed-By: pass Source has orig.tar.gz: pass Recent date: pass Output of diffstat: elinks-0.10.6/debian/changelog | 16 ++++++++++++++++ elinks-0.10.6/src/intl/charsets.c | 24 ++++++++++++------------ elinks-0.10.6/src/protocol/smb/smb.c | 7 +++++++ 3 files changed, 35 insertions(+), 12 deletions(-) Version: pass Newest version: pass Current: 0.10.6-1ubuntu3.4 Found: 0.10.6-1ubuntu3.3 Diff chaff: pass Merged changelog: ignored Patch system: patchless? (skipped patch system checks) Patch tagging: pass (no patches) Check existing binaries: found existing: elinks dapper-security found existing: elinks-lite dapper-security/universe Total: 2 (new=0, existing=2) Reverse Debdiff: pass Bad files: pass Vcs files in source: pass ------ PASS
Uploading a package
$ umt upload Source is: elinks Version is: 0.10.6-1ubuntu3.4 Version with epoch is: 0.10.6-1ubuntu3.4 Distribution is: dapper Pocket is: security Destination is: ubuntu Validating elinks version 0.10.6-1ubuntu3.4 ... ...
Local apt repository
It is often useful to test the binaries before uploading to the security queue (after which, those binaries should be tested fully). UMT provides some functionality for copying the newly built binaries into a local repository. To use:
install the necessary software:
$ sudo apt-get install apt-utils moreutils apache2
create a directory in /var/www, writable by you:
$ sudo mkdir -p /var/www/debs/testing $ sudo chown -R <username>:<username> /var/www/debs
Now, after building your binaries you can copy them into the local apt repo:
$ umt repo Package is: hello Version is: 2.4-3ubuntu1 Version with epoch is: 2.4-3ubuntu1 Upstream version is: 2.4 Changelog release is: lucid Release is: lucid Pocket is: security Component is: main Need -sa for source build? False DEB_BUILD_OPTIONS: Repository: /var/www/debs/testing/lucid Copying 'hello_2.4-3ubuntu1_i386.deb'... You need a passphrase to unlock the secret key for user: "Your Name <user@ubuntu.com>" 1024-bit DSA key, ... -- /var/www/debs/testing/lucid updated Use: deb http://192.168.122.1/debs/testing/ lucid/ deb-src http://192.168.122.1/debs/testing/ lucid/ -- SUCCESS
The output shows the progress of the command as well as the appropriate lines to add to a sources.list file to apt-get the packages. Keep in mind:
- this will copy all binaries found in ../binary relative to your toplevel source (be careful if you compiled i386 and amd64 binaries at different times to be sure you are testing what you thing you are testing)
binaries are copied to the release directory under package_tools_repo_base. Eg, in this example running umt repo on a package with lucid-security as the distribution name results in the binaries being copied to /var/www/debs/testing/lucid.
umt repo is not particularly smart -- it will overwrite packages of the same version and leave everything else in the local repo alone. It is often useful to clean out your local repository with something like:
$ rm -rf /var/www/debs/testing/*/*
The update_repo command from ubuntu-security-tools/repo-tools can also be used to update the local repository information (eg Sources, Packages, etc).
copy_sppa_to_repos
The ubuntu-security-tools branch also has a script for copying files from LP into your local apt repository, which is very useful for testing the binaries people will actually install. To use:
$ $UST/repo-tools/copy_sppa_to_repos srcpkg1 srcpkg2
Please note, this requires a configured ubuntu-cve-tracker to work.
Typical package build procedure
$ umt download -r lucid foo $ cd foo/lucid/foo-* # cd into the toplevel source $ umt changelog # runs 'dch -i' with appropriate arguments for security updates ... perform your patching, etc ... $ umt build # runs 'umt source' followed by 'umt binary' $ umt build-orig # builds the prior version, for comparing with your new version $ umt compare-log $ umt compare-bin ... view compare-bin results in ../reports/ ... $ umt check $ umt repo ... in a VM, apt-get packages from local repository to test the packages ... $ umt sign $ umt upload
SecurityTeam/BuildEnvironment (last edited 2024-07-02 14:39:12 by pfsmorigo)