BuildEnvironment

Differences between revisions 9 and 10
Revision 9 as of 2010-05-20 13:51:02
Size: 15769
Editor: pool-71-114-231-221
Comment:
Revision 10 as of 2010-05-20 13:55:20
Size: 15816
Editor: pool-71-114-231-221
Comment:
Deletions are marked like this. Additions are marked like this.
Line 176: Line 176:


=== Setting up your apt sources ===
TODO

In an effort to recreate a build environment that closely mimics the Launchpad buildds, the Ubuntu Security team has defined the following steps. This is in many ways an extension of SbuildLVMHowto. This is also documented in README.sbuild_setup.

Setting up and using Sbuild/Aufs with ddebs (Lucid)

mk-sbuild in Lucid uses 'union-type=aufs' by default with schroot now. Also, mk-sbuild passes --variant=buildd to debootstrap so you don't have to. Due to performance issues with ext4 in Ubuntu 10.04 LTS, it is recommened that you use ext3 for your builds. For build and package testing, x86_64 users are strongly encouraged to create chroots for both amd64and i386, whereas i386 users need only to create i386 chroots.

Creating the schroots

  1. Install sbuild and schroot:

    $ sudo apt-get install sbuild schroot
  2. Make sure you are in the 'sbuild' group:

    $ sudo adduser <username> sbuild
  3. OPTIONAL: If /var doesn't have enough space, create a big partition for your chroots:

    $ sudo mkfs.ext3 /dev/sdXN
    $ sudo mkdir /srv/devel
    $ ls -l /dev/disk/by-uuid/|grep sdXN   # find the UUID for the disk
    lrwxrwxrwx 1 root root 10 2010-04-22 12:14 d9300709-7b77-4fcc-8d95-77cecdef9e51 -> ../../sdXN

    Add the partition to /etc/fstab (by UUID):

    UUID=d9300709-7b77-4fcc-8d95-77cecdef9e51 /srv/devel ext3 defaults,relatime 0 0

    Mount it:

    $ sudo mount /srv/devel
    $ df -h|grep devel
    /dev/sdXN             166G  188M  157G   1% /srv/devel

    Create the schroot directory:

    $ sudo mkdir /srv/devel/schroot
  4. Create ~/.mk-sbuild.sources.ubuntu (can use simply .mk-sbuild.sources and also .mk-sbuild.sources.debian) substituting a local mirror, a nearby mirror or the official archive for '<mirror>'. Eg:

    deb http://<mirror>/ubuntu RELEASE main restricted universe multiverse
    deb-src http://<mirror>/ubuntu RELEASE main restricted universe multiverse
    deb http://<mirror>/ubuntu RELEASE-updates main restricted universe multiverse
    deb-src http://<mirror>/ubuntu RELEASE-updates main restricted universe multiverse
    deb http://<mirror>/ubuntu RELEASE-security main restricted universe multiverse
    deb-src http://<mirror>/ubuntu RELEASE-security main restricted universe multiverse
  5. Setup for extracting the ddebs:
    • Append /etc/schroot/mount-defaults:

      /home/<username>/ubuntu/scratch             /scratch             none    rw,bind     0   0

      If you use an encrypted home directory, your $HOME is mounted differently (eg /home/<username>/.Private is mounted on /home/<username>), so you will also have to add to /etc/schroot/mount-defaults:

      /home/<username>                            /home/<username>     none    rw,bind     0   0
    • Create /etc/schroot/script-get-ddebs:

      # Do whatever the default script would do
      . /etc/schroot/script-defaults
      
      # Clean up during 00check
      if [ "x$0" = "x/etc/schroot/setup.d/00check" ]; then
          if [ $1 = "setup-stop" ]; then
              # Pull out .ddebs
              mv /var/cache/buildtmp/schroot/union/overlay/"$SESSION_ID"/build/*/*.ddeb /scratch/ddebs/ 2>/dev/null || true
          fi
      fi
      # end script
    • Create the following directory:

      $ mkdir -p ~/ubuntu/scratch/ddebs
  6. Create ~/.sbuildrc:

    # Mail address where logs are sent to (mandatory, no default!)
    $mailto = 'username';
    
    # Name to use as override in .changes files for the Maintainer: field
    # (mandatory, no default!).
    $maintainer_name='Your Name <user@ubuntu.com>';
    
    # Directory for chroot symlinks and sbuild logs.  Defaults to the
    # current directory if unspecified.
    #$build_dir='/home/<username>/ubuntu/build';
    
    # Directory for writing build logs to
    $log_dir="/home/<username>/ubuntu/logs";
    
    # don't remove this, Perl needs it:
    1;

    Then make the following directories (change if specified something different in ~/.sbuildrc):

    $ mkdir -p $HOME/ubuntu/logs # $HOME/ubuntu/build (if used)
  7. Create ~/.mk-sbuild.rc:

    # mk-sbuild tunables -- SOURCE_CHROOTS_TGZ used with 'file' and SOURCE_CHROOTS_DIR with 'directory'
    SOURCE_CHROOTS_DIR="/srv/devel/schroot"         # default: /var/lib/schroot/chroots
    SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs
    source-root-users=root,sbuild,admin
    source-root-groups=root,sbuild,admin"
    SKIP_UPDATES="1"
    What this does is tells schroot that the chroots are in "/srv/devel/schroot", and to append to the configuration of the chroot in /etc/schroot/schroot.conf the contents of SCHROOT_CONF_SUFFIX. Using 'SCHROOT_CONF_SUFFIX="script-config=script-get-ddebs"' means that /etc/schroot/script-get-ddebs gets called to configure the chroot. This script sources '/etc/schroot/script-defaults' so it behaves normally, then pulls the ddebs out and puts them into the bind mounted /scratch directory in the chroot (ie into ~/scratch).
  8. Finally, create the schroots (this assumes x86_64 host; for i386, omit amd64):

    $ sg sbuild   # only needed if you were added to the sbuild group during this session
    $ for i in dapper hardy intrepid jaunty karmic lucid maverick ; do
        mk-sbuild $i --arch=amd64 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu
        mk-sbuild $i --arch=i386 --skip-updates --debootstrap-mirror=http://<mirror>/ubuntu --distro=ubuntu
    done
    
    for i in etch lenny squeeze sid ; do
        mk-sbuild $i --skip-updates --distro=debian
    done

    For arm (see ARM/RootfsFromScratch for more details) :

    $ mk-sbuild --arch armel lucid

    See man mk-sbuild for details

Deleting a schroot

  1. Edit /etc/schroot/schroot.conf to remove the stanza for the chroot
  2. Remove the chroot from the disk:

    $ sudo rm -rf /var/lib/schroot/chroots/lucid-amd64

    or if on a separate partition:

    $ sudo rm -rf /srv/devel/schroot/lucid-amd64

Using the schroot

Using a schroot is similar to the chroot command but you specify the chroot and user you want to use rather than the directory.

$ schroot -c lucid-amd64 -u root
(lucid-amd64)root@foo:/home/user# apt-get install ...
(lucid-amd64)root@foo:/home/user# su user
(lucid-amd64)user@foo$ exit
exit
(lucid-amd64)root@foo:/home/user# exit
logout
$

Chroot to the other architectures with:

$ schroot -c lucid-i386 -u root
$ schroot -c lucid-armel -u root

Or without root with:

$ schroot -c lucid-amd64

Or building via sbuild directly (see UMT, below, for the Ubuntu Security build scripts):

$ apt-get source foo
$ cd ./foo-*
... do work ...
$ dch -i
$ update-maintainer
$ debuild -S
$ sbuild -d lucid ../package_1.2.3-4.1.dsc

Setting up and using UMT

In an effort to both mimic the official buildd's and standardize building of packages, the Ubuntu Security wrote UMT, a part of ubuntu-security-tools. This branch has helper scripts and tools for performing various tasks. Arguably the most important is UMT, as this is the primary tool the Ubuntu Security team uses for building packages. Much of this can be seen in the README file in the branch.

Setting up your apt sources

TODO

Setting up UMT

First, download the ubuntu-security-tools branch:

$ bzr branch lp:ubuntu-security-tools

Add the UST environment variable to your startup scripts (eg ~/.bashrc) and have it point to the ubuntu-security-tools branch:

export UST="$HOME/bzr-pulls/ubuntu-security-tools"

Then add umt to your PATH.

$ ln -s $UST/build-tools/umt $HOME/bin/umt

Make sure the DEBEMAIL and DEBFULLNAME environment variables are setup in your startup scripts (eg ~/.bashrc):

export DEBEMAIL='user@ubuntu.com'
export DEBFULLNAME='Your Name'

Create ~/.ubuntu-security-tools.conf to have:

# list of all active releases (included devel)
release_list="dapper hardy jaunty karmic lucid maverick"

# name of the current devel release
release_devel="maverick"

# non-Ubuntu releases (used for schroots, VMs, etc)
release_extras="unstable unstable-i386 etch etch-i386"

# root of the sbuild log output (i.e. where $log_dir in ~/.sbuildrc is bind mounted in the chroot via /etc/schroot/mount-defaults)
pkgbuild_logs="/home/<username>/ubuntu/logs"

# path when building with ccache (bind mounted in the chroot via /etc/schroot/mount-defaults)
pkgbuild_ccache="/scratch/ccache"

# sets 'ulimit -v $pkgbuild_ulimit_v'. Set to "unlimited" for no limit. May
# break some builds (like openoffice.org), but is important for other
# builds (like gcc).
pkgbuild_ulimit_v="1024000"

# the URL for the regular Ubuntu archive or mirror
build_tools_sbuildmirror="http://archive.ubuntu.com/ubuntu"

# currently used by check_source_packages
package_tools_name="Your Name"
package_tools_email="user@ubuntu.com"
package_tools_type="security"
package_tools_repo_url="http://192.168.122.1/debs/testing"
package_tools_repo_base="/var/www/debs/testing"

# currently used by ddput
upload_tools_ddput_security="security:RELEASE"

Using umt

umt basically works like so:

$ umt <command> <options>

You can see all the commands with umt --help. Eg:

$ umt help
...
Uncomplicated Massive Tool (umt)
umt COMMAND [OPTIONS]

COMMAND:
search       List best source packages for each release
download     Get source packages and unpack
changelog    sdch-like new entry creation or extraction of changelog from prior
             release (dch-repeat-like)
source       Produces source package from current directory of unpacked source
binary       Produces binary packages and other outputs from ../source source
build        Do both above
build-orig   Temporarily builds the previous release to get the log file
sign         Sign the packages
check        Do a check-source-package
compare-log  Compare the build log against the prior version's log
compare-bin  Compare the binary debs against the prior version's binaries
repo         Copy all built packages into local repository
upload       Uploads with dput

OPTIONS:
type "umt COMMAND -h" to get a list of options for each command.

Download a package

To download packages for all releases:

$ umt download <source package>

For each release, this will put the source package in './<source package name>/<release>' and perform dpkg-source -x on it.

To download a package for a single release:

$ umt download -r lucid <source package>

Source build

$ umt source
Package is: postgresql-8.1
Version is: 8.1.21-0ubuntu0.6.06
Version with epoch is: 8.1.21-0ubuntu0.6.06
Upstream version is: 8.1.21
Changelog release is: dapper
Release is: dapper
Pocket is: security
Component is: main 
Need -sa for source build? False
DEB_BUILD_OPTIONS: 
Dependency resolution: auto
Chroot: dapper-amd64
Censored sources.list for main build
Get:1 http://<mirror> dapper Release.gpg [189B]
Hit http://<mirror> dapper Release  
...
dpkg-buildpackage: source only upload (original source is included)
Copying files to '../source'
Skipping debdiff (only one dsc found)
SUCCESS: source package in '../source'

Binary build

$ umt binary
Package is: postgresql-8.1
Version is: 8.1.21-0ubuntu0.6.06
Version with epoch is: 8.1.21-0ubuntu0.6.06
Upstream version is: 8.1.21
Changelog release is: dapper
Release is: dapper
Pocket is: security
Component is: main 
Need -sa for source build? False
DEB_BUILD_OPTIONS: 
Chroot: dapper-amd64
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
sbuild command is: /usr/bin/sbuild --setup-hook /tmp/umt-ws7rLM -d dapper-amd64 -A postgresql-8.1_8.1.21-0ubuntu0.6.06.dsc
Sessions still open, not unmounting
sbuild (Debian sbuild) 0.59.0 (02 Aug 2009) on ...
...
Built successfully
/var/lib/schroot/mount/dapper-amd64-a5089ef6-62ff-48e9-9977-9613ca584e3c/build/jamie-postgresql-8.1_8.1.21-0ubuntu0.6.06-amd64-kQ6GiK
--
Not removing build depends: cloned chroot in use

Signing a package

To sign your own package:

$ umt sign

To sign a sponsored package:

$ umt sign -k

Verifying a package

$ umt compare-log
... diff of last build log and current build opened in EDITOR ...

$ umt compare-bin
... creates reports in ../reports using debcompare (from ubuntu-security-tools) ...

$ umt check
Running '<absolute path to>/security-tools/package-tools/check-source-package -s -T dapper-security -b ../binary ../source/elinks_0.10.6-1ubuntu3.4.dsc' ...
Checking: elinks_0.10.6-1ubuntu3.4.dsc
Mode: security
Debdiff: elinks_0.10.6-1ubuntu3.4.debdiff

Retrieving madison output:      pass
Binary build:                   pass
PGP (verify dsc):               pass
PGP (verify source.changes):    pass
PGP (signature present):        pass
Distribution:                   pass
Pocket:                         pass
Maintainer:                     pass
Changed-By:                     pass
Source has orig.tar.gz:         pass
Recent date:                    pass
Output of diffstat:             
 elinks-0.10.6/debian/changelog       |   16 ++++++++++++++++
 elinks-0.10.6/src/intl/charsets.c    |   24 ++++++++++++------------
 elinks-0.10.6/src/protocol/smb/smb.c |    7 +++++++
 3 files changed, 35 insertions(+), 12 deletions(-)
Version:                        pass
Newest version:                 pass
 Current: 0.10.6-1ubuntu3.4
 Found:   0.10.6-1ubuntu3.3
Diff chaff:                     pass
Merged changelog:               ignored
Patch system:                   patchless? (skipped patch system checks)
Patch tagging:                  pass (no patches)
Check existing binaries:        
 found existing: elinks dapper-security
 found existing: elinks-lite dapper-security/universe
 Total: 2 (new=0, existing=2)
Reverse Debdiff:                pass
Bad files:                      pass
Vcs files in source:            pass
------
PASS

Uploading a package

$ umt upload
Source is: elinks
Version is: 0.10.6-1ubuntu3.4
Version with epoch is: 0.10.6-1ubuntu3.4
Distribution is: dapper
Pocket is: security
Destination is: ubuntu
Validating elinks version 0.10.6-1ubuntu3.4 ...
...

Typical package build procedure

$ umt download -r lucid foo
$ cd foo/lucid/foo-*    # cd into the toplevel source
... perform your patching, etc ...
$ umt build             # runs 'umt source' followed by 'umt binary'
$ umt build-orig        # builds the prior version, for comparing with your new version 
$ umt compare-log
$ umt compare-bin
$ umt check
$ umt repo
... in a VM, apt-get packages from local repository to test the packages ...
$ umt sign
$ umt upload

SecurityTeam/BuildEnvironment (last edited 2024-03-20 22:39:43 by seth-arnold)