testing on canonistack: add hint about SSH/ICMP security group ALLOW rules
|Deletions are marked like this.||Additions are marked like this.|
|Line 37:||Line 37:|
|Reproducing tests as they are run in production requires access to an OpenStack-based cloud (e.g. Canonistack). Source the relevant `nova.rc` file with the cloud credentials. Ensure that you have autopkgtest 3.16 or newer installed (the deb installs on all supported Ubuntu releases without further dependencies).||Reproducing tests as they are run in production requires access to an OpenStack-based cloud (e.g. Canonistack). Source the relevant `nova.rc` file with the cloud credentials. Ensure that you have autopkgtest 3.16 or newer installed (the deb installs on all supported Ubuntu releases without further dependencies). Also make sure to allow SSH/ICMP access in the corresponding security group to enable autopkgtest to access your cloud instances via SSH.|
|Line 44:||Line 44:|
|-- ssh -s nova -- --flavor m1.small --image ubuntu-groovy-daily-amd64-server-20201022.1-disk1.img --keyname testkey||-- ssh -s nova -- --flavor m1.small --image ubuntu/ubuntu-groovy-daily-amd64-server-20201022.1-disk1.img --keyname testkey|
Migrating packages from -proposed to release
Frequently Asked/Answered Questions
- How could installing a package into the release pocket possibly break other packages?
- I still don't understand! The proposed-migration scripts say that this package is a valid candidate, but it still hasn't gone into the release pocket.
- My package is stalled because it's out of date on some architecture. What do I do?
- Are there any exceptions? I'm sure acmefoo has just made it into the release pocket despite not satisfying all of the requirements.
- Can you provide a real, non-trivial example?
- The update_output.txt file is completely unreadable!
- I'm seeing a squid proxy? My tests behave differently there!
- Additional information
Migrating packages from -proposed to release
Uploads to the Ubuntu development series are directed to the -proposed pocket, and must build there and pass certain tests before they are promoted to the release pocket for general use. This process is derived from that used by Debian to maintain the testing suite, but there are a few Ubuntu-specific differences. Although the observant will note that this document is also derived from Debian's documentation, it is intended to be a self-contained description for Ubuntu developers.
One important difference that those familiar with the Debian process should be aware of is that -proposed is a partial suite: that is, it only contains packages that have not yet migrated, and must be used on top of the release pocket. This can cause some confusion where removals are concerned. Furthermore, the Ubuntu process has no arbitrary delays and no consideration of release-critical bugs, so there is very little benefit to users in using -proposed during the development cycle and such use is strongly discouraged. (On the other hand, the official build infrastructure builds packages against -proposed, and this is normally also a good idea when test-building packages locally.)
A (particular version of a) package will move into the release pocket when it satisfies all of the following criteria:
It must be built and published on all architectures it is currently published for in the release pocket (this applies to each binary, but it is acceptable if other binaries from the same source for the same architecture are up to date in -proposed and the binary in question has been removed);
All of its dependencies must either be satisfiable by packages already in the release pocket, or be satisfiable by the group of packages which are going to be installed at the same time;
If the package or any of its binary or test reverse-dependencies defines any autopkgtest test suites, they must either pass or never have passed ("Always failed");
No bugs with the block-proposed tag are open against the package (this can be used to temporarily block a package on manual testing);
- The operation of installing the package into the release pocket must not break any packages currently in the release pocket.
A package which satisfies the first three of the above is said to be a "Valid Candidate". A package which does not satisfy the first three of the above is said to be "Not considered".
There is another related tag, update-excuse, which makes the tagged bug linked from the update excuses page under the package the bug is opened against. A bug with the update-excuse tag does not block the package from migration, it just documents a problem which prevents migration. The tag is typically used to track triaging effort for non-trivial problems and if the bug is closed from the package's changelog it gets closed when the package enters the release pocket. update-excuse applies only to the development series, for stable releases update-excuse-$SERIES tags can be used.
The update script shows when each package might move from -proposed into the release pocket. The output is twofold:
The update excuses: list of all candidate package versions and the basic status of their propagation into the release pocket; this is the first stage of migration, and only packages that pass it proceed to
The update output: the complete, rather crude output of the proposed-migration scripts as they recurse through the candidates.
The notest update output: much like update output, but it shows what it would look like if all autopkgtests were passing.
autopkgtest tests are run in ephemeral cloud instances as described on /AutopkgtestInfrastructure. The autopkgtests can be configured to run on different types of workers in https://git.launchpad.net/autopkgtest-cloud/. The configurations are defined in the worker-config-production folder where most files are named corresponding to the architecture on which they run and worker.conf is for amd64. The big_packages option is for packages which need more memory and CPU - by default tests run on an m1.small unit (1 vCPU and 1.7Gb Memory), while big_packages run on an m1.large (2 vCPU and 7.5 Gb Memory) unit. The long_tests are passed an increased timeout value ('--timeout-test=40000') to autopkgtest. Additionally, any big_packages are passed '--timeout-test=20000'. Any changes to autopkgtest-cloud need to be manually deployed.
The update excuses link to relevant test results, and all results can be browsed on http://autopkgtest.ubuntu.com. In addition to the files linked to on the autopkgtest website every test also produces a "result.tar" file which contains the files exitcode, testbed-packages, testpkg-version, duration, testinfo.json, and requester. The file testinfo.json contains information on the number of processors the unit on which the test ran had which can be useful for confirming a test ran on a big worker.
In most cases test results should be reproducible in a local QEMU VM.
Reproducing tests in the cloud
Reproducing tests as they are run in production requires access to an OpenStack-based cloud (e.g. Canonistack). Source the relevant nova.rc file with the cloud credentials. Ensure that you have autopkgtest 3.16 or newer installed (the deb installs on all supported Ubuntu releases without further dependencies). Also make sure to allow SSH/ICMP access in the corresponding security group to enable autopkgtest to access your cloud instances via SSH.
First make sure that your cloud knows your public SSH key, in nova keypair-list. If you don't have one yet, add it with nova keypair-add. Let's call this testkey here.
You can now run your tests with the ssh runner and the nova setup script, e. g. for the gzip package:
autopkgtest gzip --apt-pocket proposed --setup-commands setup-testbed --apt-upgrade \ -- ssh -s nova -- --flavor m1.small --image ubuntu/ubuntu-groovy-daily-amd64-server-20201022.1-disk1.img --keyname testkey
The complete command as used in production (including all proxy settings) can also be seen at the top of test logs.
Testing against a PPA
It can be useful to test autopkgtest results (particularly when attempting to test a regression fix) via PPA uploads. To launch an autopkgtest run against a PPA, open:
where RELEASE is the codename for the release to test; ARCH is the architecture to test against; SRCPKG is the package to test in the PPA; LPUSER is the launchpad username of the owner of the PPA; PPA is the PPA name; and VERSION is the version to indicate triggered the test (for the purposes of PPA testing, this seems like logically it should be the last version in the release in the PPA), and must be published in the PPA.
Presuming you have appropriate privilege to execute the test, you should shortly see the running test at http://autopkgtest.ubuntu.com/running.
Once completed, the test results will be available via an index at
The index will contain the results for each autopkgtest run against this particular PPA for this RELEASE. You can append the appropriate path from the index to
How to re-run autopkgtests with dependencies on other packages in the proposed pocket
By default (this seems to be a relatively recent change), autopkgtest will only use the trigger src package from the proposed pocket, and by default only one trigger package is mentioned. However, additionally packages can be specified in the retry URL by suffixing &trigger=SRCPKG/VERSION for each source package to use from proposed. Simply right-click the retry URL, copy & paste it into your browser and append an appropriate string.
How to run autopkgtests of a package against the version in the release pocket
Sometimes it is useful to know the test result of an autopkgtest run for a package against the release/updates pocket only, without considering any versions that are in -proposed. One example would be when we see an autopkgtest regression in tests run for a package version in -proposed but we suspect that the regression is already present in the package that's already in the release/updates pocket.
To do this, simply open an autopkgtest request in the likes of:
where PKGINRELEASE is any package currently in the release/updates pocket and it's current version. Please note that we'd best want SRCPKG and PKGINRELEASE to be different, as otherwise if we're running the SRCPKG autopkgtests when SRCPKG has a version in -proposed, the autopkgtest infra will automatically use the version in -proposed despite us mentioning an explicit version in the trigger.
In cases where we want to investigate if the given failure we see is already present in release, usually you just need to copy the retry url and modify the trigger to be any package and version from the release/updates pocket. Once the run is finished, it will be visible on the autopkgtest result page for the package (not on the update_excuses page though).
Frequently Asked/Answered Questions
How could installing a package into the release pocket possibly break other packages?
The structure of the distribution archives is such that they can only contain one version of a package; a package is defined by its name. So, when the source package acmefoo is installed into the release pocket, along with its binary packages acme-foo-bin, acme-bar-bin, libacme-foo1 and libacme-foo-dev, the old version is removed.
However, the old version may have provided a binary package with an old soname of a library, such as libacme-foo0. Removing the old acmefoo will remove libacme-foo0, which will break any packages which depend on it.
Evidently, this mainly affects packages which provide changing sets of binary packages in different versions (in turn, mainly libraries). However, it will also affect packages upon which versioned dependencies have been declared of the ==, <= or << varieties.
When the set of binary packages provided by a source package change in this way, all the packages that depended on the old binaries will have to be updated to depend on the new binaries instead. Because installing such a source package into the release pocket breaks all the packages that depended on it in the release pocket, some care now has to be taken: all the depending packages must be updated and ready to be installed themselves so that they won't be broken. In some cases, manual intervention by the release team may be required.
If you are having problems with complicated groups of packages like this, contact the ubuntu-devel mailing list or the #ubuntu-release IRC channel for help.
This tends to happen when in some way, directly or indirectly, installing the package will break some other package.
Remember to consider your package's dependencies. Suppose your package depends on libtool, or libltdlX. Your package won't go into the release pocket until the right version of libtool is ready to go in with it.
In turn, that won't happen until installing libtool doesn't break things already in the release pocket. In other words, until all other packages which depend on libltdlY (where Y is the earlier version) have been recompiled, and all their autopkgtests pass, etc, none of these packages will enter the release pocket.
This is where the textual output is useful: it gives hints (albeit very terse ones) as to which packages break when a valid candidate is added to the release pocket.
My package is stalled because it's out of date on some architecture. What do I do?
Check the status of your package in Launchpad. If the package doesn't build, it will be marked as failed; investigate the build logs and fix any of the problems that are caused by your package's sources.
If you happen to notice that some architectures have built the new version of your package, but it isn't showing up in the output of the proposed-migration scripts, then you just have to be a bit more patient until the publisher runs. This should happen well within half an hour.
If you have explicitly dropped the architecture from the Architecture list in the control file, and the package has been built for that architecture before, you will need to request that the old binary package for this architecture be removed from the archive before your package can transition to the release pocket. You need to file a bug on your package requesting removal of the dropped architecture's packages, and subscribe the ubuntu-archive team.
The release team can override the rules in a few ways:
- They can decide that the breakage caused by the installation of a new library will make things better rather than worse, and let it go in along with its flotilla of dependents.
- They can also manually remove packages from the release pocket that would be broken, so that new stuff can be installed.
They can force autopkgtest failures to be ignored. The hints contain the ignored failures and explanation why they are ignored. If you would like to add more hints please open a merge request against the hints repository.
Can you provide a real, non-trivial example?
Here's one: when the source package apache is installed into the release pocket, along with its binary packages apache, apache-common, apache-dev and apache-doc, the old version is removed.
However, all Apache module packages depend on apache-common (>= something), apache-common (<< something), so this change breaks all of those dependencies. Consequently, all Apache modules need to be recompiled against the new version of Apache in order for the release pocket to be updated.
Let's elaborate on this a bit further: after all of the modules have been updated in unstable to work with a new Apache, the proposed-migration scripts try apache-common and find out that it breaks all the Apache modules because they have Depends: apache-common (<< the current version), and then try libapache-foo to find out that it doesn't install because it has Depends: apache-common (>= the new version).
However, later they'll apply a different logic (sometimes prompted by a manual intervention): they'll ignore the fact apache-common breaks stuff, and keep going with things that work; if it still doesn't work after we've done everything we can, too bad, but maybe it will work. Later they'll try all the random libapache-foo packages and see that they indeed work.
After everything's been tried, they check how many packages have been broken, work out if that's better or worse than what there was originally and either accept everything or forget about it. You'll see this in update_output.txt on "recur:" lines.
recur: [foo bar] baz
basically says "having already found that foo and bar make things better, I'm now trying baz to see what happens, even though that breaks things". The lines of update_output.txt that start with "accepted" indicate things that appear to make things better, and "skipped" lines make things worse.
The update_output.txt file is completely unreadable!
That is not a question.
Let's take an example:
skipped: cln (0) (150+4) got: 167+0: a-40:a-33:h-49:i-45 * i386: ginac-cint, libginac-dev
This means that if cln goes into the release pocket, ginac-cint and libginac-dev become uninstallable in the release pocket on i386. Note that the architectures are checked in a fixed order and only the problems on the first architecture with problems are shown - that's why the i386 architecture is shown so often.
The "got" line includes the number of problems in the release pocket on the different architectures (until the first architecture where a problem is found — see above). The "i-45" means that if cln would go into the release pocket, there would be 45 uninstallable packages on i386. Some of the entries above and below cln show there were 43 uninstallable packages in the release pocket on i386 at the time.
The "skipped: cln (0) (150+4)" line means that there are still 150 packages to go through after this package until this check of all packages is completed, and that 4 packages have been found already that won't be planned to be upgraded because they would break dependencies. The "(0)" is irrelevant; you can safely ignore it.
Note that there are several checks of all packages in one proposed-migration script run.
Jules Bean initially assembled the frequently asked questions and answers. Colin Watson modified them for Ubuntu.
I'm seeing a squid proxy? My tests behave differently there!
Thanks to LXD or VMs you should be able to reproduce the environment pretty easily. Try the following:
Create one LXD or VM to host squid:
lxc launch ubuntu: squid-proxy lxc exec squid-proxy bash apt-get update apt-get install squid # Open up squid in /etc/squid/squid.conf # Very open example: http://paste.ubuntu.com/24875282/ service restart squid exit
Run your autopkgtest in another LXD, and point it at your squid. Note that this is a big long commmand that is updated as of this writing (2017-Jun-16), but you can always see the first few lines of test output for your failure log as the command line is printed right at the top, then just customize to your needs:
# N.B., Assuming LXD resolves at: squid-proxy.lxd, you can use IP if you need to PKG_NAME=<pkg name here> autopkgtest-build-lxd images:ubuntu/artful/amd64 autopkgtest --shell-fail -U --setup-commands 'printf "http_proxy=http://squid-proxy.lxd:3128\nhttps_proxy=http://squid-proxy.lxd:3128\nno_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpad.net\n" >> /etc/environment' --apt-pocket=proposed=src:$PKG_NAME $PKG_NAME -- lxd autopkgtest/ubuntu/artful/amd64
You can run britney locally for testing or to set it up to run against PPAs.
If you would like to filter update_excuses.html for packages that just have autopkgtest regressions, run the following command:
wget -q -O - http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html | grep Regression | sed 's/^<li>autopkgtest for \([a-z0-9-]*\).*/\1/'| sort | uniq
Anthony Towns takes credit for the original implementation of testing in Debian. Colin Watson is at fault for most of the Ubuntu adaptations.