Launchpad Entry: super-piuparts
Packages affected: piuparts
- What would be a good way to report results?
Piuparts is a tool for testing that .deb packages can be installed, upgraded, and removed. It is currently somewhat limited in its scope, fragile in its implementation, and hard to use for large-scale testing of entire package archives.
This spec proposes to improve it by writing automated test suites for better robustness, implementing the use of KVM in addition to chroot for virtualization, and adding a plugin system to allow more flexible testing. The new version of piuparts will use ideas and code from Michael Vogt's update-manager, Ian Haken's vlosuts, and Ian Jackson's autopkgtest, or add code to integrate those tools.
This spec has no direct end-user impact. It may lead Ubuntu to find and fix some bugs.
The current version of piuparts has grown organically by several people, and has no test suite. Thus, in recent times, any changes made to piuparts have had a tendency to break things. This has affected even changes to fix bugs. Thus, a test suite is sorely needed.
Piuparts uses chroot for simulating a full system. This is problematic, because chroot is not a very safe method of virtualization. For safety reasons, piuparts attempts to prevent installed services from starting, which means it tests different code paths from those that actually happen on real systems. KVM is a better way do virtualization, and piuparts should add a way to use it. While that support is being added, the virtualization layer should be made abstract, so that other methods may be added later: Xen, for example.
The testing piuparts currently does is highly centralized on individual packages. It is badly suited for testing whole-system upgrades. Even for individual packages, piuparts only tests for a few things related to package management, and does not verify that the installed software actually works. More flexibility for this would be good. The best way to add such flexibility would be to add a sufficiently powerful plugin system, with each test being a plugin.
There is a need to separate between "test case" and what we will call "test scenario". A test case is an individual test, for example, "are there broken symlinks?" or "has any file changed?". A test scenario is a sequence of package management operations done to the system: install some packages, remove packages, upgrade pacakges, upgrade the entire system, etc. The user will specify a scenario to run, and which test cases to run in that scenario. However, test cases are not tied to individual scenarios, although some may only make sense for some scenarios.
The separation of test cases and scenarios will allow piuparts to distinguish between external errors and test failures.
Finally, piuparts outputs very large amounts of log files. It does not entirely reliably recognize when a test has passed or failed: sometimes a test fails because of external reasons, such as the package mirror being off-line. Piuparts does not separate between external failures and real failures. Thus, interpreting the results requires some manual effort, which means piuparts is badly suited for running on the entire package archive. This can be fixed by making piuparts smarter, and to make it output more useful output than a huge log file.
- Alphonse maintains a few packages for Ubuntu. Before uploading them, he wants to make sure they works, so he runs piuparts on them. He wants a quick answer to the question "does this new version of my package break everything horribly?". If something does break, he wants to know what broke so that he can quickly fix it. He wants the testing cycle to go very fast, and give easy answers to what needs fixing.
- Michelle is concerned about the overall quality of Ubuntu. She wants to make sure everything works, and keeps working, even when packges are uploaded. She runs piuparts on all packages in the archive, and wants to know when packages that used to work stop working. She is less concerned about debugging and fixing each broken package than having an overall view of the state of the entire archive.
- Hubert maintains a large subsystem in Ubuntu, upon which large parts of the distribution are built. His upstream makes regular releases, and Hubert wants to know that nothing serious breaks before he uploads each new release. If testing does show that something breaks, he wants to know what it was, so that it can be fixed.
- Piuparts has fast HTTP access to a mirror of the package archive.
- Piuparts is run on a reasonably fast machine.
- Super-user access in the host is always needed for chroot, but must not be required for KVM.
Everything will be a plugin:
- Setting up the virtual environment.
- Installing the base system.
- The test scenarios.
- Individual tests.
- Reporting of results.
There will be a plugin manager that loads and instantiates the plugins, and invokes them at opportune moments.
The exact design of each type of plugin is still open.
The plan for implementing the new version is to start over from scratch, using Test Driven Development, to ensure quality. Given the relatively small size, but fragile code base of the current piuparts, this should be a more efficient way of doing things than adding tests to the existing piuparts code. (Obviously, when old code can be re-used, it will be. However, the current code is not really designed for testability.)
The plugin mechanism is easy: scan for .py files in a list of directories, import them, scan for classes that derive from plugin base classes (one per type of plugin).
The virtualization plugins should also be straighforward: chroot code can be lifted from the current version, and KVM can be based on libvirt. This may become somewhat tricky, however, since the interface needs to be flexible enough to accomodate many kinds of virtualization, but still be simple to avoid complicating the main line of code.
The debootstrap and ubuntu-vm-builder tools will take care of setting up base systems.
Test scenario plugins will adapt the base system in various ways, and should be possible to write as a sequence of apt-get/aptitude/update-manager invocations.
Test cases are reasonably straightforward. Indeed, by isolating them into separate plugins, each with its own single-minded purpose in life, they become quite simple, and thus easy to write.
Reporting of results is easy to implement, but requires some discussion and thought as to what the result should be.
The piuparts UI is the command line. It is not interactive at all. There will be changes to the command line syntax (new options), and the output format will change to something more useful, but no radical changes.
The current piuparts implementation is around 1500 lines of code. Some of it can be re-used, most will probably end up being rewritten.
During implementation, we will add unit tests and black box tests for every code change. Unit tests shall have 100% code coverage (as measured by coverage.py in the python-coverage package).
Black box tests shall test that the whole program works, by running through every scenario and every test using a static set of selected packages. The tests will include packages that pass the tests, and those that fail them.
The test packages will be included in the source tree to avoid the tests from failing in the future, when they are no longer available in the package archive.
Additionally, we will take a snapshot of the Ubuntu archive (possibly only main), and run both the old and new versions of piuparts on it. Both versions should report pass/fail the same way. Any differences will be analyzed manually. If the difference is a bug in the new version, it gets fixed. If it is a bug in the old vewrsion, the package gets added to a blacklist. If the difference is not a bug, we look harder.
- How to clone an existing system to be run as the base for piuparts tests. This would be especially useful for system administrators who want to see what happens if they upgrade their production system to the next release of Ubuntu.