Stable Kernel Testing
Some types of testing are more valuable than others for Linux kernels. This page will explain the differences between them and why some have more value. For the purposes of this discussion, testing will be divided into "verification testing", which verifies that specific bugs are fixed, and "regression testing", which in this case means making sure that a system boots and has some minimal functionality (Boots, Runs X, has networking, and can perform updates).
The reasons that the kernel requires different testing than applications are these:
- The kernel contains the drivers which interface directly to the hardware. In order to test specific device drivers, the actual hardware supported by those drivers must be present
- Problems reported on specific brands and models of hardware may only be reproducible on that exact hardware.
- Virtual execution environments present a set of virtual devices to the kernel, which do not use the same drivers that the majority of users do. For example, if networking works on virtualized kernel, this only means that the virtualization driver for networking and networking on the host kernel are functional.
- Virtual execution environments may present abstracted file systems that do not correspond to real storage device physical geometry, preventing file system problems from being detected.
- Virtualized execution environments have locking and mutual exclusions that result in single thread behavior in some situations that real hardware won't exhibit.
- In a virtualized kernel, functionality like cache clearing and tlb flush may be emulated by the VM, and not provide a valid test.
Bare metal testing
- This is usually the only sort of testing that is acceptable for hardware-specific fixes during verification testing.
- This is the most valuable sort of testing for regression testing, in order to assure that storage and networking are functional.
Testing on Virtualized Hardware
- This is useful for a very simple boot test during regression testing to assure that nothing is grossly wrong with the kernel. It does almost nothing to assure us that X will run, since graphics drivers are specific to hardware, and in a virtualized environment, VESA or VGA graphics are used.
- This is useful for testing kernels which are designed to run as guest kernels in a virtualized environment (server only).
- When following the Ubuntu stable kernel cadence, by the time regression testing begins, the kernels to be test have already been boot tested by the kernel team on a few machines, and additional testing in a virtualized machine is not terribly useful, except for server (uec) kernels.