Stable_kABI

Revision 3 as of 2008-06-23 22:38:00

Clear message

Include(KernelTeamHeader)

Discussion on Stable kABI

The problem

In order for Linux to be successful in the marketplace, Independent Hardware Vendors (IHVs) and Independent Software Vendors (ISVs) require stability and predictability. Canonical recognizes this and works with our partners to enable their hardware and software in this stable and predictable manner. However, changes to the kernel to improve functionality, hardware enablement and security can often cause incompatibilities with third party modules both hardware and software.

The Kernel Application Binary Interface (kABI) is the primary hot button where these incompatibilities show up the most. If kABI not kept stable, it is possible that a third party module, weather hardware or software will fail to load after a routine kernel update. This best case would be disruptive. Worst case, the machine will not boot, resorting to rescue methods, known good kernels and lots of downtime. Many enterprise customers demand a stable kABI and have become accustomed this practice due to Red Hat Enterprise Linux (RHEL) & Suse Linux Enterprise Server (SLES) guaranteeing a stable kABI within a major version release.

kABI Issues

By default the Linux kernel builds with numerous kernel modules. Many modules are hardware device drivers, others provide traditional kernel functions, such as networking. Technically speaking, the Linux kernel is a monolithic kernel, that is all modules live in the same address space in the static part of the kernel and, as such have access to all kernel structures.

Modules interact with the kernel by accessing symbols that the kernel or other modules export. Currently the Ubuntu Hardy generic i386 kernel exports about 6900 symbols. Exported symbols along with the data structure layouts and versions info make up the Kernel Module Binary Interface, most commonly referred to as kABI. When compiling the kernel, it is assumed that all modules are compiled along with the static part of the kernel. This allows all data structures, kernel configuration, compiler version, and compiler flags to be in agreement, so all modules and the static kernel can work together. Modules compiled with a different configuration or compiler or even for a different kernel version cause problems. As a guard against these problems the kernel build system allows the versioning of kernel symbols. This mechanism is enabled in Ubuntu kernels. The version numbers are generated automatically by calculating checksums over the function and data structure declarations. The system is designed to not miss a change, so the symbol version changes any time the system suspects that the symbol has changed. This prevents against silent breakage would typically result in kernel memory corruption.

When a module requires a symbol with the wrong version number, the module is not loaded into the kernel. While this will result in loss of functionality it is less harmful than memory corruption.

Generally the kernel symbol versions issue is not a problem for the Linux community, as well as Ubuntu. Ubuntu recompiles the complete kernel, including all shipped modules, for every kernel it releases, so this volatility is not a problem for Canonical. Recompiling the module will usually adapt to changed options or differently laid-out data structures. The problem does affect parties who provide kernel modules that are not part of the Ubuntu distribution, normally partners, both IHV & ISV. This happens because because these modules cannot be recompiled by the Ubuntu. In addition, changes in the kAPI could occur and require changes to the module source code.

Both of these cases are highly undesirable for anyone who provides kernel modules that are not part of the product distribution. Customers expect to receive timely updates that fix critical bugs, security issues hardware enablement.

Historical kABI Data

Going back and pulling repo data on the last two LTS releases, Dapper and Hardy respectively there were numerous kABI bumps to shipped kernels. The table below illustrates the number of changes that went into each of the kABI bumped kernels (marked in red) vs. the non kABI bumped.

Insert Tables

While this data gives an impression on the amount of churn in the kernel it specifically does not tell us exactly how many of the security or updates were actually kABI breakers. This is due to the existing policy of bumping kABI anytime is “was needed”, coupled with loose kABI tracking at changelog commit time.

kABI Stability Options

Why stabilize kABI? In short to keep 3rd party kernel modules from breaking every time Ubuntu releases a kernel that bumps kABI. The classic argument from the upstream Linux community is clean your code up and get it accepted in the upstream kernel. While that would be the optimal choice it is not practical for a variety of reasons, everything from IP issues to upstream refusal. Additionally there are numerous high end Enterprise customers that write their own hooks into the kernel for various operations and wish to have a stable kABI.

Option 1

Need kernel team discussion here Some suggestions:

  • Limit to LTS releases
  • Stable kABI for the 1st two years, then the expectation is to move to the next LTS release.
  • Guarantee a subset of symbols, possibly by polling the lead vendors and analyzing what symbols they actively use.
  • Tightening up our kABI bump policy and bumping only after assessing the risk of not doing it.

Option 2

Need kernel team discussion

Draft kABI Policy Statement

Need kernel team discussion


CategoryKernel