Ubuntu Wiki

• Launchpad Entry: get-rid-of-python-central-and-support

• Created: 2008-06-02

• Contributors: LarsWirzenius

• Packages affected: python, python-central, python-support (+ everything Python related)

Summary

The python-central and python-support packages will be removed. They are currently needed so that Python modules and extensions can be installed once per system, rather than once per installed Python interpreter. This will require changing the Python interpreter to support having byte code files and extension .so files for different versions of the interpreter in the same directory.

Release Note

Python packaging will change somewhat. Packages using debhelper or cdbs will require only minor changes to their source. Users of packages will see no change.

Rationale

The main Python implementation, also known as CPython, does not provide a stable ABI for its byte code files or binary extension (.so files). This means that Python modules and extensions (usually written in C) must be compiled for each version of the interpreter separately.

Originally this was done by having a binary package specific to each Python interpreter in the archive: a package providing the foo extension needed to provide the binary packages python2.3-foo and python2.4-foo to support both version 2.3 and 2.4 of the interpreter.

This approach had several drawbacks:

• Adding a new version of the interpreter required updating all Python module and extension packages, and usually this required source changes.
• There was a large number of unnecessary binary packages in the archive.
• If you had more than one version of the interpreter in the archive, you needed multiple copies of the module source code (.py files), which wastes disk space.

Eventually in 2006 Debian (and then Ubuntu) changed to support a new approach. The packaging helper tools python-central and python-support were implemented. Now:

• Each module and extension source package declares which interpreter versions it supports. Most of them declare that they support all of them.
• Binary packages now bundle support for all interpreter versions in

  the same package. Where there previously were python2.3-foo and python2.4-foo, there is now only python-foo. This reduces the number of binary packages a lot, and simplifies transitions.

• When a new Python interpreter version is added to the archive, most module and extension packages can just be rebuilt, without source changes. This the transition can mostly be handled by the archive maintainers in an automated way.

The new approach has serious benefits, which must be retained. It does have some additional problems, however. The main one is the added complexity: all Python module and extension packages need special handling in their postinst and postrm scripts. Although most packages get this from python-central and python-support, it is still more complicated and fragile than is desired. It is also undesired to have two helper packages doing essentially the same thing, but with their own individual sets of bugs.

Use Cases

• Anatole maintains a package of totleigh, a pure Python module that works with any Python interpreter version in the 2.x series.

• Bertram maintains a package of dahlia, an extension to Python written in C that also supports all Python interpreter versions in the 2.x series.

Assumptions

At least in the initial drafts of this spec, we will assume Python 2.x. Python 3.x will not be entirely backwards compatible even at the syntax level, so the transition from 2.x will require a lot of careful thinking.

Design

The basic problem is that Python byte code files (.pyc and .pyo) and extensions (.so) are not compatible between major releases, although they are between minor revisions of the major releases (2.x.y and 2.x.z are compatible; 2.x and 2.y are not).

The design to solve this is to embed the version number in the name of the byte code and extension files.

Implementation

• Add a /usr/lib/python2-site-packages directory. This will be created in the postinst of all Python interpreter packages, and removed in their postrm, if it's empty.
• Change Python interpreter packages to add that directory to their search path.
• Packages will place modules and extensions there if they work with any version of 2.x.
• A module is named foo.py.

• A package is named foo/ (i.e., foo/init.py exists).

• An extension is named foo_x.y.so (where x.y is the version number of the Python interpreter it was built against, e.g. 2.3 or 2.4).
• The Python interpreter is modified to look for foo_x.y.pyc before foo.pyc, and foo_x.y.pyo before foo.pyo.
• The Python interpreter is modified to optionally produce foo_x.y.pyc and foo_x.y.pyo when compiling. This will require setting an option, the default will still be foo.pyc and foo.pyo.
• The Python interpreter packages will be changed to have triggers that compiled/remove version specific byte code files.
• Python module and extension packages will be changed to invoke the trigger.

debhelper and CDBS Changes

FIXME: This needs more thought.

Migration

FIXME: Needs thought.

Test/Demo Plan

• Implement the spec in a private apt repository.
• Rebuild all affected packages.
• Test upgrade from hardy (Ubuntu), and etch (Debian).
• After upgrade, verify that system and packages work, by executing some automatic tests for selected packages.

Outstanding Issues

Ubuntu wants to implement this early on in the intrepid development cycle, which means summer of 2008. Debian is nearing its freeze and is unlikely to want to adopt the new Python packaging approach this late in the development cycle. Thus it may be necessary to implement this in Ubuntu first, and port it to Debian after lenny has been released.

It is possible that these changes should be discussed with Python upstream. This is particularly true if they can't be implemented by minor changes to search paths, and the module/extension loaders.

CategorySpec