Create a mechanism for fully automated packaging of python modules based on an upstream release. Support different Python implementations and different versions of CPython (needed, when not all software can run with the latest/default python version when an Ubuntu release is going to happen).


To make as much Python software as possible available for immediate and trivial installation in Ubuntu. Adding Java and Mono support should include support for Python implementations on these platforms as well (IronPython, Jython).

Scope and Use Cases

  • A new Python module appears on PyPI: it should appear in Ubuntu a short time later
  • A new release of a Python module appears: it should appear in Ubuntu a short time later
  • Additional, non-automatable packaging metadata must be added to the package: it should be preserved across future automated updates
  • A new CPython release is incorporated into Ubuntu: all existing pure Python modules should be made available to it, and native code modules built for it, within a short time (or immediately)
  • A new Python implementation is added to the system: all installed modules (provided they are compatible with this new version) are available without user interaction.

Implementation Plan

Cheese Shop (PackagesFromPyPi)

  • Suggest (write PEP) to add more metadata information. This is required for an automatic packaging attempt.
    • Fields to add:
    • Copyright / Licensing (already there? <VincenzoDiMassa>)

    • Build dependencies which are not Python modules
    • Documentation Files
    • Python version required
  • Backport changes to distutils in 2.5 to 2.4 packages, and/or update python-setuptools, to support newer metadata

"" needed for PyPi packaging""

Module packaging

  • Add some of the PyPi metadata to the python package (i.e. Module-Provides)

  • For architecture independent packages:
    • build one binary package and specify which python implementations it works with
    • TODO later: if different versions are required to run with packaged python implementations, package different versions of the module in different source packages.
      • Example: the lastest version of foo uses generator expressions, which cannot be used with jython.
  • Architecture dependent packages:
    • Small extension modules (.so file smaller than some threshold value, 50k?):
      • The extensions for all compatible python implementations/versions are put into the same binary package.
    • Large extension modules:
      • Do as before: put them in a separate binary package (python2.y-foo), add a dependency package.
  • Add two packages 'python-all' and 'python-all-dev'.
    • These packages depend on all supported runtime / development versions supported by the distribution. Use these packages as build dependencies for python modules, to determine for which python implementations/versions a module package is supposed to build. Using this information, a Python module package can extract the default Python version, and all Python versions it needs to build for, from the build environment. When generating the control file, the module's package does not need to be modified when the default Python version is changed or support for a new version and/or implementation is added.
  • Add a new field to the control file for binary packages, X-Python-Package. The field should list all python versions which the module/extension is supposed to install to.
    • A special value of "all" causes installation to all Python versions (so that architecture independent modules do not have to be re-uploaded when a new Python implementation/version enters the distribution). We have to use a new field, because we cannot determine all installed python modules when installing a new Python version.
  • Provide a template (dh_make, debhelper) to package a Python module / a python application.

Python Registry

  • The registry provides a set of scripts used in the installation and removal process of packages.
  • Python modules are installed in a directory tree which is managed by the registry.
    • No directory of the registry is part of sys.path of any python implementation.
  • On package installation: modules are made available to every compatible installed Python implementation on the system (technically: copy the directory tree and symlink every file).
    • The files are then byte-compiled.
  • On package removal: remove files/directories created during installation.
    • Delete byte-compiled files.
  • When installing a new python implementation, all installed python modules are made available to it.
    • Then, byte-compile the files.

Automated Cheese Shop packaging

  • Current Python packages lack PyPi metadata information. Until these are added, we are not able to resolve PyPi's Depends: fields.

  • PyPi uses its own naming scheme (First come, first wins) for package names and the names for dependencies (external Depends).

    • We need a map from PyPi to Debian package names.

  • We need to preserve non-automatable packaging metadata (i.e., files manually added to the package) across future automated updates.
    • Supported python versions / implementations
    • Packaging decisions (e.g.: architecture-dependent packages: seperate or combined?).
  • Design and implement the packaging script.

Packaging IronPython

  • Depends on the current version of Mono entering breezy

Packaging Jython

  • Depends on Java beeing built with updated toolchain
  • Modify for python registry

Data Preservation and Migration

Packages Affected

  • All packages providing Python implementations
  • All python module packages and packages having internal python modules. Existing python packages continue to work and can be converted over time.

User Interface Requirements

Outstanding Issues

  • Determine priorities
  • Just add metadata fields to the control file, or use a separate metadata database (like e.g. doc-base)?

UDU BOF Agenda

  • Methodology for simple and painless packaging of Python modules
    • High-level tools? CDBS?
    • Take maximum advantage of upstream packaging facilities
  • Packaging PyPI
    • Interplay between hand-packaged and auto-packaged modules
    • Auto-packaging new modules as they appear
    • Auto-packaging new releases as they appear
  • Simplify transitions
    • Simplify packaging of pure Python modules (only one binary package)
    • Auto-generated packaging for native Python modules?
  • Packaging modules for IronPython (Mono) and Java (jython)

UDU Pre-Work

UbuntuDownUnder/BOFs/PythonModulePackaging (last edited 2008-08-06 16:37:22 by localhost)