Python2.6And3.0

Revision 3 as of 2009-01-30 18:06:16

Clear message
  • Launchpad Entry: python-for-jaunty

  • Created:

  • Contributors:

  • Packages affected: python2.6, python3.0, python-defaults

Summary

Introduce new versions of python for the 2.x series (2.6) and python3.0.

Release Note

This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)

It is mandatory.

  • Please fill this in before approval, or delete the section if you don't feel that a release note is needed (it isn't always). --ColinWatson

Rationale

We'll introduce new python versions and drop older python versions for jaunty, and addressing issues with the current way of packaging of python extensions.

  • Python 3.0 introduces language changes and in almost all cases requires different source for 2.x and 3.0. To keep third party modules for 2.x and 3.0 independent on the other python versions, the packaging in a different binary package is proposed.
  • The distutils currently packaged with python is configured to install in the same location for manually installed packages and for packages included in the distribution, leading to upgrade problems and inconsistencies in the installations.
  • Different packaging tools currently install in different locations, breaking assumptions of upstream software. An upstream package should not behave differently when packaged with a different packaging helper. This includes relative imports, import order of modules and extensions and upstream plugin systems.
  • Some packages current have their files installed in package and helper specific directories on the file system, not using the conflict checking in the dpkg and apt packaging tools and deferring this to the python packaging tools. To robustify this, we install files which are not directly installed into sys.path in a common location (/usr/share/pyshared for 2.x, and /usr/share/py3shared), with the prefix /usr/lib/pythonx.y/{dist,site}-packages dropped.
  • To handle python transitions more gracefully, we currently package third party modules in a way which is independent of the python version, not adding the support for a specific python version as files/symlinks in the package itself, but creating this support during package installation. Depending on the packaging tool this is either incomplete or seen as scary. During installation a python module is not available for use. Robustify the packaging so that a module is already usable after unpacking and no action (besides byte-compiling) is needed.
  • Provide a common way to handle so called namespace packages using the tools already available in the distribution (diversions).
  • Removing support for one python version currently leads to uninstallability of packages and requires rebuilds of depending packages (extensions). Packages needing a python version which is not the default should handle these dependencies themselves, and not burden the archive with housekeeping.
    • What does "handle these dependencies themselves" mean in this case? I thought packages needing a non-default Python version already had to depend explicitly on pythonX.Y (or perhaps python (>= X.Y)) rather than python? --ColinWatson

Design

Python 2.6

The upgrade to 2.6 as the default python version is a bit easier than the update from 2.4 to 2.5. 2.6 didn't see that many changes on the extension site as 2.5 did have. The upgrade should be done in two steps:

  • 1) Build extension and modules for 2.6 as an additional supported python version. 2) Make 2.6 the default python version. 3) Optionally drop support for python 2.5 in jaunty.
  • This appears to be three steps, not two ... --ColinWatson

Python 3.0

A python 3.0 module has different source code than for 2.x. Our current helper support doesn't support different source code for one module that well, and package and installation sizes do increase. Package modules for 3.0 in binary packages python3-foo to be used for python versions 3.x. The binary package for 3.0 can be built out of the same source package, but doesn't have to.

Installation locations

Currently /usr/lib/pythonx.y/site-packages is used for installation of python extensions and modules, plus an additional directory /usr/local/lib/pythonx.y/site-packages, the latter conflicting with the site-packages with a python installation with --prefix=/usr/local.

  • Use /usr/lib/pythonX.Y/dist-packages for the installation of all public modules and extensions included in the distribution. Add a new option to distutils --install to install into this location (--install-layout=deb).
  • Use /usr/local/lib/pythonx.y/dist-packages for the installation of locally installed modules and extensions to be used with the python included within the distribution. This is the default location when using distutils.

Names for these directories discussed with Martin v. Löwis and Barry Warsaw; the names are not perfect, but we did want to stick to the same directory in /usr and /usr/local.

Installation of shared files

Files which can be shared between different Python versions are installed into /usr/pyshared. The layout inside /usr/pyshared is the same which can be currently found in /usr/lib/pythonx.y/site-packages. There is no package specific subdirectory in /usr/pyshared/.

Files shared between different python3.x versions are installed into /usr/py3shared.

  • We shouldn't create new subdirectories of /usr. This should be /usr/share/py*shared/ or /usr/lib/py*shared/, depending on the contents (if it contains any architecture-dependent portions and can't easily be split across /usr/share/ and /usr/lib/, then put it all in /usr/lib/), noting that /usr/share/pyshared/ is already in use so if you use that then you'll need to be compatible in whatever ways are relevant. --ColinWatson

The symlinks to the files available in sys.path should be part of the package and created during packaging (as the default, keep the old behaviour as an option).

  • Won't this have the consequence that we go back to the old behaviour where we had to rebuild every Python module package every time we introduced a new version of Python? I'd hoped we could find a compromise between these two extremes. What happens to dependencies if packages contain fixed symlinks? --ColinWatson

Python 2.4

Keep the python2.4 package, but drop support building extensions for all available packages. For some applications like zope2.x and plone we still need some extensions built for 2.4.

Implementation

  • Helper tools (dh_py*):
    • Search for stuff in dist-packages and do the right stuff.
    • Add options to include the symlinking in the package.
    • Extend support for python3.x
  • PythonX.Y
    • new option for distutils --install: --install-layout=deb
    • Add py_compilefiles to the core packages.
  • Python modules and extensions: Compatibility fixes for 2.6.
    • Perhaps add a link here to the upstream release notes with typical changes that are needed. --ColinWatson

Unresolved issues

It looks like Python 3.0 is very short-lived. A Python 3.1 release is targeted for April 2009, which is too late for Jaunty. To be able to build our modules and extensions from the same source, Python 3.0 must be included into main.


CategorySpec