Launchpad Entry: python-for-jaunty
Created:
Contributors:
Packages affected: python2.6, python3.0, python-defaults
Summary
Introduce new versions of python for the 2.x series (2.6) and python3.0.
Rationale
We'll introduce new python versions and drop older python versions for jaunty, and addressing issues with the current way of packaging of python extensions.
- Python 3.0 introduces language changes and in almost all cases requires different source for 2.x and 3.0. To keep third party modules for 2.x and 3.0 independent on the other python versions, the packaging in a different binary package is proposed.
- The distutils currently packaged with python is configured to install in the same location for manually installed packages and for packages included in the distribution, leading to upgrade problems and inconsistencies in the installations.
- Different packaging tools currently install in different locations, breaking assumptions of upstream software. An upstream package should not behave differently when packaged with a different packaging helper. This includes relative imports, import order of modules and extensions and upstream plugin systems.
- Some packages current have their files installed in package and helper specific directories on the file system, not using the conflict checking in the dpkg and apt packaging tools and deferring this to the python packaging tools. To robustify this, we install files which are not directly installed into sys.path in a common location (/usr/share/pyshared for 2.x, and /usr/share/py3shared), with the prefix /usr/lib/pythonx.y/{dist,site}-packages dropped.
- To handle python transitions more gracefully, we currently package third party modules in a way which is independent of the python version, not adding the support for a specific python version as files/symlinks in the package itself, but creating this support during package installation. Depending on the packaging tool this is either incomplete or seen as scary. During installation a python module is not available for use. Robustify the packaging so that a module is already usable after unpacking and no action (besides byte-compiling) is needed.
- Provide a common way to handle so called namespace packages using the tools already available in the distribution (diversions).
- Removing support for one python version currently leads to uninstallability of packages and requires rebuilds of depending packages (extensions). Packages needing a python version which is not the default should handle these dependencies themselves, and not burden the archive with housekeeping. Background: The python-support helper currently adds dependencies for extensions of the form pythonX.Y-foo for each python version that the extension supports, and it adds provides in the python-foo package of the form pythonX.Y-foo. Dropping support for one python version leads to uninstallability of all these depending packages until these are rebuilt.
Design
Python 2.6
The upgrade to 2.6 as the default python version is a bit easier than the update from 2.4 to 2.5. 2.6 didn't see that many changes on the extension site as 2.5 did have. The upgrade should be done in two steps plus an optional third step:
- Build extension and modules for 2.6 as an additional supported python version.
- Make 2.6 the default python version.
- Optionally drop support for python 2.5 in jaunty.
Python 3.0
A python 3.0 module has different source code than for 2.x. Our current helper support doesn't support different source code for one module that well, and package and installation sizes do increase. Package modules for 3.0 in binary packages python3-foo to be used for python versions 3.x. The binary package for 3.0 can be built out of the same source package, but doesn't have to.
It looks like Python 3.0 is very short-lived. A Python 3.1 release is targeted for April 2009, which is too late for Jaunty. We will provide basic support for 3.0 (using the 3.0.1 release).
To be able to build our modules and extensions from the same source, Python 3.0 must be included into main.
Installation locations
Currently /usr/lib/pythonx.y/site-packages is used for installation of python extensions and modules, plus an additional directory /usr/local/lib/pythonx.y/site-packages, the latter conflicting with the site-packages with a python installation with --prefix=/usr/local.
- Use /usr/lib/pythonX.Y/dist-packages for the installation of all public modules and extensions included in the distribution. Add a new option to distutils --install to install into this location (--install-layout=deb).
- Use /usr/local/lib/pythonx.y/dist-packages for the installation of locally installed modules and extensions to be used with the python included within the distribution. This is the default location when using distutils.
Names for these directories discussed with Martin v. Löwis and Barry Warsaw; the names are not perfect, but we did want to stick to the same directory in /usr and /usr/local.
Installation of shared files
Files which can be shared between different Python versions are installed into /usr/share/pyshared. The layout inside /usr/share/pyshared is the same which can be currently found in /usr/lib/pythonx.y/site-packages. There is no package specific subdirectory in /usr/share/pyshared/.
Files shared between different python3.x versions are installed into /usr/share/py3shared.
The symlinks to the files available in sys.path should be part of the package and created during packaging (as the default, keep the old behaviour as an option).
Won't this have the consequence that we go back to the old behaviour where we had to rebuild every Python module package every time we introduced a new version of Python? I'd hoped we could find a compromise between these two extremes. What happens to dependencies if packages contain fixed symlinks? --ColinWatson
- We have to do the rebuild for extensions anyway. This adds rebuilds for binary-indep python packages as well. These new rebuilds can be done automated (binNMU'd). Dependencies for these binary-indep packages become the same as for binary-arch packages. We're not back at the old extreme of building different packages for different python versions.
- The Ubuntu release managers have concerns about rebuilding all architecture-independent packages each time when a new python version is introduced into the archive. The impact on the archive and the archive mirrors seems to be too big (Note: Migration to testing in Debian is not affected, because no new dependencies on architecture dependent packages are introduced). For the jaunty release we will have a compromise of having the symlinks included in each package which is included on the CD, avoiding any runtime dependence on python-central or python-support for these packages.
Python 2.4
Keep the python2.4 package, but drop support building extensions for all available packages. For some applications like zope2.x and plone we still need some extensions built for 2.4.
Implementation
- Helper tools (dh_py*):
- Search for stuff in dist-packages and do the right stuff.
- Add options to include the symlinking in the package.
- Extend support for python3.x
- PythonX.Y
- new option for distutils --install: --install-layout=deb
- Add py_compilefiles to the core packages.
- Python modules and extensions: Compatibility fixes for 2.6.
See http://docs.python.org/whatsnew/2.6.html for changes.