Ubuntu Wiki

• Launchpad Entry: desktop-karmic-automagic-python-build-system

• Created: 2009-05-23

• Contributors: MartinPitt, RickSpencer

• Packages affected: python-distutils-extra, quickly

Summary

In this blueprint we propose extending python-distutils-extra by a "do what I mean" mode where it automatically chooses the right destination path and action for various known file types such as Python modules, exectuable scripts, icons, desktop files, etc. The goal is that the typical setup.py only has to contain meta information like description, version, and author.

We also discuss Debian packaging generation.

Release Note

Ubuntu 9.10 now provides a fully automatic Python distutils based build system for Python applications in the python-distutils-extra package. Following a "convention over configuration" approach, this automatically determines most build and install actions for gettext, GtkBuilder, Qt UI, Python packages/modules, etc. For simple cases, setup.py only needs to contain package metadata, and POTFILES.in, setup.cfg, and MANIFEST.in are not required at all.

Rationale

Python's native and very popular build systems is "distutils" which comes shipped with Python itself. It is fairly limited, though, and requires the user to learn a lot about build systems. python-distutils-extra adds some functionality for i18n, but there are still some standard use cases missing.

With our goal of supporting the opportunistic programmer and our pre-selection of technologies, a lot of the build system design can be made implicit ("convention over configuration"), which will greatly reduce the amount of learning necessary for installing a project and building a package.

User stories

Jockey

The current Jockey project needs a very large and redundant setup.py, setup.cfg, and POTFILES.in, and MANIFEST.in.

With the improved distutils-extras, setup.cfg, POTFILES.in, and MANIFEST.in can be dropped completely, and setup.py can be dropped to the metadata:

from DistUtilsExtra.auto import setup

setup(
    name='jockey',
    version='0.5.1',
    description='UI for managing third-party and non-free drivers',
    url='https://launchpad.net/jockey',
    license='GPL v2 or later',
    author='Martin Pitt',
    author_email='martin.pitt@ubuntu.com',
)

Assumptions

• Uses standard PyGTK and Python libraries and technologies.
• Automatically generated Debian package produces single binary only. No library/public package building.

Scope

This project is by and large an upstream build system extension. It does not deal in any way with distribution, code hosting, revision control, Launchpad, etc. These need to be handled in a higher level in the stack.

Design

Project structure

• All directories with __init__.py are Python packages which are used and shipped.

• Most files are identified by extension and/or contents, not by location.
• Use conventions for locations of files where it is ambiguous whether to install them, and where (such as executable scripts).

setup.py

For a standard project which uses well-known file types and a standard directory layout, setup.py should only contain metadata about the project which cannot be inferred from the source code.

Required fields:

• name
• version
• license
• description
• author

Optional fields:

• author_email
• url (homepage)

sdist

DistUtilsExtra.auto already knows which files in the source tree are "source" and which ones merely build products/noise. This can be turned into a reasonable implicit default MANIFEST.in. An existing file supersedes the automatic implicit one.

Debian packaging

• fully automatically generated from scratch, working, Policy compliant

Implementation

distutils-extra extensions

Current python-distutils-extra needs to learn about compiling PyKDE .ui files to .py files with pykdeuic4.

The existing functionality in distutils should be proposed upstream as a PEP and ideally merged into standard distutils gradually.

distutils-extra automatic setup.py

DistUtilsExtra.auto will support the following file types and automatically determine the default values in setup.py:

• Python packages (directory with __init__.py)

• D-Bus configuration (*.conf with magic string)

• D-Bus service (*.service with magic string)

  • If it has an User= line → system service, otherwise session service

• .desktop.in

• .policy.in

• .notifyrc.in

• *.po, build *.pot: already provided by distutils-extras

• Icons in data/icons/size/category/*.{png,svg}: already provided by distutils-extras

• po/POTFILES.in

• cmdclass (default to distutils-extra classes)

• supplementary data files: data/foo → /usr/share/project/foo

• scripts: all in ./bin/* and ./projectname which are executable

DistUtilsExtra.auto's goals of "convention over configuration" conflicts with the upstream paradigm of "explicit is better than implicit", and thus will most likely not be considered upstream. We will maintain this project in Debian/Ubuntu for now.

Debian packaging

Based on the information in setup.py and the source code, we can also generate a working Debian packaging.

The van.pydeb project (packages, svn) [already provides a some mechanics, so this should be used by distutils-extra if appropriate.

python-distutils-extra will provide a new command python-mkdebian which generates a debian/ directory with the necessary files from the information contained in the Python .egg-info:

changelog

• project name, version, author, email from setup.py
• description: "new release"

• distro target: needs argument from environment; default to lsb_release -c?

• If already exists, will just add a new changelog entry for a new version.

compat

• constant (6 for hardy support)
• Do not change if already existing.

rules

• static, cdbs+python-distutils.mk
• Do not change if already existing.

copyright

• Grep source for "([cC])" statements and add them.
• Support some standard values of "license" and add GPL stub and author name.
• Do not change if already existing.

control

• no dh_install, upstream build system DTRT
• Source, Package, Maintainer, Description: from setup.py
• Section, Priority, Standards-Version, XS-Python-Version: constant
• Build-Depends: static (no tests), or binary depends (if test cases available)
• Depends: grep all *.py for import statements, check which package provides them

  • ./setup.py can also specify them explicitly with the requires keyword

  • Debian specific code needs to find out which package ships it; this is a heuristic, but should get it right in 80% of the cases at least

  • The Debian Python Modules Team has a couple of scripts to do this

• If already existing, update build/binary dependencies.

Test/Demo Plan

python-distutils-extra contains an automatic test suite which covers all functionality of the DistUtilsExtra.auto automatic build system. It is run on package build.

For a manual test, check out lp:jockey (upstream trunk), and confirm that it does not have any of setup.cfg, MANIFEST.in, POTFILES.in, and only a very minimal setup.py. Test the following:

• ./setup.py install --root=/tmp/x does a complete and working installation

• ./setup.py sdist produces a complete source tarball

• python-mkdebian produces a working debian/ tree; calling debuild produces a working .deb, no lintian errors, and only trivial lintian warnings (just "binary without manpage" at the time of this writing)

CategorySpec