AutomagicPythonBuildSystem

Differences between revisions 4 and 17 (spanning 13 versions)
Revision 4 as of 2009-05-23 20:07:37
Size: 6159
Editor: 86
Comment:
Revision 17 as of 2009-07-29 08:00:04
Size: 8616
Editor: pD9EB3788
Comment:
Deletions are marked like this. Additions are marked like this.
Line 20: Line 20:
This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)

It is mandatory.
Ubuntu 9.10 now provides a fully automatic Python distutils based build system
for Python applications in the `python-distutils-extra` package. Following a
"convention over configuration" approach, this automatically determines most
build and install actions for gettext, GtkBuilder, Qt UI, Python
packages/modules, etc. For simple cases, `setup.py` only needs to contain
package metadata, and `POTFILES.in`, `setup.cfg`, and `MANIFEST.in` are not
required at all.
Line 26: Line 30:
One of Python's native and very popular build systems is "distutils" which
comes shipped with Python itself. It is fairly limited, though, and requires
the user to learn a lot about build systems. python-distutils-extra adds some
Python's native and very popular build systems is "distutils" which comes
shipped with Python itself. It is fairly limited, though, and requires the user
to learn a lot about build systems. python-distutils-extra adds some
Line 42: Line 46:
[[http://bazaar.launchpad.net/%7Ejockey-hackers/jockey/trunk/annotate/548/setup.cfg|POTFILES.in]].

With the improved distutils-extras, `setup.cfg` and `POTFILES.in` can be dropped completely, and `setup.py` can be dropped to the metadata:
[[http://bazaar.launchpad.net/%7Ejockey-hackers/jockey/trunk/annotate/548/POTFILES.in|POTFILES.in]], and
[[http://bazaar.launchpad.net/%7Ejockey-hackers/jockey/trunk/annotate/548/MANIFEST.in|MANIFEST.in]]
.

With the improved distutils-extras, `setup.cfg`, `POTFILES.in`, and `MANIFEST.in` can be dropped completely, and `setup.py` can be dropped to the metadata:
Line 52: Line 57:
Line 66: Line 70:
== Scope ==

This project is by and large an upstream build system extension. It does not
deal in any way with distribution, code hosting, revision control, Launchpad,
etc. These need to be handled in a higher level in the stack.
Line 70: Line 80:
 * All directories with `__init__.py` are Python packages which are used and shipped
 * TBD: Scripts in `./bin/` ? `main.py`? ''projectname''`.py`?
 * All directories with `__init__.py` are Python packages which are used and shipped.
 * Most files are identified by extension and/or contents, not by location.
 * Use conventions for locations of files where it is ambiguous whether to install them, and where (such as executable scripts).
Line 92: Line 103:
=== sdist ===

`DistUtilsExtra.auto` already knows which files in the source tree are "source" and which ones merely build products/noise. This can be turned into a reasonable implicit default `MANIFEST.in`. An existing file supersedes the automatic implicit one.
Line 100: Line 115:
 * Compiling PyKDE .ui files to .py files with `pykdeuic4` Current python-distutils-extra needs to learn about compiling PyKDE .ui files
to .py files with `pykdeuic4`.

The existing functionality in distutils should be proposed upstream as a PEP and ideally merged into standard distutils gradually.
Line 103: Line 121:

`DistUtilsExtra.auto` will support the following file types and automatically determine the default values in `setup.py`:
Line 127: Line 147:
 * scripts: TBD  * scripts: all in `./bin/*` and `./`''projectname'' which are executable

`DistUtilsExtra.auto`'s goals of "convention over configuration" conflicts with
the upstream paradigm of "explicit is better than implicit", and thus will most
likely not be considered upstream. We will maintain this project in
Debian/Ubuntu for now.
Line 134: Line 159:
TBD:
 * integration? `./setup.py debian` ?
 * into distutils-extras or quickly or ???
 * create if not existing/update if existing? (changelog/dependencies)
The [[http://pypi.python.org/pypi/van.pydeb/|van.pydeb]] project
([[http://packages.vanguardistas.net/public/pool/main/v/van.pydeb/|packages]],
[[http://svn.debian.org/viewsvn/pkg-zope/van.pydeb/trunk/debian/|svn]])
[already provides a some mechanics, so this should be used by
distutils-extra if appropriate.

`python-distutils-extra` will provide a new command `python-mkdebian` which generates a `debian/` directory with the necessary files from the information contained in the Python `.egg-info`:
Line 144: Line 172:
 * If already exists, will just add a new changelog entry for a new version.
Line 147: Line 176:
constant (6 for hardy support)

==== rules =====
 * constant (6 for hardy support)
 * Do not change if already existing.

==== rules ====
Line 152: Line 182:
 * Do not change if already existing.
Line 154: Line 185:
TBD
 * Grep source for "([cC])" statements and add them.
 * Support some standard values of "license" and add GPL stub and author name.
 * Do not change if already existing.
Line 163: Line 197:
   * `./setup.py` can also specify them explicitly with the `requires` keyword
   * Debian specific code needs to find out which package ships it; this is a heuristic, but should get it right in 80% of the cases at least
   * The Debian Python Modules Team has a [[http://svn.debian.org/viewsvn/python-modules/tools/|couple of scripts]] to do this
 * If already existing, update build/binary dependencies.
Line 166: Line 204:
It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

== Unresolved issues ==

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

== BoF agenda and discussion ==

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.
`python-distutils-extra` contains an automatic test suite which covers all
functionality of the `DistUtilsExtra.auto` automatic build system. It is run on
package build.

For a manual test, check out `lp:jockey` (upstream trunk), and confirm that it
does not have any of `setup.cfg`, `MANIFEST.in`, `POTFILES.in`, and only a very
minimal `setup.py`. Test the following:

 * `./setup.py install --root=/tmp/x` does a complete and working installation
 * `./setup.py sdist` produces a complete source tarball
 * `python-mkdebian` produces a working `debian/` tree; calling `debuild` produces a working `.deb`, no lintian errors, and only trivial lintian warnings (just "binary without manpage" at the time of this writing)

Summary

In this blueprint we propose extending python-distutils-extra by a "do what I mean" mode where it automatically chooses the right destination path and action for various known file types such as Python modules, exectuable scripts, icons, desktop files, etc. The goal is that the typical setup.py only has to contain meta information like description, version, and author.

We also discuss Debian packaging generation.

Release Note

Ubuntu 9.10 now provides a fully automatic Python distutils based build system for Python applications in the python-distutils-extra package. Following a "convention over configuration" approach, this automatically determines most build and install actions for gettext, GtkBuilder, Qt UI, Python packages/modules, etc. For simple cases, setup.py only needs to contain package metadata, and POTFILES.in, setup.cfg, and MANIFEST.in are not required at all.

Rationale

Python's native and very popular build systems is "distutils" which comes shipped with Python itself. It is fairly limited, though, and requires the user to learn a lot about build systems. python-distutils-extra adds some functionality for i18n, but there are still some standard use cases missing.

With our goal of supporting the opportunistic programmer and our pre-selection of technologies, a lot of the build system design can be made implicit ("convention over configuration"), which will greatly reduce the amount of learning necessary for installing a project and building a package.

User stories

Jockey

The current Jockey project needs a very large and redundant setup.py, setup.cfg, and POTFILES.in, and MANIFEST.in.

With the improved distutils-extras, setup.cfg, POTFILES.in, and MANIFEST.in can be dropped completely, and setup.py can be dropped to the metadata:

from DistUtilsExtra.auto import setup

setup(
    name='jockey',
    version='0.5.1',
    description='UI for managing third-party and non-free drivers',
    url='https://launchpad.net/jockey',
    license='GPL v2 or later',
    author='Martin Pitt',
    author_email='martin.pitt@ubuntu.com',
)

Assumptions

  • Uses standard PyGTK and Python libraries and technologies.
  • Automatically generated Debian package produces single binary only. No library/public package building.

Scope

This project is by and large an upstream build system extension. It does not deal in any way with distribution, code hosting, revision control, Launchpad, etc. These need to be handled in a higher level in the stack.

Design

Project structure

  • All directories with __init__.py are Python packages which are used and shipped.

  • Most files are identified by extension and/or contents, not by location.
  • Use conventions for locations of files where it is ambiguous whether to install them, and where (such as executable scripts).

setup.py

For a standard project which uses well-known file types and a standard directory layout, setup.py should only contain metadata about the project which cannot be inferred from the source code.

Required fields:

  • name
  • version
  • license
  • description
  • author

Optional fields:

  • author_email
  • url (homepage)

sdist

DistUtilsExtra.auto already knows which files in the source tree are "source" and which ones merely build products/noise. This can be turned into a reasonable implicit default MANIFEST.in. An existing file supersedes the automatic implicit one.

Debian packaging

  • fully automatically generated from scratch, working, Policy compliant

Implementation

distutils-extra extensions

Current python-distutils-extra needs to learn about compiling PyKDE .ui files to .py files with pykdeuic4.

The existing functionality in distutils should be proposed upstream as a PEP and ideally merged into standard distutils gradually.

distutils-extra automatic setup.py

DistUtilsExtra.auto will support the following file types and automatically determine the default values in setup.py:

  • Python packages (directory with __init__.py)

  • D-Bus configuration (*.conf with magic string)

  • D-Bus service (*.service with magic string)

    • If it has an User= line → system service, otherwise session service

  • .desktop.in

  • .policy.in

  • .notifyrc.in

  • *.po, build *.pot: already provided by distutils-extras

  • Icons in data/icons/size/category/*.{png,svg}: already provided by distutils-extras

  • po/POTFILES.in

  • cmdclass (default to distutils-extra classes)

  • supplementary data files: data/foo/usr/share/project/foo

  • scripts: all in ./bin/* and ./projectname which are executable

DistUtilsExtra.auto's goals of "convention over configuration" conflicts with the upstream paradigm of "explicit is better than implicit", and thus will most likely not be considered upstream. We will maintain this project in Debian/Ubuntu for now.

Debian packaging

Based on the information in setup.py and the source code, we can also generate a working Debian packaging.

The van.pydeb project (packages, svn) [already provides a some mechanics, so this should be used by distutils-extra if appropriate.

python-distutils-extra will provide a new command python-mkdebian which generates a debian/ directory with the necessary files from the information contained in the Python .egg-info:

changelog

  • project name, version, author, email from setup.py
  • description: "new release"
  • distro target: needs argument from environment; default to lsb_release -c?

  • If already exists, will just add a new changelog entry for a new version.

compat

  • constant (6 for hardy support)
  • Do not change if already existing.

rules

  • static, cdbs+python-distutils.mk
  • Do not change if already existing.

  • Grep source for "([cC])" statements and add them.
  • Support some standard values of "license" and add GPL stub and author name.
  • Do not change if already existing.

control

  • no dh_install, upstream build system DTRT
  • Source, Package, Maintainer, Description: from setup.py
  • Section, Priority, Standards-Version, XS-Python-Version: constant
  • Build-Depends: static (no tests), or binary depends (if test cases available)
  • Depends: grep all *.py for import statements, check which package provides them
    • ./setup.py can also specify them explicitly with the requires keyword

    • Debian specific code needs to find out which package ships it; this is a heuristic, but should get it right in 80% of the cases at least
    • The Debian Python Modules Team has a couple of scripts to do this

  • If already existing, update build/binary dependencies.

Test/Demo Plan

python-distutils-extra contains an automatic test suite which covers all functionality of the DistUtilsExtra.auto automatic build system. It is run on package build.

For a manual test, check out lp:jockey (upstream trunk), and confirm that it does not have any of setup.cfg, MANIFEST.in, POTFILES.in, and only a very minimal setup.py. Test the following:

  • ./setup.py install --root=/tmp/x does a complete and working installation

  • ./setup.py sdist produces a complete source tarball

  • python-mkdebian produces a working debian/ tree; calling debuild produces a working .deb, no lintian errors, and only trivial lintian warnings (just "binary without manpage" at the time of this writing)


CategorySpec

DesktopTeam/Specs/Karmic/AutomagicPythonBuildSystem (last edited 2009-07-29 08:00:04 by pD9EB3788)