AutogeneratePackging

Summary

Rationale

Many developers do not wish to learn to package, or claim that it is too complex. Often these same people will be working on projects which do not require complex packaging, or the implied familiarity with the system, tools and processes. Providing these developers with tools which can autogenerate packaging for them would remove one of the barriers to distributing their code on Ubuntu, while still having the package managed by dpkg, and without removing the ability to modify the packaging if the complexity of the project grows.

Several tools, such as debhelper, quickly, python-stdeb, python-mkdebian and dh-make-perl have shown the utility of an approach which knows how to deal with specific classes of package. Following this style will provide an extensible system that can cope with the multitude of technologies developers can use.

User stories

  • mpt has written an app for elegant mockups, and wants to distribute it for Ubuntu. He is able to generate packaging that will create a usable package, and can upload this to a PPA to distribute.

Assumptions

  • We do not have to be able to automatically recreate the packaging of every package in Ubuntu.
  • This will typically be used by the developers themselves, so they can often fix the project so that the standard methods work, rather than dealing it with packaging.
  • Perhaps only consider single binary packages to start with.

Design

"quickly package" points to an approach that will serve us well.

It makes use of python-mkdebian, with certain options, to create the packaging by introspecting the projects.

However, it is tied to Python code, so we want to make it generic. We don't want to rewrite it for every language, as much of what it does is language independent.

We should then define an interface which language/project style communities can implement in order to provide the information that is not generic.

Looking at the information that can be provided:

  • debian/copyright: not particularly project-style specific (though some could provide it). Can be guessed (licensecheck), standard used (selection of license and we just trust them), or an empty file used as it is not required for a functional package.
  • debian/changelog: not project-style specific, except that the version number may want to be based on something such as setup.py. Perhaps VCS integration, or other ways of updating it.
  • debian/rules: generate a dh v7 file, and assume that there are no tweaks required. If there are tweaks that can be detected, attempt to push this in to the debhelper backend build system. Perhaps have a way to provide tweaks as overrides.
  • debian/control: a mix of data. Things such as maintainer are probably not project type specific. description may be, as some may want to take it from e.g. setup.py, though there will often be tweaks required for the format. Build dependencies are something that language/project-style backends are required for.
  • There may be need to create various debhelper input files (debian/links etc.)

The tool should also be able to work incrementally, i.e. with an existing debian directory. This means that you don't need to understand how to add a new build dependency to the packaging, or to cope with the packaging implications of other changes you make.

We also may want "3-way merge" of the packaging, so that it can handle manual changes to the packaging.

Implementation

This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

UI Changes

Should cover changes required to the UI, or specific UI that is required to implement this

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Migration

Include:

  • data migration, if any
  • redirects from old URLs to new ones, if any
  • how users will be pointed to the new way of doing things, if necessary.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

BoF agenda and discussion

Want to autogenerate as much as possible. Don't need new binary format.

Don't need to handle complexity of everything in Ubuntu.

Some existing packages are crufty for historical reasons. Are simplifying such things in scope? Sort of. Maybe by blowing away debian/, autogenerating, and seeing what breaks.

Not just fire-once, but also for handling updates and such.

Designed for authors themselves, not Ubuntu developers. Less work for us by not overriding what upstream does.

Start by focusing on packages that generate just one binary package.

Tried to extend dhmake, but was not architected like we need.

Existing stuff: dhmake, python-stddeb, quickly. Last two only work for python. Want more extensible solution.

We don't care about patch systems, this is for upstream.

Subproblems:

  • debian/copyright (not a blocker to get right)
  • debian/changelog (not a blocker, except version number is)
  • debian/rules
  • debian/control
  • Dependencies

What about making sure copyright file is valid? Autogenerating it by looking at code?

  • Don't want to block or make this difficult for developers. It's not a hard requirement for deb packages. Just a requirement for Debian/Ubuntu.

Packaging binary artifacts like .jar files... Not easy to package mass amounts of java with weird build systems. Easier to stuff jar files.

debhelper is pointing in a good direction with figuring out what you are trying to do.

Changelog. Version number is only required bit.

  • Version numbers are not always well understood (how to do it right). Maybe a bit of education about how best to use version numbers.

Licensing is also poorly understood frequently by developers. Not odd for no copyright headers in source.

This tool is a layer below the IDE or quickly. Education to user is probably more appropriate in that IDE layer.

Rules:

  • Just make a dh7 tiny rules.
  • Except probably specify --build-system=XXX so we don't have two systems trying to detect that.
  • As gravy, add in commented out common overrides with some helpful comments
    • Or link to a wiki page or a text file that helps with this, rather than leaving this text in place as it bit-rots

Control:

  • Maintainer mailing address from environment, same way changelog gets it
  • Not important to 100% get email right, they can correct
  • Description: easy for user to fill out
    • setup.py's description not always formatted right
    • We can autogenerate any old summary/description. Not a blocker for this

What about dependencies?

  • Completely handled by language backend
  • substvars? Can't, because you need build-deps installed before we start building

Backend should probably have a hook in everything, just in case, but mostly expect them only for a few things like dependencies.

Misc. files like .install, .links, etc. dh7 introspects most of that for us today.

  • What about mimetyping? ".mime" Can't really autogenerate.

Should be able to be incremental. So that if you run again after adding a dependency in source, this would notice.

  • What about customizations by developer?
    • Try to incorporate them? Don't go crazy about it, just try to use standard merging.
    • In some cases you can try to be careful not to break it by removing dependencies, only adding.
    • Needs to customize are arguably a bug in this tool.
    • Still, we need to at least allow changes so we don't block user while we fix it.
    • But no need to allow them to run tool again (i.e. we can overwrite data for now, no need to focus on merging)

What language/interface should we have backends in?

  • Whatever language. Just call a function with a defined interface, spits data out on sdtout in a standard format.
  • Can just work on current directory
  • Have to be careful about defining arguments to each script
  • Common errors (like bad email/name or something) need well-defined return codes as part of this interface.
  • How to ask user that they need to fill in setup.py so the backend can do the right thing?
    • GUI could pre-prompt for data and pass it to backend
    • backend could tell GUI that it needs certain well-defined fields
    • Can ask backend, 'try to guess name' and then only prompt name if that fails

What do we need from a backend?

  • Detection of whether this backend fits the source
  • Build dependencies

Maybe /usr/share/XXX/BACKEND/hooks/deps or something

Decisions:

  • Don't worry about allowing modifications by user (i.e. we can overwrite any files in debian/)
  • Name? pkgme

Action Items:

  • [james_w] python reference implementation
  • [mterry] flash
  • [mterry] vala
  • [JamesPage] java

  • [clint-fewbar] ruby


CategorySpec

Specs/N/AutogeneratePackging (last edited 2010-10-27 22:03:22 by host194)