AutomagicBinaryPackaging

Revision 5 as of 2011-08-19 12:19:34

Clear message

Summary

A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.

Release Note

https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Uploaded tarballs are now available for testing within minutes.

Rationale

Ubuntu has a SoftwareCenter where people can buy applications. We want to make it very, very easy for application developers to publish their apps in the software center. For many such folk, this means publishing binary applications.

User stories

  • As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.
  • As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Constraints and Requirements

Must

  • Take a binary tarball and turn it into a working source package
  • Upload the finished source package to a PPA

Nice to have

  • Figure out version from the containing directory in the uploaded tarball, if necessary

Must not

Out of scope

  • Anything other than proprietary, binary tarballs

Assumptions

  • Given a binary tarball and information from the developer portal
  • Binary is built using C or C++
  • The layout of the tarball is the same as the intended layout when running on a user's system

  • The tarball contents can be copied straight into /opt/<package_name>/ and be run from there by ordinary users

Design

The core process:

  • Take a tarball plus the metadata provided in the webform
  • Create a debian/ directory and then a source package
  • Upload the source package to a PPA on Launchpad
  • Our goal is something that can be integrated into the existing myapps portlet.
  • The essence of the 'automagic' is guessing, this means
    • users must be given the opportunity to review the guesses
    • users must be able to correct the guesses
    • the guesswork has to be done within the time of a user's attention span
    • some things will always not work
      • therefore, we need to figure out how best to handle those failures

Question: How much is the pkgme command-line a deliverable for this? Are we right to just concentrate on the web app

pkgme backend

Proof-of-concept exists at lp:~jml/+junk/pkgme-binary.

'pkgme' is constrained to be as simple as possible. It is meant to be a tool that can be used mindlessly and ignorantly to package upstream code. It is a command-line tool, and we expect the users to be open source developers who are currently using Ubuntu.

Symbol dependency database

Guessing dependencies relies on a symbol dependency database. Currently, we are cribbing one from Debian, but that's actually a technically incorrect thing to do.

Posit: we should build & maintain our own symbol dependency database as a separate web service.

Launchpad / software center integration

  • Once the package is built, we need to upload it to a private PPA
    • Need "commercial admin" privileges to create a Launchpad PPA now
      • Julian wants to change this, but has no plan

  • XXX What needs to be done to get that PPA visible in software center

  • XXX Launchpad has no facilities for telling us when something is done, save email. Does this matter?

Integration with myapps

Implementation

UI Changes

IMPORTANT: No current plan for this. This is the biggest risk, IMO, especially if we rely on the Canonical Web team to provide the UI. – jml

We will need to change the myapps UI to allow for work to be done post-upload processing.

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Known issues with binary backend

  • Relies on external database to do symbol resolution
  • Does not fill in changelog properly or usefully
  • Does not take description from metadata and put it in control file
  • No support for copyright handling
  • Incorrectly specifies architecture as any, even though only builds for the architecture of the binary
  • Could actually build for both i386 and amd64 architectures, but only builds for the architecture of the binary
  • Does not support icons

  • Creates a desktop file & installs a desktop file, but for some reason installed app does not appear in desktop

  • Dependency guessing should take symbols and versions into account, instead only relies on .so file names
  • Duplicates code from dpkg-shlibdeps

Potential issues with Launchpad integration

Migration

Include:

  • data migration, if any
  • redirects from old URLs to new ones, if any
  • how users will be pointed to the new way of doing things, if necessary.

Test/Demo Plan

Test data

Expected good

  • Minimal GTK+ application
  • Minimal OpenGL application
  • Existing proprietary apps

Expected bad

  • A Python tarball
  • Tarballs without binaries
  • Tarballs with debian/ directory that already exists

Scenarios

  • Everything is golden and wonderful
  • User needs to change some of the packaging guesses
  • Package fails to build on Launchpad
  • Automagic packaging system temporarily down

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

CategorySpec