AutomagicBinaryPackaging

Summary

A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.

Implementation Plan

Contents

  1. Summary
  2. Release Note
  3. Rationale
    1. Adding a commercial application to the software center
  4. User stories
    1. Initial upload
    2. Initial upload, packaging included
    3. Provide upgrade
    4. Provide upgrade, packaging included
    5. Review application
  5. Constraints and Requirements
    1. Must
    2. Nice to have
    3. Must not
    4. Out of scope
    5. Assumptions
  6. Design
    1. 1. Web interface (changes to myapps)
    2. 2. Server-side pkgme
      1. Launchpad / software center integration
      2. Potential issues with Launchpad integration
    3. 3. Binary packaging system
    4. 4. Symbol dependency database
  7. Implementation
    1. General notes
    2. 1. Web interface (Changes to myapps)
    3. 2. Server-side pkgme
      1. PPA upload notes
    4. 3. Binary packaging
    5. 4. Symbol dependency database
      1. Possible schema
  8. Security
  9. Test/Demo Plan
    1. Test data
      1. Expected good
      2. Expected bad
      3. Scenarios
  10. Future work
  11. Reading

Release Note

https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Packages built from uploaded tarballs are now available for testing within minutes.

Rationale

Ubuntu has a SoftwareCenter where people can buy applications. We want to make it very, very easy for application developers to publish their apps in the software center. For many such folk, this means publishing binary applications.

The work of turning a tarball of binaries and supporting files in to a package is usually straightforward, and so we aim to automate it, so that experienced packagers can spend their time on more interesting tasks.

Adding a commercial application to the software center

Here are the steps that Canonical goes through in order to add a commercial application to the software center once an application is in the review queue:

  • Review entry for errors
  • Download package
  • Review package - Request more information if the package is a binary deb - ask for package source
  • Package up application
  • Once application is fully packaged and tested locally you can upload the app into the ppa:commercial-ppa-uploaders/test ppa for testing/review
  • For publishing
    • Create PPA in the ~commercial-ppa-uploaders team

    • Ask a LOSA to priv+comm the newly created PPA BEFORE UPLOADING

    • Copy over from test ppa or upload separately

  • In MyApps

    • Start review
    • Add the package name
    • Select which distroseries and arch the app will support
    • Add the ppa detail
    • Add the key for the ppa (this is always the same)
    • Submit to QA for review

We want to do less work here and have less opportunity for error.

User stories

Initial upload

As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Option #1: No review of guessing for submitter

NOTE: This option provides minimal change in user experience from the current developer upload story.

  1. Go to https://myapps.developer.ubuntu.com

  2. Click “Submit a new application”

  3. Specify:
    1. The name of the application

    2. The name of the package

    3. The version of the application (optional)

    4. A tagline for the application
    5. Its price
    6. A tarball for uploading
  4. Click next then specify:

    1. The “Department” the application belongs to

    2. A description for the application
    3. Some keywords
  5. Click next then specify:
    1. A screenshot
    2. Icons of various sizes
  6. Click next then specify:
    1. A support URL
    2. A license
  7. Click next then specify your payment details
  8. Submit your application

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

Option #2: Review of guesses for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Click “Submit a new application”

  3. Specify:
    1. The name of the application

    2. The name of the package

    3. The version of the application (optional)

    4. A tagline for the application
    5. Its price
    6. A tarball for uploading
  4. Click next then specify:

    1. The “Department” the application belongs to

    2. A description for the application
    3. Some keywords
  5. Click next then specify:
    1. A screenshot
    2. Icons of various sizes
  6. Click next then specify:
    1. A support URL
    2. A license
  7. Click next then specify your payment details
  8. Submit your application

  9. Wait while myapps packages the application

  10. You will be presented with the “guesses” made by the automagic packaging, and given an opportunity to change them:

    • path to the main executable
    • list of Debian package dependencies
    • (possibly) the version of the package
    • (possibly) the distroseries to target
    • (possibly) the architecture to run on
  11. After making any changes, confirm your application submission

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

NOTE: Maybe the version of the package will be manually specified by the software developer. See https://bugs.launchpad.net/developer-portal/+bug/827490.

At some time after the application is submitted, the developer will see a link to a PPA with a built version of the package.

Initial upload, packaging included

As above. No guesses displayed regardless of UI option taken.

Provide upgrade

As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Option #1: No review of guessing for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Select your application

  3. In the “Your app” section, select “Edit”

  4. Upload a new version of your tarball
  5. (possibly) Specify the version number for this new upload

  6. Click “Save changes”

Then myapps will make the generated source package available to the reviewer, but not upload it to the PPA.

Option #2: Review of guesses for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Select your application

  3. In the “Your app” section, select “Edit”

  4. Upload a new version of your tarball
  5. (possibly) Specify the version number for this new upload

  6. Click “Save changes”

  7. Wait while myapps processes the new tarball

  8. If myapps thinks the dependencies, architecture or main executable have changed, then you will be presented with the new guesses for approval
  9. After making any changes, confirm your application submission

Then myapps will make the generated source package available to the reviewer, but not upload it to the PPA.

Provide upgrade, packaging included

As above. No guesses displayed regardless of UI option taken.

If previous version was auto-packaged, completely blow away auto-packaging.

Review application

Once an application has been submitted by a developer, it is then placed in the review queue.

For an initial upload, there will be the uploaded tarball and a link to the new PPA.

For an upgrade, there will both the uploaded tarball and the built source package.

Option: We could instead upload the source package to a staging PPA - the benefit being that the reviewer can test the install without building the package.

  • TODO: decide whether we want staging PPAs for updates, or rely on a human to build, test, and then push to the existing PPA.

XXX: Should we have some sort of visual indicator within myapps for the state that the PPA upload is in? (i.e. uploaded, building, successfully built, failed to build, published)

Constraints and Requirements

Must

  • Take a binary tarball and turn it into a working source package. It is acceptable to have a list of rules that the binary tarball must obey in order to work with the system, provided that the rules are understandable to third-party developers.
  • Upload the finished source package to a PPA
  • Support upgrades of packages

  • Integrate into https://myapps.developer.ubuntu.com

  • Support amd64 and i386 binaries
  • Preserve ability for developers to upload tarballs with their own packaging

Nice to have

  • Figure out version from the containing directory in the uploaded tarball, if necessary

Must not

Out of scope

  • Anything other than commercial binary tarballs

    • Although we really love both source code and non-commercial software, we aren't doing it as part of this initiative. All of the underlying software & infrastructure is capable of handling it though.

  • Any architectures other than i386 or amd64

Assumptions

  • Given a binary tarball and information from the developer portal
  • Binary is built using C or C++
  • The layout of the tarball is the same as the intended layout when running on a user's system

  • The tarball contents can be copied straight into /opt/<package_name>/ and be run from there by ordinary users

Design

Core components:

  1. Web interface (Changes to myapps)

  2. Server-side pkgme

  3. Binary packaging system

  4. Dependency database

1. Web interface (changes to myapps)

Changes to the myapps web interface need to be governed by the following:

  1. The essence of 'automagic' is guessing, which means that it will sometimes guess wrongly or not be able to guess at all.
  2. Making the guesses, generating the package and uploading to Launchpad are all slow.

There are two main options for changes to the myapps UI:

  1. Trust the guessing. Leave everything the same, except for adding new documentation & necessary fields.

  2. Don't trust the guessing. Give developers an opportunity to review the guesses made by the system.

TODO: make a decision on this based on the information we get from the initial versions of the system. If we find that pkgme guesses are trustworthy then we will trust them to reduce UX complexity.

Note that myapps is the only closed-source system in this spec.

2. Server-side pkgme

pkgme is constrained to be as simple as possible. It is meant to be a tool that can be used mindlessly and ignorantly to package upstream code. It is a command-line tool, and we expect the users to be open source developers who are currently using Ubuntu.

Launchpad / software center integration

  • Once the package is built, we need to upload it to a private PPA
    • Need "commercial admin" privileges to create a Launchpad PPA now
      • Julian wants to change this, but has no plan

  • XXX What needs to be done to get that PPA visible in software center? (as soon as the developer application is approved and published, whatever ppa identifier is recorded on the application will be published to USC)

  • XXX Launchpad has no facilities for telling us when something is done, save email & us polling it. Does this matter?

Potential issues with Launchpad integration

3. Binary packaging system

Needs to take a tarball + metadata and generate a Debian source package.

See Assumptions above.

Will need to guess:

  • path to executable
  • dependencies
  • architecture
  • distroseries
    • maybe we can actually figure this out based on dependencies / symbols

We could guess the version, based on tarball file name and/or the directory name within the tarball. Whether we need to guess it depends on the UI. Assuming for now that the uploader would prefer to specify the version.

Note: pkgme supports multiple backends, this is only one.

4. Symbol dependency database

Guessing dependencies relies on a symbol dependency database. Thus, we must build & maintain our own symbol dependency database.

Such a service is useful outside of the automagic packaging context and should become available as a public webservice.

Implementation

pkgme-servicediagram.svg

General notes

  • We want detailed monitoring of all levels of this service

1. Web interface (Changes to myapps)

  • Need to make a decision about whether to show guesses & provide opportunity to correct them.

  • Need to call out to pkgme-service and provide a callback for handling its return values. See below.

IMPORTANT: No matter what, we need input from the Canonical Web team as to how the user interface should change as a result of this new functionality.

Option: If the tarball has Debian packaging, then don't forward it to server-side pkgme. This means that such tarballs will not be published to PPAs automatically.

2. Server-side pkgme

The service will need to publish functions like:

  • guess_packaging_details :: devportal-metadata -> callback_url -> guesses

  • build_package :: devportal-metadata -> guess_overrides -> source package

Note: Since sometimes the tarball will come with its own packaging, the server should be able to detect when a tarball has packaging and just trust the packaging.

The devportal-metadata will look a bit like this: 

{
"name": "Angry Birds",
"package_name": "angrybirds",
"description": "A very popular game that lets you throw birds at pigs. I think.",
"tagline": "The best pig/bird combination since bacon and eggs",
"price": "5.00",
"tarball": "http://internal.url/to/tarball",
}

The guesses will look a bit like this: 

{
"executable": "bin/angrybirds",
"architecture": "i386",
"dependencies": "libc6",
}

The source package will be sent as a URL.

Both of these "functions" will need to be asynchronous, since both guessing and building the packages take a while.

Since the tarball is likely to be a commercial application for sale, both it and the built source package must be available only to components within this system, and not to the general public.

  • Have a separate web service that publishes some sort of functions like the ones described above

    • Tarball & other blobs are sent as URLs

  • Include a callback URL in the metadata when requesting guesses or upload
  • Have the web service do the guessing and packaging, and call the URL when done

Note that we do not want to guarantee high availability for this new webservice. Thus, MyApps should only perform very quick requests to the service, and store up any requests that fail or timeout for later processing.

Since the functions carry no state, we could have arbitrarily many pkgme-services handling incoming requests from MyApps. Storage of applications and their built debian/ directories on disk could cause scaling issues.

PPA upload notes

  • All PPAs currently owned by a specific team: ~commercial-ppa-owners or something
  • Each application in myapps knows its archive ID and signing key

  • XXX: Which component has the responsibility of uploading to PPA?

  • XXX: Who signs the upload to PPA?

  • XXX: Who is an appropriate changelog entry author for this service?

  • XXX: What is the name of the PPA?

  • XXX: Might need separate interfaces for new upload vs upgrade

  • Uploading to PPAs will put increased demand on an already-taxed system

3. Binary packaging

Proof-of-concept exists at lp:pkgme-binary.

Known issues: https://bugs.launchpad.net/pkgme-binary

4. Symbol dependency database

  • Only need to store information for distributions that have the Software Center
  • To be truly up-to-date, needs to import new symbols whenever new packages with libraries are published to the Ubuntu archive
  • Probably need to perform queries like dpkg-shlibdeps does: here are my needed symbols, return a list of packages and their versions.
  • Will need data from both symbols and shlibs files, as not all packages have symbol files

Implement using lp:udd to build and maintain the database. Initial cut of the database to use sqlite, but move to something more serious as load demands.

Initially, co-host with pkgme-service. Later, turn into separate, public service.

Possible schema

This schema is to demonstrate the structure of the data, and give a hint as to how it might be implemented in Launchpad or in the database of an independent system. It doesn't take query efficiency into account at all. 

CREATE TABLE libdep (
  source_package_name TEXT, 
  library TEXT, 
  dependency TEXT, 
  CONSTRAINT libdep_uniq UNIQUE (source_package_name, library, dependency)
);

Security

  • PPA and builders are expensive and will be used even more. That can cause a lot of problems down the road
    • One way of limiting is to do one app at once, but that limits number which can be done in a day.
  • In the end you will be able to upload your code and get a private PPA with your code in it -- which currently is a paid service

Test/Demo Plan

Overall strategy is to get something live and deployed and operating on incoming user data as quickly as possible. Once this is up, we'll use that data to decide where to focus efforts.

More concretely, we could run every binary app that's currently in the Software Center through the system and see what happens.

Test data

Expected good

  • Minimal GTK+ application
  • Minimal OpenGL application
  • Existing proprietary apps
  • Expected dependencies are available for all distro series

Expected bad

  • A Python tarball
  • Tarballs without binaries
  • Tarballs with debian/ directory that already exists

Scenarios

  • Everything is golden and wonderful
  • User needs to change some of the packaging guesses
  • Package fails to build on Launchpad
  • Automagic packaging system temporarily down

Future work

  • Extend the system to support other pkgme backends. This will probably require using the Launchpad build farm, as most build systems require execution of arbitrary code.
  • Support other architectures, ARM in particular. This will probably require more resources from the Launchpad build farm

Reading

CategorySpec

AutomagicBinaryPackaging (last edited 2012-03-08 11:44:11 by cpc11-acto4-2-0-cust359)