Summary

A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.

Implementation Plan

Contents

  1. Summary
  2. Release Note
  3. Rationale
    1. Adding a commercial application to the software center
  4. User stories
    1. Initial upload
    2. Initial upload, packaging included
    3. Provide upgrade
    4. Provide upgrade, packaging included
    5. Review application
  5. Constraints and Requirements
    1. Must
    2. Nice to have
    3. Must not
    4. Out of scope
    5. Assumptions
  6. Design
    1. 1. Web interface (changes to myapps)
    2. 2. Server-side pkgme
      1. Launchpad / software center integration
      2. Potential issues with Launchpad integration
    3. 3. Binary packaging system
    4. 4. Symbol dependency database
  7. Implementation
    1. General notes
    2. 1. Web interface (Changes to myapps)
    3. 2. Server-side pkgme
      1. PPA upload notes
    4. 3. Binary packaging
    5. 4. Symbol dependency database
      1. Possible schema
  8. Security
  9. Test/Demo Plan
    1. Test data
      1. Expected good
      2. Expected bad
      3. Scenarios
  10. Future work
  11. Reading

Release Note

https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Packages built from uploaded tarballs are now available for testing within minutes.

Rationale

Ubuntu has a SoftwareCenter where people can buy applications. We want to make it very, very easy for application developers to publish their apps in the software center. For many such folk, this means publishing binary applications.

The work of turning a tarball of binaries and supporting files in to a package is usually straightforward, and so we aim to automate it, so that experienced packagers can spend their time on more interesting tasks.

Adding a commercial application to the software center

Here are the steps that Canonical goes through in order to add a commercial application to the software center once an application is in the review queue:

We want to do less work here and have less opportunity for error.

User stories

Initial upload

As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Option #1: No review of guessing for submitter

NOTE: This option provides minimal change in user experience from the current developer upload story.

  1. Go to https://myapps.developer.ubuntu.com

  2. Click “Submit a new application”

  3. Specify:
    1. The name of the application

    2. The name of the package

    3. The version of the application (optional)

    4. A tagline for the application
    5. Its price
    6. A tarball for uploading
  4. Click next then specify:

    1. The “Department” the application belongs to

    2. A description for the application
    3. Some keywords
  5. Click next then specify:
    1. A screenshot
    2. Icons of various sizes
  6. Click next then specify:
    1. A support URL
    2. A license
  7. Click next then specify your payment details
  8. Submit your application

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

Option #2: Review of guesses for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Click “Submit a new application”

  3. Specify:
    1. The name of the application

    2. The name of the package

    3. The version of the application (optional)

    4. A tagline for the application
    5. Its price
    6. A tarball for uploading
  4. Click next then specify:

    1. The “Department” the application belongs to

    2. A description for the application
    3. Some keywords
  5. Click next then specify:
    1. A screenshot
    2. Icons of various sizes
  6. Click next then specify:
    1. A support URL
    2. A license
  7. Click next then specify your payment details
  8. Submit your application

  9. Wait while myapps packages the application

  10. You will be presented with the “guesses” made by the automagic packaging, and given an opportunity to change them:

    • path to the main executable
    • list of Debian package dependencies
    • (possibly) the version of the package
    • (possibly) the distroseries to target
    • (possibly) the architecture to run on
  11. After making any changes, confirm your application submission

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

NOTE: Maybe the version of the package will be manually specified by the software developer. See https://bugs.launchpad.net/developer-portal/+bug/827490.

At some time after the application is submitted, the developer will see a link to a PPA with a built version of the package.

Initial upload, packaging included

As above. No guesses displayed regardless of UI option taken.

Provide upgrade

As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Option #1: No review of guessing for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Select your application

  3. In the “Your app” section, select “Edit”

  4. Upload a new version of your tarball
  5. (possibly) Specify the version number for this new upload

  6. Click “Save changes”

Then myapps will make the generated source package available to the reviewer, but not upload it to the PPA.

Option #2: Review of guesses for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Select your application

  3. In the “Your app” section, select “Edit”

  4. Upload a new version of your tarball
  5. (possibly) Specify the version number for this new upload

  6. Click “Save changes”

  7. Wait while myapps processes the new tarball

  8. If myapps thinks the dependencies, architecture or main executable have changed, then you will be presented with the new guesses for approval
  9. After making any changes, confirm your application submission

Then myapps will make the generated source package available to the reviewer, but not upload it to the PPA.

Provide upgrade, packaging included

As above. No guesses displayed regardless of UI option taken.

If previous version was auto-packaged, completely blow away auto-packaging.

Review application

Once an application has been submitted by a developer, it is then placed in the review queue.

For an initial upload, there will be the uploaded tarball and a link to the new PPA.

For an upgrade, there will both the uploaded tarball and the built source package.

Option: We could instead upload the source package to a staging PPA - the benefit being that the reviewer can test the install without building the package.

XXX: Should we have some sort of visual indicator within myapps for the state that the PPA upload is in? (i.e. uploaded, building, successfully built, failed to build, published)

Constraints and Requirements

Must

Nice to have

Must not

Out of scope

Assumptions

Design

Core components:

  1. Web interface (Changes to myapps)

  2. Server-side pkgme

  3. Binary packaging system

  4. Dependency database

1. Web interface (changes to myapps)

Changes to the myapps web interface need to be governed by the following:

  1. The essence of 'automagic' is guessing, which means that it will sometimes guess wrongly or not be able to guess at all.
  2. Making the guesses, generating the package and uploading to Launchpad are all slow.

There are two main options for changes to the myapps UI:

  1. Trust the guessing. Leave everything the same, except for adding new documentation & necessary fields.

  2. Don't trust the guessing. Give developers an opportunity to review the guesses made by the system.

TODO: make a decision on this based on the information we get from the initial versions of the system. If we find that pkgme guesses are trustworthy then we will trust them to reduce UX complexity.

Note that myapps is the only closed-source system in this spec.

2. Server-side pkgme

pkgme is constrained to be as simple as possible. It is meant to be a tool that can be used mindlessly and ignorantly to package upstream code. It is a command-line tool, and we expect the users to be open source developers who are currently using Ubuntu.

Launchpad / software center integration

Potential issues with Launchpad integration

3. Binary packaging system

Needs to take a tarball + metadata and generate a Debian source package.

See Assumptions above.

Will need to guess:

We could guess the version, based on tarball file name and/or the directory name within the tarball. Whether we need to guess it depends on the UI. Assuming for now that the uploader would prefer to specify the version.

Note: pkgme supports multiple backends, this is only one.

4. Symbol dependency database

Guessing dependencies relies on a symbol dependency database. Thus, we must build & maintain our own symbol dependency database.

Such a service is useful outside of the automagic packaging context and should become available as a public webservice.

Implementation

pkgme-servicediagram.svg

General notes

1. Web interface (Changes to myapps)

IMPORTANT: No matter what, we need input from the Canonical Web team as to how the user interface should change as a result of this new functionality.

Option: If the tarball has Debian packaging, then don't forward it to server-side pkgme. This means that such tarballs will not be published to PPAs automatically.

2. Server-side pkgme

The service will need to publish functions like:

Note: Since sometimes the tarball will come with its own packaging, the server should be able to detect when a tarball has packaging and just trust the packaging.

The devportal-metadata will look a bit like this: 

{
"name": "Angry Birds",
"package_name": "angrybirds",
"description": "A very popular game that lets you throw birds at pigs. I think.",
"tagline": "The best pig/bird combination since bacon and eggs",
"price": "5.00",
"tarball": "http://internal.url/to/tarball",
}

The guesses will look a bit like this: 

{
"executable": "bin/angrybirds",
"architecture": "i386",
"dependencies": "libc6",
}

The source package will be sent as a URL.

Both of these "functions" will need to be asynchronous, since both guessing and building the packages take a while.

Since the tarball is likely to be a commercial application for sale, both it and the built source package must be available only to components within this system, and not to the general public.

Note that we do not want to guarantee high availability for this new webservice. Thus, MyApps should only perform very quick requests to the service, and store up any requests that fail or timeout for later processing.

Since the functions carry no state, we could have arbitrarily many pkgme-services handling incoming requests from MyApps. Storage of applications and their built debian/ directories on disk could cause scaling issues.

PPA upload notes

3. Binary packaging

Proof-of-concept exists at lp:pkgme-binary.

Known issues: https://bugs.launchpad.net/pkgme-binary

4. Symbol dependency database

Implement using lp:udd to build and maintain the database. Initial cut of the database to use sqlite, but move to something more serious as load demands.

Initially, co-host with pkgme-service. Later, turn into separate, public service.

Possible schema

This schema is to demonstrate the structure of the data, and give a hint as to how it might be implemented in Launchpad or in the database of an independent system. It doesn't take query efficiency into account at all. 

CREATE TABLE libdep (
  source_package_name TEXT, 
  library TEXT, 
  dependency TEXT, 
  CONSTRAINT libdep_uniq UNIQUE (source_package_name, library, dependency)
);

Security

Test/Demo Plan

Overall strategy is to get something live and deployed and operating on incoming user data as quickly as possible. Once this is up, we'll use that data to decide where to focus efforts.

More concretely, we could run every binary app that's currently in the Software Center through the system and see what happens.

Test data

Expected good

Expected bad

Scenarios

Future work

Reading

CategorySpec

AutomagicBinaryPackaging (last edited 2012-03-08 11:44:11 by cpc11-acto4-2-0-cust359)