AutomagicBinaryPackaging

Differences between revisions 4 and 7 (spanning 3 versions)
Revision 4 as of 2011-07-18 09:55:10
Size: 2125
Editor: jml
Comment:
Revision 7 as of 2011-08-19 14:11:08
Size: 7511
Editor: jml
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
Given binary tarballs from upstream software vendors, turn those into packages in a PPA as quickly and smoothly as possible.  * '''Launchpad Entry''':
 * '''Created''': [[<<Date(2011-08-19T12:25:27Z)>>]]
 * '''Contributors''': JonathanLange
 * '''Packages affected''':
Line 3: Line 6:
'''Contact:''' JonathanLange <<BR>>
'''On Launchpad:''' [[https://bugs.launchpad.net/pkgme/+bugs?field.tag=binary|binary]]
== Summary ==

A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.

== Release Note ==

https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Uploaded tarballs are now available for testing within minutes.
Line 10: Line 18:
== Stakeholders == == User stories ==
Line 12: Line 20:
|| Name || Interest || Last consulted ||
|| Rick Spencer || Ubuntu Engineering manager || 2011-07-11 ||
|| Steve George || Business development || 2011-07-04 ||
|| ??? || || ||

## == User stories ==

## These are a collection of things that users want to do, and why they want to do them. Try very, very hard to phrase them in terms of user needs rather than implementation details.

## === $STORY_NAME ===

## '''As a ''' $PERSON<<BR>>
## '''I want ''' $FEATURE<<BR>>
## '''so that ''' $BENEFIT<<BR>>

## Have as many as you like. Group user stories together into meaningfully deliverable units. They can then be used as the driving elements of exploratory testing.
 * As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.
 * As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.
Line 35: Line 29:
 * Definitely need to support upgrades
Line 46: Line 41:
 * Anything other than proprietary, binary tarballs
Line 51: Line 47:
  == Measures of success ==  * The layout of the tarball is the same as the intended layout when running on a user's system
 * The tarball contents can be copied straight into /opt/<package_name>/ and be run from there by ordinary users
Line 54: Line 50:
Ability to re-package existing binary applications in the SoftwareCenter. Testing against:
 * The Clockwork Man 2
 * Uplink
 * World of Goo
 * bukkon
== Design ==
Line 60: Line 52:
== Thoughts? == The core process:
 * Take a tarball plus the metadata provided in the webform
 * Create a debian/ directory and then a source package
 * Upload the source package to a PPA on Launchpad
Line 62: Line 57:
## Put everything else here. Better out than in. Notes:
 * Our goal is something that can be integrated into the existing myapps portlet.
 * The essence of the 'automagic' is guessing, this means
   * users must be given the opportunity to review the guesses
   * users must be able to correct the guesses
   * the guesswork has to be done within the time of a user's attention span
   * some things will always not work
     * therefore, we need to figure out how best to handle those failures
Line 64: Line 66:
There are three separate phases to this:
 1. Create a valid source package
 1. Upload that to Launchpad
 1. Integrate with the existing developer portal
'''Question:''' How much is the pkgme command-line a deliverable for this? Are we right to just concentrate on the web app

=== pkgme backend ===

Proof-of-concept exists at lp:~jml/+junk/pkgme-binary.

'pkgme' is constrained to be as simple as possible. It is meant to be a tool
that can be used mindlessly and ignorantly to package upstream code. It is a
command-line tool, and we expect the users to be open source developers who
are currently using Ubuntu.

Guesses:
 * executable path
 * dependencies
 * architecture
 * distroseries
   * maybe we can actually figure this out based on dependencies / symbols
 * version?


==== Symbol dependency database ====

Guessing dependencies relies on a symbol dependency database. Currently, we are cribbing one from Debian, but that's actually a technically incorrect thing to do.

Posit: we should build & maintain our own symbol dependency database as a separate web service.

=== Launchpad / software center integration ===

 * Once the package is built, we need to upload it to a private PPA
   * Need "commercial admin" privileges to create a Launchpad PPA now
     * Julian wants to change this, but has no plan
 * '''XXX''' What needs to be done to get that PPA visible in software center
 * '''XXX''' Launchpad has no facilities for telling us when something is done, save email. Does this matter?

=== Integration with myapps ===

== Implementation ==

 * All PPAs currently owned by a specific team: ~commercial-ppa-owners or something
 * Each application knows its archive ID and signing key

=== Questions ===

 * How much metadata to send to packaging?
 * How much integration with myapps?
   * i.e. do we actually go back and get the user to confirm values? (sg can answer)
 * How many things does the app actually guess?
 * Need to sign the package for upload -- which signature
 * How to get the tarball from app server to packaging server
 * What authentication & authorization to use for packaging?
   * packaging service might need access to tarball via web
   * either way, would need hook to call back to myapps on


=== UI Changes ===

'''IMPORTANT:''' No current plan for this. This is the biggest risk, IMO, especially if we rely on the Canonical Web team to provide the UI. &ndash; jml

We will need to change the myapps UI to allow for work to be done post-upload processing.

=== Code Changes ===

Code changes should include an overview of what needs to change, and in some cases even the specific details.

==== Known issues with binary backend ====

 * Relies on external database to do symbol resolution
 * Does not fill in changelog properly or usefully
 * Does not take description from metadata and put it in control file
 * No support for copyright handling
 * Incorrectly specifies architecture as any, even though only builds for the architecture of the binary
 * Could actually build for both i386 and amd64 architectures, but only builds for the architecture of the binary
 * Does not support icons
 * Creates a desktop file & installs a desktop file, but for some reason installed app does not appear in desktop
 * Dependency guessing should take symbols and versions into account, instead only relies on .so file names
 * Duplicates code from dpkg-shlibdeps

==== Potential issues with Launchpad integration ====

  * https://bugs.launchpad.net/launchpadlib/+bug/814595 -- unrecoverable error when auth fails
  * https://bugs.launchpad.net/launchpad/+bug/814633 -- cannot delete ppa over api
  * https://bugs.launchpad.net/launchpad/+bug/814605 -- dodgy errors when cannot create
  * https://bugs.launchpad.net/launchpad/+bug/814725 -- no feedback between acceptance upload & rejection
  * https://bugs.launchpad.net/launchpad/+bug/637915 -- cannot tell if a PPA is deleted or not over API
  * https://bugs.launchpad.net/launchpad/+bug/568981 -- cannot upload GPG key via API


=== Migration ===

Include:
 * data migration, if any
 * redirects from old URLs to new ones, if any
 * how users will be pointed to the new way of doing things, if necessary.

== Test/Demo Plan ==

=== Test data ===

==== Expected good ====

 * Minimal GTK+ application
 * Minimal OpenGL application
 * Existing proprietary apps

==== Expected bad ====

 * A Python tarball
 * Tarballs without binaries
 * Tarballs with debian/ directory that already exists

==== Scenarios ====

 * Everything is golden and wonderful
 * User needs to change some of the packaging guesses
 * Package fails to build on Launchpad
 * Automagic packaging system temporarily down

== Unresolved issues ==

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.


----
CategorySpec

Summary

A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.

Release Note

https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Uploaded tarballs are now available for testing within minutes.

Rationale

Ubuntu has a SoftwareCenter where people can buy applications. We want to make it very, very easy for application developers to publish their apps in the software center. For many such folk, this means publishing binary applications.

User stories

  • As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.
  • As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Constraints and Requirements

Must

  • Take a binary tarball and turn it into a working source package
  • Upload the finished source package to a PPA
  • Definitely need to support upgrades

Nice to have

  • Figure out version from the containing directory in the uploaded tarball, if necessary

Must not

Out of scope

  • Anything other than proprietary, binary tarballs

Assumptions

  • Given a binary tarball and information from the developer portal
  • Binary is built using C or C++
  • The layout of the tarball is the same as the intended layout when running on a user's system
  • The tarball contents can be copied straight into /opt/<package_name>/ and be run from there by ordinary users

Design

The core process:

  • Take a tarball plus the metadata provided in the webform
  • Create a debian/ directory and then a source package
  • Upload the source package to a PPA on Launchpad

Notes:

  • Our goal is something that can be integrated into the existing myapps portlet.
  • The essence of the 'automagic' is guessing, this means
    • users must be given the opportunity to review the guesses
    • users must be able to correct the guesses
    • the guesswork has to be done within the time of a user's attention span
    • some things will always not work
      • therefore, we need to figure out how best to handle those failures

Question: How much is the pkgme command-line a deliverable for this? Are we right to just concentrate on the web app

pkgme backend

Proof-of-concept exists at lp:~jml/+junk/pkgme-binary.

'pkgme' is constrained to be as simple as possible. It is meant to be a tool that can be used mindlessly and ignorantly to package upstream code. It is a command-line tool, and we expect the users to be open source developers who are currently using Ubuntu.

Guesses:

  • executable path
  • dependencies
  • architecture
  • distroseries
    • maybe we can actually figure this out based on dependencies / symbols
  • version?

Symbol dependency database

Guessing dependencies relies on a symbol dependency database. Currently, we are cribbing one from Debian, but that's actually a technically incorrect thing to do.

Posit: we should build & maintain our own symbol dependency database as a separate web service.

Launchpad / software center integration

  • Once the package is built, we need to upload it to a private PPA
    • Need "commercial admin" privileges to create a Launchpad PPA now
      • Julian wants to change this, but has no plan
  • XXX What needs to be done to get that PPA visible in software center

  • XXX Launchpad has no facilities for telling us when something is done, save email. Does this matter?

Integration with myapps

Implementation

  • All PPAs currently owned by a specific team: ~commercial-ppa-owners or something
  • Each application knows its archive ID and signing key

Questions

  • How much metadata to send to packaging?
  • How much integration with myapps?
    • i.e. do we actually go back and get the user to confirm values? (sg can answer)
  • How many things does the app actually guess?
  • Need to sign the package for upload -- which signature
  • How to get the tarball from app server to packaging server
  • What authentication & authorization to use for packaging?

    • packaging service might need access to tarball via web
    • either way, would need hook to call back to myapps on

UI Changes

IMPORTANT: No current plan for this. This is the biggest risk, IMO, especially if we rely on the Canonical Web team to provide the UI. – jml

We will need to change the myapps UI to allow for work to be done post-upload processing.

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Known issues with binary backend

  • Relies on external database to do symbol resolution
  • Does not fill in changelog properly or usefully
  • Does not take description from metadata and put it in control file
  • No support for copyright handling
  • Incorrectly specifies architecture as any, even though only builds for the architecture of the binary
  • Could actually build for both i386 and amd64 architectures, but only builds for the architecture of the binary
  • Does not support icons
  • Creates a desktop file & installs a desktop file, but for some reason installed app does not appear in desktop

  • Dependency guessing should take symbols and versions into account, instead only relies on .so file names
  • Duplicates code from dpkg-shlibdeps

Potential issues with Launchpad integration

Migration

Include:

  • data migration, if any
  • redirects from old URLs to new ones, if any
  • how users will be pointed to the new way of doing things, if necessary.

Test/Demo Plan

Test data

Expected good

  • Minimal GTK+ application
  • Minimal OpenGL application
  • Existing proprietary apps

Expected bad

  • A Python tarball
  • Tarballs without binaries
  • Tarballs with debian/ directory that already exists

Scenarios

  • Everything is golden and wonderful
  • User needs to change some of the packaging guesses
  • Package fails to build on Launchpad
  • Automagic packaging system temporarily down

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.


CategorySpec

AutomagicBinaryPackaging (last edited 2012-03-08 11:44:11 by jml)