AutomagicBinaryPackaging

Differences between revisions 3 and 17 (spanning 14 versions)
Revision 3 as of 2011-07-18 09:53:54
Size: 2052
Editor: eth0
Comment:
Revision 17 as of 2011-08-23 14:15:05
Size: 15099
Editor: 78-105-1-164
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
Given binary tarballs from upstream software vendors, turn those into packages in a PPA as quickly and smoothly as possible.

'''Contact:''' JonathanLange <<BR>>
'''On Launchpad:''' [[https://bugs.launchpad.net/pkgme/+bugs?field.tag=binary|binary]]
 * '''Launchpad Entry''': https://blueprints.launchpad.net/developer-portal/+spec/automagic-binary-packaging
 * '''Created''': [[<<Date(2011-08-19T12:25:27Z)>>]]
 * '''Contributors''': JonathanLange
 * '''Packages affected''':

== Summary ==

A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.

== Release Note ==

https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Uploaded tarballs are now available for testing within minutes.
Line 10: Line 18:
== Stakeholders ==

|| Name || Interest || Last consulted ||
|| Rick Spencer || Ubuntu Engineering manager || 2011-07-11 ||
|| ??? || || ||
Line 18: Line 20:
## These are a collection of things that users want to do, and why they want to do them. Try very, very hard to phrase them in terms of user needs rather than implementation details.

=== $STORY_NAME ===

'''As a ''' $PERSON<<BR>>
'''I want ''' $FEATURE<<BR>>
'''so that ''' $BENEFIT<<BR>>

## Have as many as you like. Group user stories together into meaningfully deliverable units. They can then be used as the driving elements of exploratory testing.
=== Initial upload ===

As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

'''Option #1: No review of guessing for submitter'''

'''NOTE:''' This option provides no change in user experience from the current developer upload story.

 1. Go to https://myapps.developer.ubuntu.com
 1. Click &ldquo;Submit a new application&rdquo;
 1. Specify:
   1. The name of the application
   1. '''The name of the package'''
   1. A tagline for the application
   1. Its price
   1. A tarball for uploading
 1. Click next then specify:
   1. The &ldquo;Department&rdquo; the application belongs to
   1. A description for the application
   1. Some keywords
 1. Click next then specify:
   1. A screenshot
   1. Icons of various sizes
 1. Click next then specify:
   1. A support URL
   1. A license
 1. Click next then specify your payment details
 1. Submit your application

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.


'''Option #2: Review of guesses for submitter'''

 1. Go to https://myapps.developer.ubuntu.com
 1. Click &ldquo;Submit a new application&rdquo;
 1. Specify:
   1. The name of the application
   1. '''The name of the package'''
   1. A tagline for the application
   1. Its price
   1. A tarball for uploading
 1. Click next then specify:
   1. The &ldquo;Department&rdquo; the application belongs to
   1. A description for the application
   1. Some keywords
 1. Click next then specify:
   1. A screenshot
   1. Icons of various sizes
 1. Click next then specify:
   1. A support URL
   1. A license
 1. Click next then specify your payment details
 1. Submit your application
 1. '''Wait while myapps packages the application'''
 1. You will be presented with the &ldquo;guesses&rdquo; made by the automagic packaging, and given an opportunity to change them:
   * path to the main executable
   * list of Debian package dependencies
   * (possibly) the version of the package
   * (possibly) the distroseries to target
   * (possibly) the architecture to run on
 1. After making any changes, confirm your application submission

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

'''NOTE:''' Maybe the version of the package will be manually specified by the software developer. See https://bugs.launchpad.net/developer-portal/+bug/827490.

=== Provide upgrade ===

As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

'''Option #1: No review of guessing for submitter'''

 1. Go to https://myapps.developer.ubuntu.com
 1. Select your application
 1. In the &ldquo;Your app&rdquo; section, select &ldquo;Edit&rdquo;
 1. Upload a new version of your tarball
 1. (possibly) Specify the version number for this new upload
 1. Click &ldquo;Save changes&rdquo;

Then myapps will make the generated source package available to the reviewer, but ''not'' upload it to the PPA.

'''Option #2: Review of guesses for submitter'''

 1. Go to https://myapps.developer.ubuntu.com
 1. Select your application
 1. In the &ldquo;Your app&rdquo; section, select &ldquo;Edit&rdquo;
 1. Upload a new version of your tarball
 1. (possibly) Specify the version number for this new upload
 1. Click &ldquo;Save changes&rdquo;
 1. ''Wait while myapps processes the new tarball''
 1. If myapps thinks the dependencies, architecture or main executable have changed, then you will be presented with the new guesses for approval
 1. After making any changes, confirm your application submission

Then myapps will make the generated source package available to the reviewer, but ''not'' upload it to the PPA.


=== Review application ===

Once an application has been submitted by a developer, it is then placed in the review queue.

If the change was an initial upload, there will be the uploaded tarball and a link to the new PPA.

If the change was an upgrade, there will both the uploaded tarball and the built source package.

'''XXX:''' Should we have some sort of visual indicator within myapps for the state that the PPA upload is in? (i.e. uploaded, building, successfully built, failed to build, published)
Line 34: Line 133:
 * Support upgrades of packages
 * Integrate into https://myapps.developer.ubuntu.com
Line 45: Line 146:
 * Anything other than commercial binary tarballs
   * Although we really love both source code and non-commercial software, we aren't doing it as part of this initiative. All of the underlying software & infrastructure is capable of handling it though.
Line 50: Line 153:
  == Measures of success ==

Ability to re-package existing binary applications in the SoftwareCenter. Testing against:
 * The Clockwork Man 2
 * Uplink
 * World of Goo
 * bukkon

== Thoughts? ==

## Put everything else here. Better out than in.

There are three separate phases to this:
 1. Create a valid source package
 1. Upload that to Launchpad
 1. Integrate with the existing developer portal
 * The layout of the tarball is the same as the intended layout when running on a user's system
 * The tarball contents can be copied straight into /opt/<package_name>/ and be run from there by ordinary users

== Design ==

Core components:
 1. Web interface (Changes to myapps)
 1. Server-side pkgme
 1. Binary packaging system
 1. Dependency database

=== 1. Web interface (changes to myapps) ===

Changes to the myapps web interface need to be governed by the following:
 1. The essence of 'automagic' is guessing, which means that it will sometimes guess wrongly or not be able to guess at all.
 1. Making the guesses, generating the package and uploading to Launchpad are all slow.

There are two main options for changes to the myapps UI:
 1. Trust the guessing. Leave everything the same, except for adding new documentation & necessary fields.
 1. Don't trust the guessing. Give developers an opportunity to review the guesses made by the system.


=== 2. Server-side pkgme ===

[[https://pkgme.net/|pkgme]] is constrained to be as simple as possible. It is meant to be a tool that can be used mindlessly and ignorantly to package upstream code. It is a command-line tool, and we expect the users to be open source developers who are currently using Ubuntu.

==== Launchpad / software center integration ====

 * Once the package is built, we need to upload it to a private PPA
   * Need "commercial admin" privileges to create a Launchpad PPA now
     * Julian wants to change this, but has no plan
 * '''XXX''' What needs to be done to get that PPA visible in software center?
 * '''XXX''' Launchpad has no facilities for telling us when something is done, save email. Does this matter?

==== Potential issues with Launchpad integration ====

 * https://bugs.launchpad.net/launchpadlib/+bug/814595 -- unrecoverable error when auth fails
 * https://bugs.launchpad.net/launchpad/+bug/814633 -- cannot delete ppa over api
 * https://bugs.launchpad.net/launchpad/+bug/814605 -- dodgy errors when cannot create
 * https://bugs.launchpad.net/launchpad/+bug/814725 -- no feedback between acceptance upload & rejection
 * https://bugs.launchpad.net/launchpad/+bug/637915 -- cannot tell if a PPA is deleted or not over API
 * https://bugs.launchpad.net/launchpad/+bug/568981 -- cannot upload GPG key via API


=== 3. Binary packaging system ===

Needs to take a tarball + metadata and generate a Debian source package.

See Assumptions above.

Will need to guess:
 * path to executable
 * dependencies
 * architecture
 * distroseries
   * maybe we can actually figure this out based on dependencies / symbols

'''XXX:''' We ''could'' guess the version, based on tarball file name and/or the directory name within the tarball. Whether we need to guess it depends on the UI.


=== 4. Symbol dependency database ===

Guessing dependencies relies on a symbol dependency database. Currently, we are cribbing one from Debian, but that's actually a technically incorrect thing to do.

Thus, we must build & maintain our own symbol dependency database. Such a service is useful outside of the automagic packaging context and should probably be available as a public webservice.


== Implementation ==

=== 1. Web interface (Changes to myapps) ===

 * Need to make a decision about whether to show guesses & provide opportunity to correct them.
 * Need to call out to server-side pkgme and provide a callback for handling its return values. See below.

'''IMPORTANT:''' No matter what, we need input from the Canonical Web team as to how the user interface should change as a result of this new functionality.

=== 2. Server-side pkgme ===

 * Probably should do in Python
 * All PPAs currently owned by a specific team: ~commercial-ppa-owners or something
 * Each application in myapps knows its archive ID and signing key

The service will need to publish functions like:
 * `guess_packaging_details :``: devportal-metadata -> guesses`
 * `build_package :``: devportal-metadata -> guess_overrides -> source package`

The devportal-metadata will look a bit like this:
{{{
{
"name": "Angry Birds",
"package_name": "angrybirds",
"description": "A very popular game that lets you throw birds at pigs. I think.",
"tagline": "The best pig/bird combination since bacon and eggs",
"price": "5.00",
"tarball": "http://internal.url/to/tarball",
}
}}}

The guesses will look a bit like this:
{{{
{
"executable": "bin/angrybirds",
"architecture": "i386",
"dependencies": "libc6",
}
}}}

The tarball could also be sent as a blob instead of as a URL.

The source package could be sent either as a URL to a blob that can be downloaded or as the blob itself.

'''TODO:''' Define messages exactly. (esp. version & how blobs are sent)

However, these functions will need to be asynchronous, since both guessing and building the packages take a while. Thus, a callback URL will also need to be included and a mechanism defined for calling it back.

Since the tarball is likely to be a commercial application for sale, both it and the built source package must be available ''only'' to components within this system, and ''not'' to the general public.

 * '''XXX:''' Need to decide what web server technology will be used. jml favours Twisted.
 * '''XXX:''' What authentication / authorization mechanisms will we use?
 * '''XXX:''' Which component has the responsibility of uploading to PPA?
 * '''XXX:''' Who signs the upload to PPA?
 * '''XXX:''' Who is an appropriate changelog entry author for this service?
 * '''XXX:''' What is the name of the PPA?
 * '''XXX:''' Might need separate interfaces for new upload vs upgrade

=== 3. Binary packaging ===

Proof-of-concept exists at lp:~jml/+junk/pkgme-binary.

==== Known issues ====

 * Relies on external database to do symbol resolution
 * Does not fill in changelog properly or usefully
 * Does not take description from metadata and put it in control file
 * No support for copyright handling
 * Incorrectly specifies architecture as any, even though only builds for the architecture of the binary
 * Could actually build for both i386 and amd64 architectures, but only builds for the architecture of the binary
 * Does not support icons
 * Creates a desktop file & installs a desktop file, but for some reason installed app does not appear in desktop
 * Dependency guessing should take symbols and versions into account, instead only relies on .so file names
 * Duplicates code from dpkg-shlibdeps

=== 4. Dependency database ===

 * Only need to store information for distributions that have the Software Center
 * To be truly up-to-date, needs to import new symbols whenever new packages with libraries are published to the Ubuntu archive
 * Need to perform queries like dpkg-shlibdeps does: here are my needed symbols, return a list of packages and their versions.

Could do as a new independent system, or as a change to Launchpad.

As a new, independent system, might grow into a useful tool for performing map/reduce style operations on the whole archive.

As a change to Launchpad, would immediately gain all of Launchpad's existing infrastructure & deployment, including uptime guarantees.

'''TODO:''' Define the schema for this database.


== Test/Demo Plan ==

=== Test data ===

==== Expected good ====

 * Minimal GTK+ application
 * Minimal OpenGL application
 * Existing proprietary apps

==== Expected bad ====

 * A Python tarball
 * Tarballs without binaries
 * Tarballs with debian/ directory that already exists

==== Scenarios ====

 * Everything is golden and wonderful
 * User needs to change some of the packaging guesses
 * Package fails to build on Launchpad
 * Automagic packaging system temporarily down

== Unresolved issues ==

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

'''Question:''' How much is the pkgme command-line a deliverable for this? Are we right to just concentrate on the web app

  -- JamesWestby points out that it would be useful for local debugging of packages. The more easily they can be debugged on the web, the less need.


== Reading ==

 * [[http://www.debian.org/doc/debian-policy/ch-sharedlibs.html|Debian Policy Manual, Chapter 8 - Shared libraries]]
 * [[http://wiki.debian.org/Projects/ImprovedDpkgShlibdeps|Improved dpkg-shlibdeps project]]
 * [[http://wiki.debian.org/UsingSymbolsFiles|Using symbols files]]
 * [[http://manpages.ubuntu.com/dpkg-shlibdeps.1.html|dpkg-shlibdeps(1)]]
 * [[http://manpages.ubuntu.com/dpkg-gensymbols.1.html|dpkg-gensymbols(1)]]
 * [[http://manpages.ubuntu.com/deb-symbols.5.html|deb-symbols(5)]]

----
CategorySpec

Summary

A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.

Release Note

https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Uploaded tarballs are now available for testing within minutes.

Rationale

Ubuntu has a SoftwareCenter where people can buy applications. We want to make it very, very easy for application developers to publish their apps in the software center. For many such folk, this means publishing binary applications.

User stories

Initial upload

As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Option #1: No review of guessing for submitter

NOTE: This option provides no change in user experience from the current developer upload story.

  1. Go to https://myapps.developer.ubuntu.com

  2. Click “Submit a new application”

  3. Specify:
    1. The name of the application
    2. The name of the package

    3. A tagline for the application
    4. Its price
    5. A tarball for uploading
  4. Click next then specify:
    1. The “Department” the application belongs to

    2. A description for the application
    3. Some keywords
  5. Click next then specify:
    1. A screenshot
    2. Icons of various sizes
  6. Click next then specify:
    1. A support URL
    2. A license
  7. Click next then specify your payment details
  8. Submit your application

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

Option #2: Review of guesses for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Click “Submit a new application”

  3. Specify:
    1. The name of the application
    2. The name of the package

    3. A tagline for the application
    4. Its price
    5. A tarball for uploading
  4. Click next then specify:
    1. The “Department” the application belongs to

    2. A description for the application
    3. Some keywords
  5. Click next then specify:
    1. A screenshot
    2. Icons of various sizes
  6. Click next then specify:
    1. A support URL
    2. A license
  7. Click next then specify your payment details
  8. Submit your application
  9. Wait while myapps packages the application

  10. You will be presented with the “guesses” made by the automagic packaging, and given an opportunity to change them:

    • path to the main executable
    • list of Debian package dependencies
    • (possibly) the version of the package
    • (possibly) the distroseries to target
    • (possibly) the architecture to run on
  11. After making any changes, confirm your application submission

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

NOTE: Maybe the version of the package will be manually specified by the software developer. See https://bugs.launchpad.net/developer-portal/+bug/827490.

Provide upgrade

As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Option #1: No review of guessing for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Select your application
  3. In the “Your app” section, select “Edit”

  4. Upload a new version of your tarball
  5. (possibly) Specify the version number for this new upload
  6. Click “Save changes”

Then myapps will make the generated source package available to the reviewer, but not upload it to the PPA.

Option #2: Review of guesses for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Select your application
  3. In the “Your app” section, select “Edit”

  4. Upload a new version of your tarball
  5. (possibly) Specify the version number for this new upload
  6. Click “Save changes”

  7. Wait while myapps processes the new tarball

  8. If myapps thinks the dependencies, architecture or main executable have changed, then you will be presented with the new guesses for approval
  9. After making any changes, confirm your application submission

Then myapps will make the generated source package available to the reviewer, but not upload it to the PPA.

Review application

Once an application has been submitted by a developer, it is then placed in the review queue.

If the change was an initial upload, there will be the uploaded tarball and a link to the new PPA.

If the change was an upgrade, there will both the uploaded tarball and the built source package.

XXX: Should we have some sort of visual indicator within myapps for the state that the PPA upload is in? (i.e. uploaded, building, successfully built, failed to build, published)

Constraints and Requirements

Must

  • Take a binary tarball and turn it into a working source package
  • Upload the finished source package to a PPA
  • Support upgrades of packages
  • Integrate into https://myapps.developer.ubuntu.com

Nice to have

  • Figure out version from the containing directory in the uploaded tarball, if necessary

Must not

Out of scope

  • Anything other than commercial binary tarballs
    • Although we really love both source code and non-commercial software, we aren't doing it as part of this initiative. All of the underlying software & infrastructure is capable of handling it though.

Assumptions

  • Given a binary tarball and information from the developer portal
  • Binary is built using C or C++
  • The layout of the tarball is the same as the intended layout when running on a user's system
  • The tarball contents can be copied straight into /opt/<package_name>/ and be run from there by ordinary users

Design

Core components:

  1. Web interface (Changes to myapps)
  2. Server-side pkgme
  3. Binary packaging system
  4. Dependency database

1. Web interface (changes to myapps)

Changes to the myapps web interface need to be governed by the following:

  1. The essence of 'automagic' is guessing, which means that it will sometimes guess wrongly or not be able to guess at all.
  2. Making the guesses, generating the package and uploading to Launchpad are all slow.

There are two main options for changes to the myapps UI:

  1. Trust the guessing. Leave everything the same, except for adding new documentation & necessary fields.

  2. Don't trust the guessing. Give developers an opportunity to review the guesses made by the system.

2. Server-side pkgme

pkgme is constrained to be as simple as possible. It is meant to be a tool that can be used mindlessly and ignorantly to package upstream code. It is a command-line tool, and we expect the users to be open source developers who are currently using Ubuntu.

Launchpad / software center integration

  • Once the package is built, we need to upload it to a private PPA
    • Need "commercial admin" privileges to create a Launchpad PPA now
      • Julian wants to change this, but has no plan
  • XXX What needs to be done to get that PPA visible in software center?

  • XXX Launchpad has no facilities for telling us when something is done, save email. Does this matter?

Potential issues with Launchpad integration

3. Binary packaging system

Needs to take a tarball + metadata and generate a Debian source package.

See Assumptions above.

Will need to guess:

  • path to executable
  • dependencies
  • architecture
  • distroseries
    • maybe we can actually figure this out based on dependencies / symbols

XXX: We could guess the version, based on tarball file name and/or the directory name within the tarball. Whether we need to guess it depends on the UI.

4. Symbol dependency database

Guessing dependencies relies on a symbol dependency database. Currently, we are cribbing one from Debian, but that's actually a technically incorrect thing to do.

Thus, we must build & maintain our own symbol dependency database. Such a service is useful outside of the automagic packaging context and should probably be available as a public webservice.

Implementation

1. Web interface (Changes to myapps)

  • Need to make a decision about whether to show guesses & provide opportunity to correct them.

  • Need to call out to server-side pkgme and provide a callback for handling its return values. See below.

IMPORTANT: No matter what, we need input from the Canonical Web team as to how the user interface should change as a result of this new functionality.

2. Server-side pkgme

  • Probably should do in Python
  • All PPAs currently owned by a specific team: ~commercial-ppa-owners or something
  • Each application in myapps knows its archive ID and signing key

The service will need to publish functions like:

  • guess_packaging_details :: devportal-metadata -> guesses

  • build_package :: devportal-metadata -> guess_overrides -> source package

The devportal-metadata will look a bit like this:

{
"name": "Angry Birds",
"package_name": "angrybirds",
"description": "A very popular game that lets you throw birds at pigs. I think.",
"tagline": "The best pig/bird combination since bacon and eggs",
"price": "5.00",
"tarball": "http://internal.url/to/tarball",
}

The guesses will look a bit like this:

{
"executable": "bin/angrybirds",
"architecture": "i386",
"dependencies": "libc6",
}

The tarball could also be sent as a blob instead of as a URL.

The source package could be sent either as a URL to a blob that can be downloaded or as the blob itself.

TODO: Define messages exactly. (esp. version & how blobs are sent)

However, these functions will need to be asynchronous, since both guessing and building the packages take a while. Thus, a callback URL will also need to be included and a mechanism defined for calling it back.

Since the tarball is likely to be a commercial application for sale, both it and the built source package must be available only to components within this system, and not to the general public.

  • XXX: Need to decide what web server technology will be used. jml favours Twisted.

  • XXX: What authentication / authorization mechanisms will we use?

  • XXX: Which component has the responsibility of uploading to PPA?

  • XXX: Who signs the upload to PPA?

  • XXX: Who is an appropriate changelog entry author for this service?

  • XXX: What is the name of the PPA?

  • XXX: Might need separate interfaces for new upload vs upgrade

3. Binary packaging

Proof-of-concept exists at lp:~jml/+junk/pkgme-binary.

Known issues

  • Relies on external database to do symbol resolution
  • Does not fill in changelog properly or usefully
  • Does not take description from metadata and put it in control file
  • No support for copyright handling
  • Incorrectly specifies architecture as any, even though only builds for the architecture of the binary
  • Could actually build for both i386 and amd64 architectures, but only builds for the architecture of the binary
  • Does not support icons
  • Creates a desktop file & installs a desktop file, but for some reason installed app does not appear in desktop

  • Dependency guessing should take symbols and versions into account, instead only relies on .so file names
  • Duplicates code from dpkg-shlibdeps

4. Dependency database

  • Only need to store information for distributions that have the Software Center
  • To be truly up-to-date, needs to import new symbols whenever new packages with libraries are published to the Ubuntu archive
  • Need to perform queries like dpkg-shlibdeps does: here are my needed symbols, return a list of packages and their versions.

Could do as a new independent system, or as a change to Launchpad.

As a new, independent system, might grow into a useful tool for performing map/reduce style operations on the whole archive.

As a change to Launchpad, would immediately gain all of Launchpad's existing infrastructure & deployment, including uptime guarantees.

TODO: Define the schema for this database.

Test/Demo Plan

Test data

Expected good

  • Minimal GTK+ application
  • Minimal OpenGL application
  • Existing proprietary apps

Expected bad

  • A Python tarball
  • Tarballs without binaries
  • Tarballs with debian/ directory that already exists

Scenarios

  • Everything is golden and wonderful
  • User needs to change some of the packaging guesses
  • Package fails to build on Launchpad
  • Automagic packaging system temporarily down

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

Question: How much is the pkgme command-line a deliverable for this? Are we right to just concentrate on the web app

  • -- JamesWestby points out that it would be useful for local debugging of packages. The more easily they can be debugged on the web, the less need.

Reading


CategorySpec

AutomagicBinaryPackaging (last edited 2012-03-08 11:44:11 by cpc11-acto4-2-0-cust359)