AutomagicBinaryPackaging

Differences between revisions 1 and 13 (spanning 12 versions)
Revision 1 as of 2011-07-11 13:14:07
Size: 1701
Editor: jml
Comment:
Revision 13 as of 2011-08-23 10:09:03
Size: 12195
Editor: jml
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
Given binary tarballs from upstream software vendors, turn those into packages in a PPA as quickly and smoothly as possible.

'''Contact:''' JonathanLange <<BR>>
'''On Launchpad:''' ## Link to a blueprint, milestone or (best) a bug tag search across launchpad-project
 * '''Launchpad Entry''':
 * '''Created''': [[<<Date(2011-08-19T12:25:27Z)>>]]
 * '''Contributors''': JonathanLange
 * '''Packages affected''':

== Summary ==

A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.

== Release Note ==

https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Uploaded tarballs are now available for testing within minutes.
Line 10: Line 18:
== Stakeholders ==

|| Name || Interest || Last consulted ||
|| Rick Spencer || Ubuntu Engineering manager || 2011-07-11 ||
|| ??? || || ||
Line 18: Line 20:
## These are a collection of things that users want to do, and why they want to do them. Try very, very hard to phrase them in terms of user needs rather than implementation details.

=== $STORY_NAME ===

'''As a ''' $PERSON<<BR>>
'''I want ''' $FEATURE<<BR>>
'''so that ''' $BENEFIT<<BR>>

## Have as many as you like. Group user stories together into meaningfully deliverable units. They can then be used as the driving elements of exploratory testing.
=== Initial upload ===

As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

'''Option #1: No review of guessing for submitter'''

'''NOTE:''' This option provides no change in user experience from the current developer upload story.

 1. Go to https://myapps.developer.ubuntu.com
 1. Click &ldquo;Submit a new application&rdquo;
 1. Specify:
   1. The name of the application
   1. '''The name of the package'''
   1. A tagline for the application
   1. Its price
   1. A tarball for uploading
 1. Click next then specify:
   1. The &ldquo;Department&rdquo; the application belongs to
   1. A description for the application
   1. Some keywords
 1. Click next then specify:
   1. A screenshot
   1. Icons of various sizes
 1. Click next then specify:
   1. A support URL
   1. A license
 1. Click next then specify your payment details
 1. Submit your application

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.


'''Option #2: Review of guesses for submitter'''

 1. Go to https://myapps.developer.ubuntu.com
 1. Click &ldquo;Submit a new application&rdquo;
 1. Specify:
   1. The name of the application
   1. '''The name of the package'''
   1. A tagline for the application
   1. Its price
   1. A tarball for uploading
 1. Click next then specify:
   1. The &ldquo;Department&rdquo; the application belongs to
   1. A description for the application
   1. Some keywords
 1. Click next then specify:
   1. A screenshot
   1. Icons of various sizes
 1. Click next then specify:
   1. A support URL
   1. A license
 1. Click next then specify your payment details
 1. Submit your application
 1. '''Wait while myapps packages the application'''
 1. You will be presented with the &ldquo;guesses&rdquo; made by the automagic packaging, and given an opportunity to change them:
   * path to the main executable
   * list of Debian package dependencies
   * (possibly) the version of the package
   * (possibly) the distroseries to target
   * (possibly) the architecture to run on
 1. After making any changes, confirm your application submission

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

'''NOTE:''' Maybe the version of the package will be manually specified by the software developer. See https://bugs.launchpad.net/developer-portal/+bug/827490.

=== Provide upgrade ===

As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

'''Option #1: No review of guessing for submitter'''

 1. Go to https://myapps.developer.ubuntu.com
 1. Select your application
 1. In the &ldquo;Your app&rdquo; section, select &ldquo;Edit&rdquo;
 1. Upload a new version of your tarball
 1. (possibly) Specify the version number for this new upload
 1. Click &ldquo;Save changes&rdquo;

Then myapps will make the generated source package available to the reviewer, but ''not'' upload it to the PPA.

'''Option #2: Review of guesses for submitter'''

 1. Go to https://myapps.developer.ubuntu.com
 1. Select your application
 1. In the &ldquo;Your app&rdquo; section, select &ldquo;Edit&rdquo;
 1. Upload a new version of your tarball
 1. (possibly) Specify the version number for this new upload
 1. Click &ldquo;Save changes&rdquo;
 1. ''Wait while myapps processes the new tarball''
 1. If myapps thinks the dependencies, architecture or main executable have changed, then you will be presented with the new guesses for approval
 1. After making any changes, confirm your application submission

Then myapps will make the generated source package available to the reviewer, but ''not'' upload it to the PPA.


=== Review application ===

Once an application has been submitted by a developer, it is then placed in the review queue.

If the change was an initial upload, there will be the uploaded tarball and a link to the new PPA.

If the change was an upgrade, there will both the uploaded tarball and the built source package.

'''XXX:''' Should we have some sort of visual indicator within myapps for the state that the PPA upload is in? (i.e. uploaded, building, successfully built, failed to build, published)
Line 32: Line 131:
## What MUST the new behaviour provide?  * Take a binary tarball and turn it into a working source package
 * Upload the finished source package to a PPA
 * Support upgrades of packages
 * Integrate into https://myapps.developer.ubuntu.com
Line 36: Line 138:
## What would be nice to have, but is not essential  * Figure out version from the containing directory in the uploaded tarball, if necessary
Line 44: Line 146:
## Things that people might think are in scope, but are actually not.

== Subfeatures ==

## Other specs that form a part of this one.

== Measures of success ==

## How will we measure how well we have done?

== Thoughts? ==

## Put everything else here. Better out than in.
 * Anything other than commercial binary tarballs
   * Although we really love both source code and non-commercial software, we aren't doing it as part of this initiative. All of the underlying software & infrastructure is capable of handling it though.

=== Assumptions ===

 * Given a binary tarball and information from the developer portal
 * Binary is built using C or C++
 * The layout of the tarball is the same as the intended layout when running on a user's system
 * The tarball contents can be copied straight into /opt/<package_name>/ and be run from there by ordinary users

== Design ==

Core components:
 1. Web interface (Changes to myapps)
 1. Server-side pkgme
 1. Binary packaging system
 1. Dependency database

=== 1. Web interface (changes to myapps) ===

Changes to the myapps web interface need to be governed by the following:
 1. The essence of 'automagic' is guessing, which means that it will sometimes guess wrongly or not be able to guess at all.
 1. Making the guesses, generating the package and uploading to Launchpad are all slow.

There are two main options for changes to the myapps UI:
 1. Trust the guessing. Leave everything the same, except for adding new documentation & necessary fields.
 1. Don't trust the guessing. Give developers an opportunity to review the guesses made by the system.


=== 2. Server-side pkgme ===

[[https://pkgme.net/|pkgme]] is constrained to be as simple as possible. It is meant to be a tool that can be used mindlessly and ignorantly to package upstream code. It is a command-line tool, and we expect the users to be open source developers who are currently using Ubuntu.

==== Launchpad / software center integration ====

 * Once the package is built, we need to upload it to a private PPA
   * Need "commercial admin" privileges to create a Launchpad PPA now
     * Julian wants to change this, but has no plan
 * '''XXX''' What needs to be done to get that PPA visible in software center?
 * '''XXX''' Launchpad has no facilities for telling us when something is done, save email. Does this matter?

==== Potential issues with Launchpad integration ====

 * https://bugs.launchpad.net/launchpadlib/+bug/814595 -- unrecoverable error when auth fails
 * https://bugs.launchpad.net/launchpad/+bug/814633 -- cannot delete ppa over api
 * https://bugs.launchpad.net/launchpad/+bug/814605 -- dodgy errors when cannot create
 * https://bugs.launchpad.net/launchpad/+bug/814725 -- no feedback between acceptance upload & rejection
 * https://bugs.launchpad.net/launchpad/+bug/637915 -- cannot tell if a PPA is deleted or not over API
 * https://bugs.launchpad.net/launchpad/+bug/568981 -- cannot upload GPG key via API


=== 3. Binary packaging system ===

Proof-of-concept exists at lp:~jml/+junk/pkgme-binary.


Guesses:
 * executable path
 * dependencies
 * architecture
 * distroseries
   * maybe we can actually figure this out based on dependencies / symbols
 * version?

==== Known issues with binary backend ====

 * Relies on external database to do symbol resolution
 * Does not fill in changelog properly or usefully
 * Does not take description from metadata and put it in control file
 * No support for copyright handling
 * Incorrectly specifies architecture as any, even though only builds for the architecture of the binary
 * Could actually build for both i386 and amd64 architectures, but only builds for the architecture of the binary
 * Does not support icons
 * Creates a desktop file & installs a desktop file, but for some reason installed app does not appear in desktop
 * Dependency guessing should take symbols and versions into account, instead only relies on .so file names
 * Duplicates code from dpkg-shlibdeps


=== 4. Symbol dependency database ===

Guessing dependencies relies on a symbol dependency database. Currently, we are cribbing one from Debian, but that's actually a technically incorrect thing to do.

Thus, we must build & maintain our own symbol dependency database. Such a service is useful outside of the automagic packaging context and should probably be available as a public webservice.


== Implementation ==

 * All PPAs currently owned by a specific team: ~commercial-ppa-owners or something
 * Each application knows its archive ID and signing key

=== Questions ===

 * How much metadata to send to packaging?
 * How much integration with myapps?
   * i.e. do we actually go back and get the user to confirm values? (sg can answer)
 * How many things does the app actually guess?
 * Need to sign the package for upload -- which signature
 * How to get the tarball from app server to packaging server
 * What authentication & authorization to use for packaging?
   * packaging service might need access to tarball via web
   * either way, would need hook to call back to myapps on


=== UI Changes ===

'''IMPORTANT:''' No current plan for this. This is the biggest risk, IMO, especially if we rely on the Canonical Web team to provide the UI. &ndash; jml

We will need to change the myapps UI to allow for work to be done post-upload processing.

=== Code Changes ===

Code changes should include an overview of what needs to change, and in some cases even the specific details.

 * Probably should do packaging service in Python
 * Maybe could do dependency database in Go?




== Test/Demo Plan ==

=== Test data ===

==== Expected good ====

 * Minimal GTK+ application
 * Minimal OpenGL application
 * Existing proprietary apps

==== Expected bad ====

 * A Python tarball
 * Tarballs without binaries
 * Tarballs with debian/ directory that already exists

==== Scenarios ====

 * Everything is golden and wonderful
 * User needs to change some of the packaging guesses
 * Package fails to build on Launchpad
 * Automagic packaging system temporarily down

== Unresolved issues ==

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

'''Question:''' How much is the pkgme command-line a deliverable for this? Are we right to just concentrate on the web app

  -- JamesWestby points out that it would be useful for local debugging of packages. The more easily they can be debugged on the web, the less need.


----
CategorySpec

Summary

A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.

Release Note

https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Uploaded tarballs are now available for testing within minutes.

Rationale

Ubuntu has a SoftwareCenter where people can buy applications. We want to make it very, very easy for application developers to publish their apps in the software center. For many such folk, this means publishing binary applications.

User stories

Initial upload

As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Option #1: No review of guessing for submitter

NOTE: This option provides no change in user experience from the current developer upload story.

  1. Go to https://myapps.developer.ubuntu.com

  2. Click “Submit a new application”

  3. Specify:
    1. The name of the application
    2. The name of the package

    3. A tagline for the application
    4. Its price
    5. A tarball for uploading
  4. Click next then specify:
    1. The “Department” the application belongs to

    2. A description for the application
    3. Some keywords
  5. Click next then specify:
    1. A screenshot
    2. Icons of various sizes
  6. Click next then specify:
    1. A support URL
    2. A license
  7. Click next then specify your payment details
  8. Submit your application

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

Option #2: Review of guesses for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Click “Submit a new application”

  3. Specify:
    1. The name of the application
    2. The name of the package

    3. A tagline for the application
    4. Its price
    5. A tarball for uploading
  4. Click next then specify:
    1. The “Department” the application belongs to

    2. A description for the application
    3. Some keywords
  5. Click next then specify:
    1. A screenshot
    2. Icons of various sizes
  6. Click next then specify:
    1. A support URL
    2. A license
  7. Click next then specify your payment details
  8. Submit your application
  9. Wait while myapps packages the application

  10. You will be presented with the “guesses” made by the automagic packaging, and given an opportunity to change them:

    • path to the main executable
    • list of Debian package dependencies
    • (possibly) the version of the package
    • (possibly) the distroseries to target
    • (possibly) the architecture to run on
  11. After making any changes, confirm your application submission

Then myapps will generate a package and upload it to a PPA. The submission will then appear in the review queue.

NOTE: Maybe the version of the package will be manually specified by the software developer. See https://bugs.launchpad.net/developer-portal/+bug/827490.

Provide upgrade

As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.

Option #1: No review of guessing for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Select your application
  3. In the “Your app” section, select “Edit”

  4. Upload a new version of your tarball
  5. (possibly) Specify the version number for this new upload
  6. Click “Save changes”

Then myapps will make the generated source package available to the reviewer, but not upload it to the PPA.

Option #2: Review of guesses for submitter

  1. Go to https://myapps.developer.ubuntu.com

  2. Select your application
  3. In the “Your app” section, select “Edit”

  4. Upload a new version of your tarball
  5. (possibly) Specify the version number for this new upload
  6. Click “Save changes”

  7. Wait while myapps processes the new tarball

  8. If myapps thinks the dependencies, architecture or main executable have changed, then you will be presented with the new guesses for approval
  9. After making any changes, confirm your application submission

Then myapps will make the generated source package available to the reviewer, but not upload it to the PPA.

Review application

Once an application has been submitted by a developer, it is then placed in the review queue.

If the change was an initial upload, there will be the uploaded tarball and a link to the new PPA.

If the change was an upgrade, there will both the uploaded tarball and the built source package.

XXX: Should we have some sort of visual indicator within myapps for the state that the PPA upload is in? (i.e. uploaded, building, successfully built, failed to build, published)

Constraints and Requirements

Must

  • Take a binary tarball and turn it into a working source package
  • Upload the finished source package to a PPA
  • Support upgrades of packages
  • Integrate into https://myapps.developer.ubuntu.com

Nice to have

  • Figure out version from the containing directory in the uploaded tarball, if necessary

Must not

Out of scope

  • Anything other than commercial binary tarballs
    • Although we really love both source code and non-commercial software, we aren't doing it as part of this initiative. All of the underlying software & infrastructure is capable of handling it though.

Assumptions

  • Given a binary tarball and information from the developer portal
  • Binary is built using C or C++
  • The layout of the tarball is the same as the intended layout when running on a user's system
  • The tarball contents can be copied straight into /opt/<package_name>/ and be run from there by ordinary users

Design

Core components:

  1. Web interface (Changes to myapps)
  2. Server-side pkgme
  3. Binary packaging system
  4. Dependency database

1. Web interface (changes to myapps)

Changes to the myapps web interface need to be governed by the following:

  1. The essence of 'automagic' is guessing, which means that it will sometimes guess wrongly or not be able to guess at all.
  2. Making the guesses, generating the package and uploading to Launchpad are all slow.

There are two main options for changes to the myapps UI:

  1. Trust the guessing. Leave everything the same, except for adding new documentation & necessary fields.

  2. Don't trust the guessing. Give developers an opportunity to review the guesses made by the system.

2. Server-side pkgme

pkgme is constrained to be as simple as possible. It is meant to be a tool that can be used mindlessly and ignorantly to package upstream code. It is a command-line tool, and we expect the users to be open source developers who are currently using Ubuntu.

Launchpad / software center integration

  • Once the package is built, we need to upload it to a private PPA
    • Need "commercial admin" privileges to create a Launchpad PPA now
      • Julian wants to change this, but has no plan
  • XXX What needs to be done to get that PPA visible in software center?

  • XXX Launchpad has no facilities for telling us when something is done, save email. Does this matter?

Potential issues with Launchpad integration

3. Binary packaging system

Proof-of-concept exists at lp:~jml/+junk/pkgme-binary.

Guesses:

  • executable path
  • dependencies
  • architecture
  • distroseries
    • maybe we can actually figure this out based on dependencies / symbols
  • version?

Known issues with binary backend

  • Relies on external database to do symbol resolution
  • Does not fill in changelog properly or usefully
  • Does not take description from metadata and put it in control file
  • No support for copyright handling
  • Incorrectly specifies architecture as any, even though only builds for the architecture of the binary
  • Could actually build for both i386 and amd64 architectures, but only builds for the architecture of the binary
  • Does not support icons
  • Creates a desktop file & installs a desktop file, but for some reason installed app does not appear in desktop

  • Dependency guessing should take symbols and versions into account, instead only relies on .so file names
  • Duplicates code from dpkg-shlibdeps

4. Symbol dependency database

Guessing dependencies relies on a symbol dependency database. Currently, we are cribbing one from Debian, but that's actually a technically incorrect thing to do.

Thus, we must build & maintain our own symbol dependency database. Such a service is useful outside of the automagic packaging context and should probably be available as a public webservice.

Implementation

  • All PPAs currently owned by a specific team: ~commercial-ppa-owners or something
  • Each application knows its archive ID and signing key

Questions

  • How much metadata to send to packaging?
  • How much integration with myapps?
    • i.e. do we actually go back and get the user to confirm values? (sg can answer)
  • How many things does the app actually guess?
  • Need to sign the package for upload -- which signature
  • How to get the tarball from app server to packaging server
  • What authentication & authorization to use for packaging?

    • packaging service might need access to tarball via web
    • either way, would need hook to call back to myapps on

UI Changes

IMPORTANT: No current plan for this. This is the biggest risk, IMO, especially if we rely on the Canonical Web team to provide the UI. – jml

We will need to change the myapps UI to allow for work to be done post-upload processing.

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

  • Probably should do packaging service in Python
  • Maybe could do dependency database in Go?

Test/Demo Plan

Test data

Expected good

  • Minimal GTK+ application
  • Minimal OpenGL application
  • Existing proprietary apps

Expected bad

  • A Python tarball
  • Tarballs without binaries
  • Tarballs with debian/ directory that already exists

Scenarios

  • Everything is golden and wonderful
  • User needs to change some of the packaging guesses
  • Package fails to build on Launchpad
  • Automagic packaging system temporarily down

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

Question: How much is the pkgme command-line a deliverable for this? Are we right to just concentrate on the web app

  • -- JamesWestby points out that it would be useful for local debugging of packages. The more easily they can be debugged on the web, the less need.


CategorySpec

AutomagicBinaryPackaging (last edited 2012-03-08 11:44:11 by jml)