A website that allows upstream software vendors to upload binary tarballs and then turns those tarballs into packages in a PPA as quickly and smoothly as possible.
https://developer.ubuntu.com now automatically packages binary tarballs uploaded by software vendors. Uploaded tarballs are now available for testing within minutes.
Ubuntu has a SoftwareCenter where people can buy applications. We want to make it very, very easy for application developers to publish their apps in the software center. For many such folk, this means publishing binary applications.
- As a software developer, I want to upload my new application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.
- As a software developer, I want to upload a new version of my application to Ubuntu without having to package it, so that my app appears in the Software Center and people can buy it.
Constraints and Requirements
- Take a binary tarball and turn it into a working source package
- Upload the finished source package to a PPA
- Definitely need to support upgrades
Nice to have
- Figure out version from the containing directory in the uploaded tarball, if necessary
Out of scope
- Anything other than commercial binary tarballs
Although we really love both source code and non-commercial software, we aren't doing it as part of this initiative. All of the underlying software & infrastructure is capable of handling it though.
- Given a binary tarball and information from the developer portal
- Binary is built using C or C++
- The layout of the tarball is the same as the intended layout when running on a user's system
The tarball contents can be copied straight into /opt/<package_name>/ and be run from there by ordinary users
The core process:
- Take a tarball plus the metadata provided in the webform
- Create a debian/ directory and then a source package
- Upload the source package to a PPA on Launchpad
- Our goal is something that can be integrated into the existing myapps portlet.
- The essence of the 'automagic' is guessing, this means
- users must be given the opportunity to review the guesses
- users must be able to correct the guesses
- the guesswork has to be done within the time of a user's attention span
- some things will always not work
- therefore, we need to figure out how best to handle those failures
Question: How much is the pkgme command-line a deliverable for this? Are we right to just concentrate on the web app
-- JamesWestby points out that it would be useful for local debugging of packages. The more easily they can be debugged on the web, the less need.
Proof-of-concept exists at lp:~jml/+junk/pkgme-binary.
'pkgme' is constrained to be as simple as possible. It is meant to be a tool that can be used mindlessly and ignorantly to package upstream code. It is a command-line tool, and we expect the users to be open source developers who are currently using Ubuntu.
- executable path
- maybe we can actually figure this out based on dependencies / symbols
Symbol dependency database
Guessing dependencies relies on a symbol dependency database. Currently, we are cribbing one from Debian, but that's actually a technically incorrect thing to do.
Posit: we should build & maintain our own symbol dependency database as a separate, public, web service.
Launchpad / software center integration
- Once the package is built, we need to upload it to a private PPA
- Need "commercial admin" privileges to create a Launchpad PPA now
- Julian wants to change this, but has no plan
- Need "commercial admin" privileges to create a Launchpad PPA now
XXX What needs to be done to get that PPA visible in software center
XXX Launchpad has no facilities for telling us when something is done, save email. Does this matter?
Integration with myapps
- All PPAs currently owned by a specific team: ~commercial-ppa-owners or something
- Each application knows its archive ID and signing key
- How much metadata to send to packaging?
- How much integration with myapps?
- i.e. do we actually go back and get the user to confirm values? (sg can answer)
- How many things does the app actually guess?
- Need to sign the package for upload -- which signature
- How to get the tarball from app server to packaging server
What authentication & authorization to use for packaging?
- packaging service might need access to tarball via web
- either way, would need hook to call back to myapps on
IMPORTANT: No current plan for this. This is the biggest risk, IMO, especially if we rely on the Canonical Web team to provide the UI. – jml
We will need to change the myapps UI to allow for work to be done post-upload processing.
Code changes should include an overview of what needs to change, and in some cases even the specific details.
- Probably should do packaging service in Python
- Maybe could do dependency database in Go?
Known issues with binary backend
- Relies on external database to do symbol resolution
- Does not fill in changelog properly or usefully
- Does not take description from metadata and put it in control file
- No support for copyright handling
- Incorrectly specifies architecture as any, even though only builds for the architecture of the binary
- Could actually build for both i386 and amd64 architectures, but only builds for the architecture of the binary
- Does not support icons
Creates a desktop file & installs a desktop file, but for some reason installed app does not appear in desktop
- Dependency guessing should take symbols and versions into account, instead only relies on .so file names
- Duplicates code from dpkg-shlibdeps
Potential issues with Launchpad integration
https://bugs.launchpad.net/launchpadlib/+bug/814595 -- unrecoverable error when auth fails
https://bugs.launchpad.net/launchpad/+bug/814633 -- cannot delete ppa over api
https://bugs.launchpad.net/launchpad/+bug/814605 -- dodgy errors when cannot create
https://bugs.launchpad.net/launchpad/+bug/814725 -- no feedback between acceptance upload & rejection
https://bugs.launchpad.net/launchpad/+bug/637915 -- cannot tell if a PPA is deleted or not over API
https://bugs.launchpad.net/launchpad/+bug/568981 -- cannot upload GPG key via API
- data migration, if any
- redirects from old URLs to new ones, if any
- how users will be pointed to the new way of doing things, if necessary.
- Minimal GTK+ application
- Minimal OpenGL application
- Existing proprietary apps
- A Python tarball
- Tarballs without binaries
- Tarballs with debian/ directory that already exists
- Everything is golden and wonderful
- User needs to change some of the packaging guesses
- Package fails to build on Launchpad
- Automagic packaging system temporarily down
This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.