ThirdPartyApt

Introduction

Installing a third party Apt repository is too hard. ThirdPartyApt defines a file format that ISVs can publish on their web sites and distribute to users to download their software, optionally enabling automatic updates.

This spec makes it easy for users to install third party software and have it update automatically. By doing this in a consistent and standard way, we both discourage the use of difficult custom install methods and encourage better security than is currently practiced.

Rationale

Users want to use third party software. Third parties should be able to distribute it to them. We should provide tools to make installation, integration and maintenance of their programs easy for both users and developers. Apt is an amazingly powerful and useful tool for such software.

The current situation

ISVs which target Ubuntu create their own, usually proprietary, installation infrastructure. These installers are not as robust as dpkg and can break things, particularly during distribution upgrades. They may install files that never get removed, if they even have an uninstall option. They may conflict with or even overwrite system files.

Even if it works, our users are harmed by these installers. They have no central interface to view currently installed software, or remove it. Automatic updates become as difficult as they are on Windows.

Some more savvy third parties provide .deb files which the user can install simply by double clicking and launching with GDebI. While debs avoid many of the problems of custom install frameworks above, they introduce their own problems without apt. Splitting packages into multiple debs is a tremendous hassle, and automatically providing updates, including security updates, is impossible.

Even fewer third parties actually provide apt repositories for their packages. These packages are generally the best, but still not perfect. The user is required to copy and paste an obscure looking URL into sudoed text editor or Synaptic's Add Repository interface. The same has to be done for the apt-key which allows access to the repository. Update manager doesn't know how to handle them on dist upgrade.

What Third Party Apt does

ThirdPartyApt provides for Apt Repositories what GDebI provides for deb files. We make it easy for users to use debs and repositories so ISVs will be more likely to provide them instead of custom install frameworks.

ThirdPartyApt on its own does not force ISVs to use it. If the framework exists, and is well supported by Ubuntu, then ISVs will consider automatic updates and integration a service to their users. If not, the situation doesn't change, but there's no harm done.

Scope and Use Cases

The main use case is simple. A user should be able to be provided a link to click on by a third party ISV which will prompt the user for permission to install the software. The user should be able to accept and start the installation of that ISVs software. The ISV may provide this link on a Download page, or after an accepted payment transaction as been processed.

The installed software should track updates from the ISV automatically and integrate into the existing infrastructure. Update-notifier should notify the user when the ISVs software is updated.

For Ubuntu specific repositories, ISVs currently must give elaborate instructions for users to setup and enable the repository (eg: here for Wine, or here for Google). Third Party Apt allows them to instead distribute a single file for users to double click on.

Just before the Gutsy release, approximately 70,000 users were using the WineHQ APT repositories despite these complicated instructions, representing over 1% of Ubuntu users. While users of the WineHQ repository may no longer need to keep current once the version of Wine included with Ubuntu becomes "good enough," some free software projects will always require users to keep current with the very latest version of the software. The Spring project, for instance requires users to have the same, latest version of spring in order to play multiplayer. Since Ubuntu packages can only update every six months (or, at the very best, after a week's delay with backports enabled), Ubuntu users would be unable to play Spring without a Third Party Apt repository maintained by the Spring project.

Design

We create a .apt file format containing information necessary to install a package, optionally adding a repository and sanely handling upgrades.

We then create a command-line program to handle these .apt files, and a graphical frontend for it.

.apt file format

The .apt file is formatted with standard dpkg stanzas. Each stanza contains a repository URL and filters describing what type of platform this URL is suitable for. Consider the example stanza: 

Architecture: i386
Distribution: Ubuntu
Codename: breezy
Release: 5.10
Archive:
 http://us.archive.ubuntu.com/ubuntu breezy universe
Install: photoshop

The first four tags, Architecture, Distribution, Codename and Release specify the specific system that this stanza applies to. Each of these are read and tested against the user's system. If the user's system does not match, the stanza is ignored. This allows a single .apt file to specify delivery of an apt repository for multiple distributions and releases. A separate stanza could be constructed to apply to Ubuntu and/or Debian, pointing to different archives.

List of tags

Archive lists the APT archives to add to the sources.list file.

Mime type

The first line of the file is then set to "#@application/x-apt 0". This allows mime magic to be used to detect the file type properly. All of this is mandatory, and the client implementation should reject a non-compliant file. The "0" following the mime type is the version number of the Apt file specification that the file conforms to. As breaking changes are made, we increase this number. This number will stay 0 until the specification is actually put into use.

Signing the .apt file

The resulting file is then signed with the same private key used to sign the Releases file of the archive. Signing of the file is mandatory. A guarantee of a single identity must be available from the .apt file itself all the way to the installing of the packages. The public key required to be added to apt-key is then appended to the end of the .apt file. This way, if the user trusts the download link, then he can trust the repository software (see security below).

ISVs should be encouraged to create a specific repository key and not, for instance, adopt a personal key. 

#@application/x-apt 0
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Architecture: i386
Distribution: Ubuntu
Codename: dapper
Install: photoshop
Archive:
 deb http://archive.adobe.com/archive-ubuntu dapper multiverse

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFEPICNRTZ5+YJ1b+YRAvDoAJ9+NBnSPLa32h7J+tbgisWrh58C2QCdHHwv
<etc>
=z7nf
-----END PGP SIGNATURE-----
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

mQGiBD/G9AgRBACZ519LX9cdoyJA+7gmWC+mUsiyPhnmMWu4uOg0M+vb/JPtDdfc
<etc>
=pIE5
-----END PGP PUBLIC KEY BLOCK-----

The above is an example of a final, signed, .apt file.

Programs

A program is provided to handle the .apt file type. When this program launches a .apt file, it strips the mime type, verifies the signature, exports the public key, parses all the stanzas and determines which applies to the current environment. It then walks the user through a few simple questions asking if they are sure they want to trust the included public key to install software onto their system. After the user has confirmed all the steps (and agreed to any applicable warnings about trusting third party repositories), the public key is added to apt-key and the packages specified by Install are installed.

apt-list and GAptI

apt-list is the console backend to this program. apt-list will support installing .apt files, adding the repository within them (without installing the default packages), listing all repositories on a system, and removing a repository (optionally removing the packages it installed as well.)

The graphical frontend to apt-list, GAptI, is already partially written.

Single package install

The implementation could remove the repository and key after a successful install if upgrade tracking isn't requested. This allows ISVs to provide a single, permanent download link for an application on any version of Ubuntu. For instance, rather than linking to foo-hardy-ubuntu1.deb, I could make a howto guide pointing to foo.apt; on hardy this would then install the latest foo.deb, while on hardy+1 it could simply install the foo.deb included in the Ubuntu repository (see self destruct, below).

Modification to apt-get

apt-get can be modified to use the repository name feature introduced here to allow for easier pinning. For instance, the command apt-get -pin winehq install wine could pin the wine package to the winehq repository (ie, the one in /etc/apt/sources.list.d/winehq.list) and then install it. This prevents having to manually add pinning preferences via editing a config file when using the command line.

Modification to synaptic

Synaptic will need to be modified slightly, as the Third Party Repositories preferences tab could accommodate the metadata we have here.

Changes in /etc

/etc/apt/sources.list will be considered deprecated - all repositories will now reside in their own files in /etc/apt/sources.list.d/ For Ubuntu repositories, these will be: 

/etc/apt/sources.list.d/main.list
/etc/apt/sources.list.d/restricted.list
/etc/apt/sources.list.d/universe.list
/etc/apt/sources.list.d/multiverse.list
/etc/apt/sources.list.d/partner.list

Creation of /etc/apt/keys.d

Rather than maintaining a single keyfile, we add new keys as separate files in /etc/apt/keys.d. This makes inspection, cleanup, and future signing of keys much easier to implement than having to call separate instances of apt-key.

Security

Security is better than the status quo

Security with this feature is no worse than the current system, where users blindly copy and paste terminal commands with root permission or give Synaptic instructions. By making it easy to avoid running terminal commands, we make users even more suspcious when asked to do so by a malicious entity.

There is no good way to prevent users from installing software that they want to install; similarly, once a user is convinced he wants to install something, any attempt at preventing him simply builds frustration and a lack of trust in good security.

Preventing accidental installs and properly warning the user

Obviously, a user should not confirm a package installation which he didn't expect to happen, such as a web site opening a .apt file automatically. Our frontend interface handles this elegantly; by standardizing the look and feel of what a third party installation is and informing the user what he is about to do, a user who doesn't want to install something will know to click cancel.

By making a standardized good interface for installing third party software, we also allow the user to make an informed decision about what he is doing. This currently does not happen with shell scripts and terminal commands.

Preventing spoofing

Since the .apt file must be signed by the same key that the repository it provides uses, any apt file the user receives can be known to have come "officially" from the repository. This allows repository managers a greater control over their own default installation, allowing them to prevent possible breakages caused by installing the wrong packages from some other .apt file.

More importantly, if the user trusts the website from where he is downloading the .apt file, then he can implicitly trust the repository as well. While we still need to worry that the user has been mislead via phishing or DNS cache poisoning, this problem is no worse with .apt files than it is with proprietary installation frameworks.

One possible trick is for a spoofed website to open it's own .apt file "quicker" than the expected web site, tricking the user into agreeing to the top-most instance of the confirmation dialog. To work around this, we can use a similar locking mechanism that we use in synaptic and apt-get - a second instance of the .apt handler attempting to run would spit out a warning to the user.

Future key signing

Once the software infrastructure is implemented, we can eventually investigate other means of providing further security. For instance, we could ship a set of trusted keys, and then use those keys to sign known reputable ISVs (say, Google). That way, when a key is unknown to us we could float a warning that, for instance, Ubuntu couldn't verify the repository was from Google. Such key signing, however, is beyond the scope of this blueprint.

Discussion issues

Release Note

Users may now install and remove third party software and repositories using a simple interface. After downloading and double clicking a .apt file...(finish later)

Meeting comments

Upgrades

Releated work

Nokia .install file format described at:

The file is recognised on the basis of:

which is added to:

Example 

[install]
repo_name = Foo Catalogue
repo_deb  = deb http://foo.com/maemo mistral user
package   = foo-app

Meta-data

Standardise the meta-data format (Icons, clear text name...) for each repo.

Process

Both Automatix and WineHQ are add a line to sources.list, first task is to optimise this.

New Comments

Old Comments

ThirdPartyApt (last edited 2009-02-05 23:50:29 by 207)