ThirdPartyApt

• Launchpad Entry: https://launchpad.net/distros/ubuntu/+spec/third-party-apt

• Created: JerryHaltom

Introduction

ThirdPartyApt defines a file format that ISVs can publish on their web sites and distribute to users to download their software.

Rationale

Third parties should be able to distribute programs for installation on Ubuntu or other Apt based distributions (especially Ubuntu). They should be provided tools to make installation, integration and maintenance of their programs easy. We have existing infrastructure to provide to them, but need to do so in a consistent way while providing an easy to use interface.

Currently ISVs which target Ubuntu create their own, usual proprietary, sometimes common, installation infrastructure. This infrastructure may not be as robust as DPkg and may introduce problems into a user's system as a result of its work. This may install files onto the user's system which are never removed during uninstall. They may not even have uninstall options. This also causes an inconsistency for users of our platform. They have no central interface by which to view currently installed third party software. Each product may have its own auto update mechanism. All of this causes the user a hassle at some point.

Some more savvy third parties provide .deb files which the user can install simply by double clicking and launching with GDebI. These ISVs are not very common. They provide their users a valuable service, but the picture isn't totally rosy. Generally .deb files are split into pieces, depending on the function of the package. This requires the user to manually download a .deb for perhaps -common, -core, -art, -kde, -gnome. It also doesn't provide all of our platform services to the user. These ISVs probably desire some way to automatically advertise updates to their users. Apt and our update-notifier are perfect candidates for this.

Even fewer third parties actually provide apt repositories for their packages. These packages are generally the best, but still not perfect. The user is required to copy and paste an obscure looking URL, or multiple, into either a text editor, or Synaptic's Add Repository interface. The same has to be done for the apt-key which allows access to the repository. These manual steps are more complicated than it needs to be.

ThirdPartyApt seeks to bridge the delivery of an apt repository link to a user's desktop.

ThirdPartyApt on its own does not force ISVs to use it. If the framework exists, and is well supported by Ubuntu, then perhaps ISVs will consider automatic updates and integration a service to their users. If not, the situation doesn't change, but there's no harm done.

Scope and Use Cases

The main use case is simple. A user should be able to be provided a link to click on by a third party ISV which will prompt the user for permission to install the software. The user should be able to accept and start the installation of that ISVs software. The ISV may provide this link on a Download page, or after an accepted payment transaction as been processed.

The installed software should track updates from the ISV automatically and integrate into the existing infrastructure. Update-notifier should notify the user when the ISVs software is updated.

For Ubuntu specific repositories, ISVs currently must give elaborate instructions for users to setup and enable the repository (eg: [http://www.winehq.org/site/download-deb here] for Wine, or [http://www.google.com/linuxrepositories/apt.html here] for Google).

Design

A file format should be created that encapsulates a number of pieces of information necessary for installing a remote application or set of applications. This file format is called a ".apt" file here.

The APT file is formatted with standard DPkg stanzas. Each stanza contains a repository URL and filters describing what type of platform this URL is suitable for. Consider the example stanza:

Architecture: i386
Distribution: Ubuntu
Codename: breezy
Release: 5.10
Archive:
 http://us.archive.ubuntu.com/ubuntu breezy universe
Install: photoshop
Pin: photoshop libphotoshop0

The first four tags, Architecture, Distribution, Codename and Release specify the specific system that this stanza applies to. Each of these are read and tested against the user's system. If the user's system does not match, the stanza is ignored. This allows a single .apt file to specify delivery of an apt repository for multiple distributions and releases. A seperate stanza could be constructed to apply to Ubuntu and/or Debian, pointing to different archives.

• Architecture: Filters the stanza to the specified apt archive name. Examples would include i386 and/or powerpc.

• Distribution: Filters the stanza to only be applied to systems where the LSB Distributor ID matches. The command lsb_release -is returns this.

• Codename: Filters the stanza to only be applied to systems where the LSB Codename field matches. The command lsb_release -cs returns this.

• Release: Filters the stanza to only be applied to systems where the LSB Release field matches. The command lsb_release -rs returns this.

Archive lists the APT archives to add to the sources.list file.

Install specifies which package or packages to install from the archive, separated by spaces.

Pin describes which packages should be pinned. Basically this means they should override the default Ubuntu stuff.

The resulting file is then signed with the same private key used to sign the Releases file of the archive. Signing of the file is mandatory. A guarentee of a single identity must be available from the .apt file itself all the way to the installing of the packages. The public key required to be added to apt-key is then appended to the end of the .apt file. The first line of the file is then set to "#@application/x-apt 0". This allows mime magic to be used to detect the file type properly. All of this is mandatory, and the client implementation should reject a non-compliant file. The "0" following the mime type is the version number of the Apt file specification that the file conforms to. As breaking changes are made, we increase this number. This number will stay 0 until the specification is actually put into use.

#@application/x-apt 0
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Architecture: i386
Distribution: Ubuntu
Codename: dapper
Install: photoshop
Archive:
 deb http://archive.adobe.com/archive-ubuntu dapper multiverse

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFEPICNRTZ5+YJ1b+YRAvDoAJ9+NBnSPLa32h7J+tbgisWrh58C2QCdHHwv
<etc>
=z7nf
-----END PGP SIGNATURE-----
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

mQGiBD/G9AgRBACZ519LX9cdoyJA+7gmWC+mUsiyPhnmMWu4uOg0M+vb/JPtDdfc
<etc>
=pIE5
-----END PGP PUBLIC KEY BLOCK-----

The above is an example of a final, signed, .apt file.

Method

A program is provided to handle the .apt file type. When this program launches a .apt file, it strips the mime type, verifies the signature, exports the public key, parses all the stanzas and determines which applies to the current environment. It then walks the user through a few simple questions asking if they are sure they want to trust the included public key to install software onto their system. After the user has confirmed all the steps, the public key is added to apt-key and the packages specified by Install are installed.

-- The implementation could remove the repository and key after a successful install if upgrade tracking isn't requested.

-- The implementation could set pins for the repository to only allow the requested package plus dependencies to be installable. This might prevent common errors in third party packaging from overwriting Ubuntu packages. Just a fail safe, not a security feature.

Implementation: ["GAptI"]

Security

The obvious security considerations do not go away. Yes, a malicious software provider could provide a .apt file which adds a trusted key and repository to the user's apt-key and sources.list. Is this a problem? Users want to install third party software. Providers want to provide it. Yes, this software may make system-wide configuration changes. Yes it may REQUIRE root to run. The best we can do is provide a proper interface by which the user can manage these things, and make him as aware as possible of the consequences. Forcing third parties to come up with their own packaging method isn't it, and that's what they do now. Users have to drop to the command line and run a .bin, or a .run file, or whatever the provider provides. These packages don't integrate into the system. They can break things, install software in the wrong location, overwrite existing files. But people use them, now.

In that vein, when the "are you sure" dialog opens, if the key is not currently trusted, the window should very explicitly state that they are about to trust the provider of the software to provide updates. I'd suggest the OK button is disabled for the first 5 seconds, forcing the user to actually wait, perhaps reading the message.

Obviously, a user should not confirm a package installation which he didn't expect to happen: ie a web site opening a .apt file automatically. This presents another interesting trick, by which a spoofed website could open it's own .apt file "quicker" than the exected web site, thus tricking the user into agreeing to the top-most instance of the confirmation dialog. To work around this, I suggest that if two instances of the .apt file handler are opened at the same time, the second should do nothing and the first should fail.

Keep in mind that security with this feature is no worse than the current system, where users blindly copy and paste terminal commands with root permission or give Synaptic instructions.

Related Pages

• ["GAptI"]

• ThirdPartyPackages

• https://wiki.ubuntu.com/aptgetinstallprotocol

Comments

• MatthewPaulThomas: This seems to be a reinvention of [http://autopackage.org/ AutoPackage], but with not so much portability.

• ArwynHainsworth: You mean [http://autopackage.org/ AutoPackage] is a reinvention dpkg/apt-get, right?

• ArwynHainsworth: The proposed idea as it stands has a HUGE security flaw. If the program that handles .apt files can add any generic repository is sees in the file, what is to stop l33tHax0rs & co. from creating a repository with a deb of a slightly patched kernel (or any other core system deb)? He doesn't even have to get the user to install it directly, all he has to do is get the user to add his repository (with a completely harmless and useful deb) and wait for the user to update. Solution: 1) Remove the Archive: tag completely. or 2) Only allow repositories to be added if they are on a whitelist.

• Killerkiwi: Id vote for only allowing the install if the list repo is in the sources.list else just display an error, "You need to enable the following repo to install 'foo'"

• JohnMccabeDansted: If you want to run 3rd party software you pretty much have to trust the third party, unless you constrain the software using something like Plash or SysTrace. I guess a whitelist of third parties might be useful, or a whitelist of public keys owned by people trusted to approve new repositories.

• JerryHaltom: The last comment explains it. There is *no* answer to installing software on a user's PC that does not leave open the possibility for that software to be untrusted. None. It is currently an unsolvable problem. The entire goal is to install software that isn't already in Ubuntu, and thus it can't be considered trusted by Ubuntu. Either you make installing untrusted software impossible, or you ignore the problem. What this seeks to do however is provide an easier way, to do something people will do already.

• TristanWibberley: There needs to be several things happen, and in a particular order, I think:

  1. *All* daemons in Ubuntu repositories or scripts/programs that will configure themselves to be run from a daemon, and all suid/sgid programs should be configured off by default. Programs that are supposed to be run with sudo should have to configure sudo to allow them so they are off by default too.

  2. Do what Killerkiwi said.

  3. Repositories should be configurable with a metadata-trust rating, whereby more trusted metadata in one repository will mark changes in other repositories - this prevents dependencies/conflicts/etc from stopping security updates.
  4. Users should be able to have their own repositories and install roots so they don't have to screw with the main system repos.
  5. Pre/postinstall scripts should be run in a strictly controlled environment where suid/sgid bits and daemons cannot be configured.

• JerryHaltom: I am not trying to solve this problem. It is unsolvable, as stated earlier.

  1. You can't control that. Point 4 makes it impossible.
  2. Killerkiwi said to only allow repositories that are already listed. That completely defeates the point of the specification, which is to ADD NEW REPOSITORIES!
  3. Too complicated and impossible to enforce because of point 4.
  4. I seek to provide a method to distribute software that may require root... such as, for example: VMware. Given that, the install must run as root and must be able to add kernel modules. Given that, anything attempt to enforce security is useless, as it can be bypassed.
  5. Again, point number 4.

• TristanWibberley:

  1. Is easily possible since the Ubuntu repositories are controlled by the Ubuntu devs, and it is required to provide a predictable security boundary for those that will refuse to allow new repositories, but will want to allow packages in existing repositories to be installed via links in a web browser. IMHO a feature like this should not be available in Ubuntu without that first security feature.

  2. The second thing you should do is what Killerkiwi said: "only allowing the install if the list repo is in the sources.list else just display an error", but then once that is done and the implications understood it would be safe to move on

  3. Is possible to enforce for all repos where the repo administrator can be trusted not to provide trojans or to mess with the dpkg status file - but can't be enforced against malicious repository admins.

  4. and,
  5. are not needed to acheive what both you and I want, but they are further steps on the development of this feature.
  I think we disagree on what is an acceptably high barrier to the inadvertant introduction of security holes and an acceptably low barrier to the removal of security holes.

• JerryHaltom: I'm not sure if you're reading my proposal or somebody elses. You keep going on about Ubuntu repositories. You keep going on about not allowing this unless the repository is in sources.list. My proposal is contrary to those two. My proposal is about *automatically* adding lines to sources.list, because that is the entire point, installing software which is not in Ubuntu. My proposal is to avoid having users edit some silly text file on their own. If you don't like the idea of allowing users to install third party software on Ubuntu without hand editing some text file, then you need to present some sort of argument against that, not all this other stuff.

• TristanWibberley: The installation of third party software will normally require that dependencies be met from within the Ubuntu repositories, so management of Ubuntu packages needs to be a part of it. And since installing third party software under the direction of a resource at an URL will have to work on a third party repository that has already been added to sources.list, mere installation of Ubuntu packages from existing repositories is a proper subset of this feature. So I don't think it is either orthogonal, nor extraneous.

  I do like the idea of installing third party software on Ubuntu without hand editing some text files and I am not trying to put the stoppers on it - I had even implemented the basic (insecure) version in less than 20 lines of bash when I suggested it on the lists (just after somebody proposed mere installation from existing sources via a new url scheme, a couple of weeks ago). I am trying to ensure that Ubuntu doesn't earn a reputation for being as insecure as Windows is. This feature is more complicated to get right than it appears.

• JerryHaltom: Sure, but those dependencies will be met from the existing Ubuntu repositories, as you said. ISVs would distribute .apt files which provided specific support for specific Ubuntu releases. For instance, an ISV would provide a .apt file which would expose breezy software to a breezy user, compiled against breezy. Since ubuntu has defined and supported releases, this is feasible, and quite clean. The last point about security, in my honest opinion, is like trying to add 2+3 and get 4. It just doesn't work. You cannot both provide support for users to install software which requires kernel level access (VMware) and yet restrict it to make it safe. I may not be imagintive enough, or something. There are only a few solutions to this problem which I can see: 1) you either restrict users to installing only approved software, which guarentees it won't have an impact 2) you only allow users to install "certain" third party software that doesn't require root access 3) You come up with a system which sets up a virtual machine for every software install. 1. seems like limiting the user's choices, and is currently circumventable and currently circumvented (VMware provides it's own installer) 2. requires a central point by which all ISVs must vet their software, which is practically identical to 1)... either you keep up with all software and updates, or they'll just use their own systems anyways 3) impossible to do in any meaningful way, now and for the forseeable future. In light of all of that, the only remaining way is to provide the user a method to do what they want to do, ensuring that at least the ISV is using an approved system to deliver the software, that we can vet. That system is apt and dpkg. It allows the user to use his existing tools to manage his installs and updates. apt itself is cross paltform and can deliver even RPMs, which makes it even more compelling. Defining a .apt standard file format could be used to deliver software to many different distributions. Yes, it doesn't go as far as autopackage, making the vendors job of testing against distros certainly time consuming. Implementation of the program in charge of this: GAptI would need to have a lot of community agreement as to the wording of the message. The current GAptI implementation I have written does what I think is a reasonable job: It explains to the user what is going on, and forces him to wait 15 seconds before he can hit Install. This in my opinion is a best case option. Certainly I am not against considering furthur ways to bullet proof this process. The job of establishing a relationship with a third party repository is one which the user needs to be walked thru with the upmost care. The entire proposal, however, does rest on the approval of providing a GUI method by which a user can trust a currently untrusted third party repository.

• TristanWibberley: The thing I was most worried about with the dependencies was a third party package depending on a particular version of an Ubuntu package, which later turns out to have an exploitable hole. The application will probably still work with the new version so apt-get and friends should be made to support forced upgrades at the user's option without uninstalling packages in "lower priority metadata" repositories. The problem I see is not in malicious installed software, but thoughtless metadata issues in the third party repository preventing future upgrades of vulnerable packages. The other concern I have is with any method where an url can install a package that's in even an already configured repository. Even though you trust the package and the repository, the fourth party providing the url could be aware of a flaw in a piece of software and get a user to install it from the trusted third party. The logs of the server that provided the aptfile contents (or logs of firefox's new thoughtless "ping" feature) will tell an attacker that the chance of finding an attackable machine at the logged IP address is very high. If packages default to having high privileges and running (eg daemons, or suid/sgid binaries), attacks will be pretty easy, since the user knows the software isn't a trojan and will happily install it without considering this possiblity. The solution is for Ubuntu provided repositories to not do that (I think that should be in the scope of this spec), and maybe to eventually (in another spec) use chroots and restricted mode shells to prevent pre/post install scripts from configuring things like that (and this part is well beyond the scope of this spec but I note it here to show that the problem should be solvable if it is valued). The other things I mentioned earlier are just "future thoughts" (eg homedir installs), but I think the two security issues I mention just above are very real, and very dangerous. The reason I think installation from preconfigured sources should be implemented and introduced before moving on to the full system (which when finished should be a ton more secure than the way people source their software in the Windows world) is to slow the development of mindshare in gdebi where people download arbitrary debs and install them without adequate warnings, or dependency metadata prioritising. But if you think that gdebi will not gain such widespread adoption that new users become ingrained with the "download and run anything" mindset of Windows users then I'm terribly bothered about the two stage approach.

• TristanWibberley: Moving to another topic, could this be given an initial stanza line that can be recognised as a file magic, like "Stanza: Apt" as the very first line of the file to specify that this is a stanza file and the format and interpretation of the stanzas it contains.

• JerryHaltom: Third party packages should be properly developed and supported by those third parties. They should not depend on strict = versions of Ubuntu packages, and should follow the normal rules of packages. If they really don't work with newer versions, they should depend on >= base version | <= base version + 1. Otherwise, they will prevent a security upgrade. This moves into another point, isn't this a possibility that our general package tools need to handle. Update-notifier should make a point of saying "Package N is preventing an upgrade to your system. Your system may be out of date or open to a vulnerbility", or some such english. This is a case which could and probably should be handled now, if it's not already. Your second concern is valid, but again rendered unsolvable by our current circumstances. A third party repository could force an install of a vulnerble Ubuntu package... or it more easily just install it's own software to take over the system. I don't think coding for the longest path attack serves much purpose here. It again rests on whether we want to allow third parties to initiate package installs period. Also, it likely will not be a policy for packages to be in a disabled state by default. This question has been brought up over and over again in both Debian and Ubuntu and has almost always ended with the same conclusion: the package wouldn't have been installed if the user didn't want it running. I can dig out some mailing list archives if you want. I agree with the mime tag. I am searching for some common accepted syntax for doing this as the first line of a file. I think VI has been doing something for awhile. Can we take new conversation to ubuntu-devel@? I have created a thread asking for opinions on this spec there and would like to get more parties involved. Seeing our discussion on ubuntu-devel may get more people involved.

• KillerKiwi re adding to sources.list. it's my understanding that there is all ready a UI for adding a repo and its been promoted to the 'admin menu' in dapper (may be wrong I havnt updated yet ), So I'm not suggesting reverting to editing the file, just an extra step the user must preform between clicking and adding a repo, much like installing extensions for firefox from non white listed sites requires white listing the domain first.

  Also agree with TristanWibberley about file magic Would also be nice to be able to specify more than one package to install (from the stated repo) Lastly a thiredpartyapt script should be created (install.ubuntu.com/?package=firefox or something) linking to every package in the ubuntu repos allowing 1 click install and an easy link for posting in forums and wikis ;).

• JerryHaltom: I implemented some of Tristan's suggestions by removing the PublicKey line and introducing signing of the entire file. This means that a .apt file can only add the apt-key that it was signed with. This is a small requirement, and goes a long way to prevent editing of a .apt file distributed by a third party. If it's edited, it can't be installed with the same key. It's not really preventing any security problems, as a malicious entity could still write his own .apt file, pointing to his own repository, signed with a key whose name exactly matches that of the original key. The user would see no difference.

• JerryHaltom: Tristan, I think I finally grasped what you've been saying. I can't offer a proper solution to it though without bridging the identity gap between the web browser and gapti. My only idea there is a web browser extension which handles .apt links differently, calling gapti with the full path of the source of the file, and only allowing this to happen when the web site itself is HTTPS. The .apt file would also need the expected path encoded into it. This would guarentee at least that the web site the user is viewing, and can verify identity of through the browser's SSL viewer was the distributor of the .apt file and the signer of the repository. This is obviously very hard to accomplish, though.

• RobJCaskey: Perhaps it's time we get AppArmor packaged and add some 'AptArmor' to synaptic, gnome-app-install, & gdebi? We could have policies and associated graphical icons to summarize which permissions a program requires to run and install.

• JerryHaltom: AppArmor doesn't really help with this much. This program could still request full access at install time, and a user could still grant it. The fact is we don't know ahead of time what permissions an ISV requires. Won't prevent against rogue installs.

• Engla: OneClick/apt-plus is yet another implementation of this. They claim it alreay works with ubuntu. http://www.apt-plus.com/

• KillerKiwi: Oneclick/apt-plus is a protocol not a file/mimetype (apt-plus://tuxracer) but does indeed do this. Ethier or is fine with me but I understand some people have issues with using a protocol

• JerryHaltom: OneClick/apt-plus also doesn't cover half of the ground this does. Looks like all it does is install a named package from an existing repository. Not much use for ISVs. Doesn't add new repositories, doesn't deal with keys, etc.

• Engla: About OneClick: It's true that what Jerry says. Still, I just wanted to collect another link and see if it could add anything. Anyway, isn't a web of trust for these keys feasible? Third parties represented as organizations or companies could be trusted by having a web of keys, where Ubuntu would allow/sign (however that works) 3rd parties asking for it. If we manage to have a safe but quick response time on this, it could be easier and more feasible to just sign a distributor's key and let them provide their own packages, rather than merging it into the ubuntu repositories.

• JerryHaltom: Engla, I agree completely, and will implement this. I mentioned it in the Method section in a note. For keys which are not currently signed, the dialog would make the user wait some time period, 15 seconds or something, before it let htem proceed. It would display lots of warning messages and generally be hard to do. For a signed key, it would be easier. Perhaps it would say even "This software is guarenteed authentic by Ubuntu." Or something. Either way, it would have to be optional.

• JerryHaltom: You all should check out GAptI. It is linked from the wiki article. You can download it with bzr and build a .deb from it.

• JohnNilsson: I don't think that messing with the user will be productive. The 15s this will just get people irritated, and will eventually be circumvented by som demanded command line switch, or simply a fork. Just inform the user, visibly, that installing untrusted software will taint the system and void any support. On top of this, the web of trust can be used so that ubuntu uses different kyes for different kind of trust. "Trusted by ubuntu", "Supported by ubuntu" and other classes as needed. In the end it's the ISV that should be botherd by distributing untrusted software, not the user.

• ScottRitchie: I think you should extend the specification to include more metadata. Specifically: a description of the repository, optional metadata to help the OS choose a mirror (if any), and optional instructions for update-manager during a distro upgrade (perhaps where to download a new .apt file in case we need to change feisty to gutsy in the apt line). Also, the apt file should be stored somewhere, like within /etc/apt, so it can be removed easily. All in all, I really like it.

• JerryHaltom: Scott, I agree. At the time I wrote the original spec I had not fully considered update-manager or how it relates to this. Somebody friendly and kind should take a look at update-manager and see how Ubuntu manages dist upgrades. What files do they install in -updates that cause update-manager to offer a new distro choice? As an aside, I'd like to think about some sort of recommendation for hard non-future dependencies. For instance, VMware is NOT compatible with a new kernel. Somebody at VMware has to rebuild kernel modules. Thus, a user should not update to a new distro that contains a new kernel until VMware has built their modules. The same goes for more than the kernel. That or breaking changes in a distro should never be allowed, and I don't think anybody likes this idea. A hard dependency on ubuntu-standard's version might be a good recommendation for this. Prevents the user from upgrading to Gutsy until his third party software is updated, or he makes some decision on what to do.

• Mhall119: Would it be possible to extend the sources.list file to specify a list of packages to look for in a certain repository? That way adding a third-party repository to install package 'foo' will only look for updates to package 'foo' in that repository, and will ignore updates to, say, the Linux kernel from that repository. Basically a package white-list for untrusted repositories.

• JerryHaltom: Mhall119: I think this misses the point. That repository should not be distributing updates to Ubuntu packages. That's very, very bad.

• ScottRitchie: JerryHaltom: No it isn't - the primary use case for Third Party Apt is replacing something like http://www.winehq.org/site/download-deb -- those are updates to the packages within Ubuntu. ThirdPartyApt would be just as useful providing backports as it would additional software.

• ScottRitchie: Mhall119: There is no such thing as an "untrusted repository" - installing a package necessarily grants root level access to the system while running multiple package-defined scripts. There is no security benefit from restricting certain packages from a certain repository. It seems that what you want, however, can already be supplied by apt pinning (namely, preferring one repository over another even if the version number is lower). We may want a graphical interface to manage pinning more carefully.