AppDevUploadProcess

App Developer Upload Process

IMPORTANT: this has not been updated for click packages yet

Please note: This is a work in progress specification and it is not complete. There are parts of the current draft that will change based on feedback and research we are currently doing. This spec's creation is being coordinated by Jono Bacon, David Planella, and Michael Hall.

• If you'd like to contribute please add your feedback or participate in the discussion on the ubuntu-devel mailing list
• For any other queries please contact Jono Bacon <jono AT ubuntu DOT com>.

Problem

Today we face a significant challenge in Ubuntu. If an application author wants to get the latest version of their software in Ubuntu they have two options:

  1. An Ubuntu developer packages the software and delivers it in the development release.

  2. The app developer can submit their apps via MyApps in the stable release for review.

There are various challenges with these two approaches:

  • For (1) app developers cannot get new releases of their software on stable releases. This results in versions of their software in Ubuntu being one or more releases behind the current stable version from upstream (we have seen app developers often not want to recommend the Ubuntu Software Center a place to get their software due to this issue).
  • For (1) this requires knowing an Ubuntu developer who can help you or hoping that an Ubuntu developer cares about your software. This is less empowering for app developers who want to deliver their software directly into Ubuntu.
  • For (2), if the developer is selling their software, they are reasonably well catered: the consumer apps team will take care of them delivering their apps into Ubuntu. For Free Software developers who use approach (2) they must do more work themselves. The Application Review Board (ARB) are there to review these apps but the ARB, as a team of volunteers, is not able to scale with increases in the queue (the recent App Developer Showdown was a good example of this), resulting in long delays for the developer. The ARB frequently gets messages from developers asking if something is wrong when they haven't heard back in 24 hours, and the developers often become discouraged after a delay of a week or so, thinking their work was wasted. A bad first experience gives them no motivation to improve their skills and come back for more.
  • For (2), the developer is often required to make changes to their application and packaging in order to comply with the different packaging and installation requirements for MyApps (such as the /opt requirement).

  • Irrespective of whether the app dev uses (1) or (2), they are relying on other people to do work before they can deploy their application. This is less empowering, and for an emerging platform such as Ubuntu, we need to enable app developers to deliver their apps without such delays.

Goal

Please note: this specification only focused on applications that are uploaded to the Extras extension repository. Thus, this process does not cover developer uploads of newer versions of software in the existing main archive (which includes the main, restricted, universe and multiverse components). We do want to explore and discuss the best way to provide updates for main archive apps, but those are out of scope for the current revision of the spec.

The goal here is to provide a safe, reliable means in which app developers can upload their stable releases directly into Ubuntu.

With this goal in mind we want to resolve the common bottlenecks that we have experienced with the current process for app developers:

  • We should not rely on manual reviews of software before inclusion. Manual reviews have been found to cause a significant bottleneck in the MyApps queue and they won’t scale effectively as we grow and open up Ubuntu to thousands of apps.

  • Requiring app installation into the /opt directory is significantly challenging to app developers and not well supported in Ubuntu; this should be made transparent to the developer.

We also want to enhance the platform experience for both users and developers who are building applications for Ubuntu:

  • Provide a standard, stable and useful platform for developers, and a rapid path to delivering their apps to Ubuntu users
  • Give users a safer access to the growing ecosystem of independent applications being developed for Ubuntu.
  • Keep the user informed and in control over who is providing the software they install, and what access to their system it will have.

Proposed Solution

This section outlines the proposed solution for how app developers upload their apps, and how end users can browse a combination of apps delivered by Ubuntu/Debian developers as well as apps uploaded by app developers.

How App Devs Upload Their Apps

Here is a proposed solution to resolve these problems.

App Dev Upload diagram

  1. App Preparation - the developer can use our tools to generate a package to be included in the Ubuntu Software Center. These tools will likely include Quickly and pkgme to generate the package.

  2. Developer Request - in MyApps the developer creates a request to upload their application through MyApps. The developer will be asked to provide proof that they are either (a) the copyright or trademark owner of the application or (b) a key representative of the team that created it (e.g. a core developer for an Open Source project) or (c) has the endorsement of the project to provide packages to Ubuntu. They will not be granted permissions to any package name currently in the Ubuntu archive. The developer will also be required to agree to terms and conditions that will clearly outline what content is acceptable and unacceptable, and that the software can be removed if questionable, illegal, or infringing content is found. Once approved the developer will be provided with upload access for that specific application to extras.

  3. Security Policy Preparation - the developer then fills in a web form that outlines the resource requirements for the application. This will generate an AppArmor policy for the application and provide its access needs to the Software Center.

  4. App Review - When the developer submits a new version of their application, it will go through a series of automated checks to verify that it conforms to the Extras packaging and security rules. The developer will be responsible for ensuring the package works effectively. If it doesn’t, the ratings and reviews will surely reflect this.

  5. Upload - The developer is now able to upload versions of their package to the Ubuntu Software Center without each version requiring a specific review and thus achieving the goals highlighted in this spec. The developer will not be able to upload other packages, just the one they have access to upload. This upload will be performed by MyApps at the developer's request.

  6. Visibility - When this version of the application appears in the Ubuntu Software Center it will be made clear that the application is provided by the developer, who is solely responsible for it. Users will be able to see a combination of applications provided by Ubuntu/Debian developers (which are supported) as well as developer-uploaded applications such as outlined by this process. See the next section for more details.

How End Users Browse Apps

The goal here is to ensure that Ubuntu users can make an informed choice about what applications they want to install. This informed choice should not only be based upon the features of the application but also how support the application is.

There are two types of application that the Ubuntu Software Center will show:

  1. Officially supported apps - these are the apps in main that are supported as part of our support policy. These are likely to be older version but releases with assured integration and security fix support.

  2. Developer uploaded apps - these are the apps that app devs can upload. These are not supported by Ubuntu developers and the app developer is responsible for any quality and integration considerations.

We would need some design input on the best way of presenting this information.

Implementation

App Dev Upload Diagram (II).png

App Preparation

This step deals with how app authors will prepare their app before submitting it to MyApps and starting the app developer upload process. We want to be able to help and support our app developers to submit their applications in a format that can be easily processed for inclusion in the Ubuntu Software Center.

MyApps will accept one of the following formats (please note the priorities listed that indicate the order in which we want to build this support):

  • Debian source package (high priority) - this works out of the box with Quickly today and would be our recommend method of submitting content. On the server side, pkgme would unpack it to integrate the security profile into a newly-generated source package.

  • Source tarball (medium priority) - in this case app developers do not have to worry about packaging at all, as the package is generated automatically on the server side. Requires work on pkgme to implement an initial set of supported backends. We need to support this format for the long tail of app developers who will be bringing their apps to Ubuntu without any prior packaging experience.

  • PPA link (low priority) - this also works out of the box with Quickly and is simply a variation of the source package format format. This provides a useful means in which app developers can run a PPA for where they publish and test their packages and then simply nominate a a PPA package for release in the Ubuntu Software Center.

At the final stage of application development, when the author feels the application is ready to be published, they will be able to execute a Quickly command to prepare their code in a way that can be processed by MyApps. The command will be:

quickly submitubuntu

Alternately, the author will be able to manually submit their application through the MyApps web interface, selecting the source package files from their computer or directing MyApps to download them from a PPA. The goal is to ultimately create a package on the server side that can be distributed in the Software Centre.

Your app tab, upload controls

Your app tab, upload controls

Your app tab, upload controls, version I (click to zoom)

Your app tab, upload controls, version II (click to zoom)
Erratum: should be "As a source package", with three upload fields: "orig.tar.gz", ".dsc" and ".diff.gz/.debian.tar.gz"

We will continue requiring applications to be installed to /opt/, but will make this transparent to the developer by fully supporting relocatable packages in all of our build tools and runtime systems. Once relocation is supported, the developer will not need to make any changes to their source code or source package in order to target /opt/, they will only need to build the binary by passing a special flag to debuild. When their package is built by MyApps, the same flag will be used to ensure that the resulting binary will be properly installed into /opt/.

When needed for desktop integration, a set of whitelisted locations within /opt/ has been defined where applications will be able to install files. The same or equivalent set of tools will be run on the server side to do the checks.

The whitelist of approved installation locations is:

Whitelisted install location

Notes

/opt/extras.ubuntu.com/${appname}/

The bulk of an application's files will be installed here

/opt/extras.ubuntu.com/share/applications/${appname}.desktop

/opt/extras.ubuntu.com/share/applications/${appname}-*.desktop

/opt/extras.ubuntu.com/share/unity/lenses/${appname}/

Needed for Unity lenses and scopes

/opt/extras.ubuntu.com/share/dbus-1/services/${appname}.service

DBus service files. The file names are in reverse domain naming notation (e.g. com.ubuntu.OneConf.service)

/opt/extras.ubuntu.com/share/dbus-1/services/${appname}-*.service

Same as above

/opt/extras.ubuntu.com/share/glib-2.0/schemas/${appname}.gschema.xml

GSettings schema files. The file names are in reverse domain naming notation (e.g. org.gnome.Evince.gschema.xml)

/opt/extras.ubuntu.com/share/doc/${appname}/

/opt/extras.ubuntu.com/share/man/man{3,8}/${appname}*.*.gz

/opt/extras.ubuntu.com/share/help/<lang>/${appname}/

/opt/extras.ubuntu.com/share/locale/<lang>/LC_MESSAGES/${appname}.mo

/opt/extras.ubuntu.com/share/icons/hicolor/<size>/${appname}.{png,svg}

/opt/extras.ubuntu.com/share/python/runtime.d/${appname}.rtupdate

/opt/extras.ubuntu.com/share/apport/crashdb.conf.d/${appname}-crashdb.conf

/opt/extras.ubuntu.com/share/accounts/${appname}/

Apps will only be allowed to install files in these locationsPermalink

On the client side, Quickly and pkgme will integrate a custom Lintian profile to do the checks for whitelisted location (and any additional ones required) and will output warnings or errors to indicate whether the local app might need to be fixed before being submitted.

While this specification is concerned with apps created by the tools that Ubuntu provides (which in this case they are Quickly), app developers that write their apps outside Quickly should be able to get their software prepared in the same form Quickly does.

A further step towards better process automation would be to provide an API to enable Quickly to directly send the application to MyApps. While this would streamline the process, it is considered non-essential for the current revision of the spec.

Security Profile Testing

In order to iteratively test their application under an AppArmor security profile, developers will be provided with a tool that will generate and install a working profile built from a simple config file. This tool will be available as a stand-alone commandline program, as well as being integrated into Quickly. It will facilitate the creation and installation of a local AppArmor profile, as well as running the application under this profile to test it's functionality inside of the sandbox.

The tool will read a list of requested abstractions from a simple text file (INI, or other format), and apply those on top of the Ubuntu Application profile template using aa_easyprof. Developers using Quickly should be able to add and remove abstractions from this file using Quickly commands. Developers not using Quickly would still be able to edit the file by hand. The config file should be included as part of the source package submission so that it can be used by MyApps to pre-populate the security profile form for the developer.

Namespace conflict checking

Applications installed in a system may come from diffferent sources, that is, from different archives: mainly Main (including backports) and Extras (where apps concerned with this spec reside).

There is currently no infrastructure in place to provide a connection between the archives in terms of checking that their packages or the contents of those packages don't clash. This does not only apply to independent archives, but also to repositories within a self-contained archive. Also worth noticing is that this is not an issue introduced by this spec, but an already existing one.

In summary, there is a small but real possibility for packaged applications submitted through the app developer process to essentially (a) install files in the same location as another package coming from the Main archive or (b) to use a package name already in use in the Main archive.

Both cases would create a file or package conflict situation which the packaging tools would not be able to fix. Note that the potential scope for (a) would only affect the whitelisted set of installation locations for third party apps. Case (b) is further addressed in the Package preparation section by prepending the extras_ prefix to Extras packages to avoid collisions.

Conflict checker

A solution to this problem would be to use an automated conflict checker service. The idea is that the contents of all archives are read into a database and then a comparison is made upon any submission to MyApps. MyApps would query the conflict checker and would inform the submitter of any file conflicts that need to be fixed before upload.

The idea is that the conflict checker has data about all archives, but only runs for Extras before each upload: to inform the uploader about conflicts and prevent uploads until they have been resolved. If this proves to be a useful feature, it could theoretically be extended to all of the archives in a future iteration, so that not only Extras apps would benefit from conflict checking, but also all software in the Main archive.

For each new distro series, Extras will be open at Beta time. Just before opening the archive, the conflict checker will also be enabled for that release and will continue running for the lifecycle of the distro series. For newly submitted applications, conflicts will be detected and Extras uploads will be blocked until the conflicts have been resolved. For existing Extras applications which may present a conflict with a Main application, the Main application will take precedence in terms of file namespace ownership (as outlined below). In this case, an e-mail notification will be sent to the app developer to fix the conflict and reupload their app, and the Extras package will be marked with a Conflicts: field against the Main package. If the app developer does not act upon it, their app will be uninstalled when users do the upload to the stable release.

The conflict checker should be aware of and able to handle:

In terms namespace ownership, the proposal is to use the following priorities:

  1. Main
  2. Extras
  3. Backports

That means the archives that are higher on the list get to own the namespace when there is a conflict with an archive with a lower priority, and the uploaders on the lower priority archive need to resolve the conflict.

The current conflict checker (currently not in use) and the packages.ubuntu.com site are two examples of existing infrastructure using a similar approach.

Potential cases
  • John uploads "Wikitron" (/usr/bin/wikitron) to Extras via MyApps. Afterwards, Jane uploads "DesktopWiki" (also /usr/bin/wikitron) to Backports

    • Solution: conflict checker warns Jane about the conflict via an e-mail. She must fix the conflict before a new upload is accepted.
  • George uploads "Lightread" (/usr/bin/lightread) to Extras through MyApps. An existing package called "LightReader" (/usr/bin/lightread) exists in the main archive already

    • Solution: conflict checker warns George about the conflict via an e-mail and on the MyApps web UI. He must fix the conflict before a new upload is accepted.

  • Herbert uploads "bird-game" (/usr/bin/bird-game) to Extras via MyApps. An existing package called "animal-games" (with /usr/games/bird-game) exists in the main archive already. Calls to "bird-game" will match the one in /usr/bin/ before the one in /usr/games/

    • Solution: extend conflict checking to match any file under a directory in $PATH to any other file in a directory under $PATH. Possibly need to do the same for libraries.
  • Oswald uploads package "notes" to Extras via MyApps. Afterwards, Persephone uploads an unrelated package, also called "notes" to Debian. During the Ubuntu+1 package sync, Persephone's "notes" package will be pulled into Universe.

    • Solution: Prefix Extras packages with extras_ to avoid package name conflicts

      • If the package gets moved into Debian or Ubuntu's archives, we would need to transition the users who have the extras_ package installed over to the non-prefixed package.

Developer Request

Applying for access

To ensure that we are giving upload access only to the original author or a proper representative of the upstream project, we will require that person to request upload access for their application. The author or representative must first create an account and user profile in the MyApps portal as it currently exists.

Once their profile is created, they will need to be able to request upload access for a package, providing details about their association with the upstream project. If the submitter is not the owner or representative of the project, they will be required to provide a URL to a webpage, blog post or mailing list archive showing that the owner or representative of the project is endorsing their effort upload the application to the Ubuntu Software Center.

A link will be presented on their user profile page that will point to a new page containing a form for the developer to fill out. The details required by the form should be sufficient for a review board to determine whether or not the requesting user meets the requirements to represent that project. The review board will be responsible for developing the detail requirements, but it should include the following:

  1. The developer's first and last name (e.g. Joe Smith).

  2. A contact email for the developer (e.g. joe.smith AT email DOT com)

  3. The name of the package(s) that the user wishes to upload (e.g. accomplishments-viewer). This could be more than one package if required.

  4. If the developer is the sole owner of the application, they need to tick a box to verify that they are copyright owner, representative, or endorsed package provider for the project. This would be in the form of a textbox in which the user prepares a summary of their involvement in the upstream project with content such as the following recognized:
    • Project information page listing the user as the creator or current owner of the project
    • Links to commit logs from the user.
    • Active participation in the project strategy and discussion (such as mailing list posts).
    • Testimonials from other members of the project.
    • Links to an specific endorsement for uploading the application to the Ubuntu Software Center

App details page, Uploaders tab

Your app tab, with upload request

App details page, Uploaders tab (click to zoom)

Your app tab, with upload request (click to zoom)

Upload rights request page

Upload rights request page (click to zoom)

In addition, the form should include an agreement text stating that the requesting user assumes responsibility for the maintenance and content of the package they are requesting access to upload. This information will be saved to the MyApps database. More than one of these requests may be made by the same MyApps user, if they wish to maintain more than one package in the Extras archive.

Developer Verification

There will be two methods to get verified and thus be able to submit apps to MyApps:

  • Fast track: the developer will use Ubuntu Pay to provide a one-off nominal fee of $1 through the supported payment methods. This is the primary method of verification.

  • Physical verification: as an alternative, developers can be verified at Ubuntu events such as the Ubuntu Developer Summit (UDS), release parties, jams, etc. A set of individuals will be part of a team or review board who will be able to accept developers applications into MyApps directly.

The verification step will be fully automated, only requiring manual intervention in case of a conflict, revokation or physical verification.

After a developer has been identified, their information will be available on a page on MyApps, where only members of the review board can review the details submitted by the developer if necessary. This page will also provide a form for leaving comments on the developer's request. These comments will be visible to the board members but not the developer making the request. This page will give the board members the ability to edit the request in case it needs to be revoked.

MyApps will perform an automated check to ensure that the package name being requested is not already taken.

Revoking Access

Given the number of developers who are expected to to start using this new upload process, it is likely that there will be times when, for one reason or another, we will need to revoke their access. This will occur when the terms of conditions of the service have been breached. Some potential causes for access revocation include, but are not limited to:

  1. Attempting to upload malicious content.
  2. Uploading content in violation of the Extension Repository Policy

  3. Uploading content in violation of copyright, trademark, or other legal restrictions.
  4. The owner or representative of a project objects to their upload access.

Revoking access is something that will only occur in situations in which the terms and conditions have been breached. Access will be revoked by changing the status of the user's upload request record from "Approved" to "Revoked" in MyApps.

Access will not be revoked if an application is considered poor quality. The ratings and reviews system is a useful approach in which our users can communicate their dissatisfaction (or satisfaction) with an application and help other users to make a judgement call of whether to install it or not.

In the case of content in violation of the Extras archive policies, users can submit an abuse complaint that will be reviewed by the review board indicating the application and what content they feel is objectionable. The review board will then contact the developer to determine whether the application can be resolved so that it complies with the archive policies. If it can not, the review board will meet to determine whether the application should be removed from the archive and the developer's upload access for that application will be revoked.

In the case of a legal violation, such as copyright or trademark, either the review board or Canonical will receive the complaint from the copyright or trademark holder. In this case the review board will immediately remove the package from the archive and suspend the developer's upload access until the issue has been resolved. They will then contact the developer with information about the complaint, and determine whether the application can be fixed so that it is no longer in violation. If it can not, the review board will permanently revoke the developer's upload access for that application.

The Ubuntu Software Center, when viewing a package from the Extras archive, will display an additional link for reporting abuse by the package. The link will direct the reporter to a page on MyApps with a form for detailing the abuse being reported, and optional contact information about the reporter. The form should include a radio option listing the 4 reasons for revoking access listed above, or an optional "other" reason that the reporter must specify.

Revokation will be done by the review board through the identification page outlined in the section above.

Security Policy Preparation

Security Profile Creation Tab

Security profile creation tab in MyApps (click to zoom).

Once an application is installed, AppArmor will provide a security sandbox to protect the user from malicious or simply broken code. To accomplish this, each application will need a properly constructed, and minimally permissive profile to restrict what the application can access on the local system. Producing such a profile, and more importantly checking it for correctness, is currently a very manual process.

AppArmor Creation

In order to reduce the complexity of the AppArmor profiles used by applications submitted to Extras, the author is asked to fill out a form describing the access needs of their application. The options presented will correspond to the following pre-defined Policy Groups used by the AppArmor Easy Profile (aa-easyprof) script.

Default Permissions

Permissions are handled by using templates and then declaring additional permissions in the form of policy groups. For 13.10 the templates are:

Note: application confinement for Unity scopes will not be supported in 13.10.

The following AppArmor policy abstractions will be given by default (ie, when using the 'ubuntu-sdk' template:

  • Base - Basic application access

  • Freedesktop.org - Give read access to standard system resources such as icons, menu and application descriptors

  • Fonts - Gives access to fonts on the system

  • Ibus - Gives access to ibus on the system

In addition, apps will by default have access to certain things:

  • Unity Integration - Gives access only to the DBus services needed to run properly under Unity (specific APIs not listed since they are subject to change):

    • org.freedesktop.DBus methods to find and query the session bus

    • org.freedesktop.DBus methods to find and query the system bus

    • HUD
    • url-dispatcher
    • download manager
    • BottomBarVisibilityCommunicator

    • On Screen Keyboard (OSK)
    • accessibility bus

Some access is explictly denied, eg:

  • /com/canonical/[Uu]nity/[Dd]ebug**
  • org.gnome.GConf.Server DBus API
  • org.freedesktop.NetworkManager DBus API

  • org.ofono DBus API
  • access to /tmp (TMPDIR is set to an application specific directory)
  • various writes in $HOME
  • plugins in webviews

These policies will provide the minimal amount of access needed for client applications, including the ability to read and write the application's own user data and settings, interact with Unity services, and use shared platform resources such as system libraries, fonts and standard icons.

Additional Permissions

Any additional permissions chosen (other than those under Default above) are declared by the developer in the form of policy groups.

The author may also choose from various policy groups. In 13.10 the common set (ie, those available to all applications) consists of:

  • accounts - Can use Online Accounts
  • audio - Can play audio
  • camera - Can access the camera(s)
  • connectivity - Can access coarse network connectivity information
  • content_exchange - Can request/import data from other applications
  • content_exchange_source - Can provide/export data to other applications
  • location - Can access Location
  • microphone - Can access the microphone
  • networking - Can access the network
  • sensors - Can access the sensors
  • video - Can play video

A reserved set of policy groups is also provided for apps that need special confinement-- typically Canonical-supported apps like the gallery, media player and music player. Eg:

  • picture_files - Can read and write to picture files. This policy group is
  • picture_files_read - Can read all picture files. This policy group is reserved
  • music_files - Can read and write to music files. This policy group is
  • music_files_read - Can read all music files. This policy group is reserved
  • video_files - Can read and write to video files. This policy group is
  • video_files_read - Can read all video files. This policy group is reserved

It is also possible for apps to declare the use of additional abstractions, read paths and write paths. Developers are discouraged from using this as it will redflag the application for manual review which will slow down the review process significantly. App developers should either find another way to work within the existing permission set, or file a bug to see if it can be added to the policy.

AppArmor Integration

As mentioned, so long as the developer uses the predefined templates (excepting 'unconfined') and policy groups, the app will not require manual review.

Once the security profile is defined, the aa-easyprof program is used to generate the actual profile using the template, the list of policy groups defined by the author and a list of template variable variables defined in the packaging. The base template will contain the default set of AppArmor policies that will be needed by most client applications and which won't allow the application to interfere with any other application on the system.

By limiting the number of AppArmor options to a set of common access needs defined by aa-easyprof Policy Groups, the creation of the profile can be fully automated.

App Sandboxing

Because the security and stability of the user's system will depend more on the inability of running applications to cause problems than on a manual review of the application identifying them, a robust sandbox will be needed to prevent any un-granted access.

AppArmor is the preferred way of providing mandatory access controls (MAC) on Ubuntu, and it will also serve as the basis for most of the application sandboxing that will be applied to the application. AppArmor will provide the following necessary restrictions:

  • Limit filesystem read and write access to a pre-defined whitelist of files and directories
  • Restrict an application's ability to access the network
  • Limit what network destinations an application can request
  • Limit what DBus service the application can call
  • Prevent an application from listening in on other applications' DBus communication
  • Limit what external programs an application is able to call
  • Limit what signals an application is able to use and where to send them
  • Force any external programs to run under the same restrictions as the calling application
  • Display manager/server (requires kernel and userspace support from AppArmor as well as support from the display manager/server)

    • Prevent an application from listening in on another application's keyboard/mouse input and output
    • Restrict an application's ability to perform screenshots outside of its window
    • Restrict an application's ability to access the clipboard
    • Restrict an application's ability to perform drag and drop
    • Limit/Support 3D in some manner

In addition to AppArmor, sandboxing will be required by these other parts of Ubuntu as well (list subject to change):

  • DConf/GSettings
    • Prevent an application from writing to system-wide or session-wide settings
    • Prevent an application from reading or writing to another application's settings
    • Prevent Gtk from loading and executing additional modules based on an application's settings
  • GNOME Keyring
    • Prevent unauthorized applications from obtaining credentials
  • Ubuntu Online Accounts
    • Prevent unauthorized applications from obtaining credentials
  • Location service

On the converged device, it is expected that toolkits may need to be patched to gracefully handle sandbox denials. Eg, prior art showed that some applications will crash when faced with an X denials so we'll need to be careful about any sort of display manager/server denials in our toolkits.

Some parts of the environment will need to be adjusted for confined applications to work correctly within a confined environment (eg, setting XDG_* base directories to default values, TMPDIR or other environment variables). The SDK and application launchers will make this transparent to the developer.

Helpers

Because application authors will only be able to choose from a predefined set of access policies, some services may use a proxy service that will allow the application to extend its access beyond the confines if its AppArmor profile with the user's approval. This service will broker requests for additional access between the application and the user, and either dynamically expand the application's sandbox, put a copy of the content into the application's sandbox, or allow the application to read and write through the proxy service itself.

When the application needs to open a file outside of its sandbox, it will call the content-hub service running outside of the sandbox. The content-hub service will then prompt the user to select the file(s) using a file chooser, gallery app, etc. Only after the user makes and approves of the selection will the application be given the ability to access it.

Helper services may in the future provide content guarantees by checking and/or converting the format of what is being saved. This would allow the user to approve writing files of a certain type, such as an png image, to a location outside of the AppArmor sandbox. When the application attempts to save data to this location, the service will verify that the content matches what has been approved, and either convert it (if possible) or reject it.

If a request is made to the helper service for a file or resources that the application already has access to, it should return that resource immediately, without needing additional approval from the user, so that the developer doesn't need to use separate APIs for accessing files inside and outside of their application's sandbox. In order to encourage the use of calls to these helper services, our documentation and tutorials should recommend the use of these APIs for all file access needs, and they should be included in and used by the Quickly templates we provide.

App Review

Submission complete, package errors already found

Submission complete, package errors already found (click to zoom)

Package errors shown on standalone page

Package errors shown on standalone page (click to zoom)

As part of this process, the successful applicants will be given upload rights to their software. This will enable them to upload their packages without any further review, given that they conform to the rules set out in the App Preparation section. In addition to those rules, MyApps will perform a series of automated checks against the submitted source package to further ensure the safety and stability to a user's system.

If an app does not pass the automated checks in MyApps, it will remain in Draft status and the app author won't be able to submit it until any warnings or errors have been addressed. This will require tools like Lintian to be run on the submitted source package, which is a process that can take in the order of minutes to be finished. Thus there should be a mechanism to asynchronously show any status messages in the MyApps UI and to send a notification e-mail to the uploader. The checks can either be triggered when pressing the 'Submit' button at the end of the process, or already when the source package is uploaded.

Package preparation

MyApps will unpack the submitted Debian source package to make a small number of changes to it.

First it will add the generated AppArmor profile to the source package. By integrating the profile in MyApps, rather than having the developer do it, we both relieve the developer of an extra step, and assure ourselves and our users that the profile is properly setup.

Then it will prefix the package name with extras_ to prevent namespace conflicts. Once again, doing this on MyApps relieves the developer of an additional step that they won't necessarily care about, and helps us avoid potential name conflicts with new packages that may come in to Ubuntu or from Debian.

Finally it will rebuild the source package, signing it with the MyApps GPG key. The source package will then be uploaded to the staging PPA to be built into a binary package. A link to the resulting binary package will then be displayed on the submission page in MyApps, which the developer can use to install and test their application running sandboxed under the newly created AppArmor profile.

DBus Service Restrictions

If a submitted application will be installing a DBus session bus service, a further check will be performed to prevent service name conflicts. To accomplish this, a complete list of service names must be maintained and used to verify any new service names.

To create the list of all service names, a one time scan of the archives will be performed at the point in each development cycle where the Extras archive is opened for the new release. This scan will perform the following:

  1. Find packages with a .service file

  2. Download the package
  3. Extract any .service files from the package

  4. Parse the DBus service name from the .service file

  5. Append both the source package name and service name to the list

Whenever a new package is submitted to MyApps, it will be scanned for the presence of a .service file. If one is found, it will be extracted from the package, have its DBus service name parsed and checked against the list. If no match is found, the new package and service name will be appended to the list, and the submission will be allowed to proceed. If a match is found, the submission will be rejected. MyApps will display this conflict on a review page for the developer, showing both the conflicting service name and any existing packages that are using it.

In addition, a blacklist of DBus service name prefixes will be checked. Services using one of the following will not be allowed:

Blacklisted DBus service

Notes

com.canonical.*

net.launchpad.*

org.ayatana.*

org.gnome.*

org.gtk.*

org.freedesktop.*

Apps will not be allowed to install DBus services in these namespacesPermalink

Packaging Restrictions

The Security Team raised the concern that packaging dependencies should also be checked to ensure they cannot introduce security regressions or vulnerabilities. These will only deal with dependencies and not other packaging fields or aspects. This should be an automated check that in practical terms will be coded in a Lintian profile to be run on the server and that will reject the submission if the checks raise a warning. MyApps will display this warning on a review page for the developer.

The current list of rules to check for in the new Lintian profile is:

  • Only reject Depends relationships with = or <= in the Debian control fields of the package

  • .desktop files should not specify an interpreter for the Exec (ie, 'Exec=/bin/sh foo', 'Exec=/usr/bin/python /opt/foo/bar'). In other words, what is in the Exec line in the .desktop file must match the binary specified n the AppArmor profile. A simple but likely good enough check is to ensure that the executable starts with /opt/<namespace> (eg Exec=/opt/foo/...)

Package Updates

Developers will be able to submit new versions of their application any time after their first publication. Whenever a new version of their package is submitted through MyApps, it will undergo all of the above automated tests and modifications as their first submission. If the new version of the application fails any of the above requirements, that version will be rejected and the previously accepted version will remain.

Upload

Once all the app itself, its metadata and security profile have been integrated into a source package, the server will initiate the process of uploading, building and publishing a distributable binary package.

pkgme will be the piece of infrastructure that will do most of the work at this this stage. The main goal is again automation and not having to rely on human intervetion for the upload step. Following this premise, the steps will be:

  1. Pkgme uploads the final source package containing the AppArmor profile to the ARB PPA

  2. The ARB PPA is synced automatically (hourly) to extras.ubuntu.com
  3. The final binary package appears on extras.ubuntu.com and is distributable at that point

Extras Archive

The Extras archive is central to providing an improved process for uploading applications to the Ubuntu Software Center. This archive will combine the features of both the distro archives and Launchpad PPAs in a way that provides both developers and users with reliable access to the latest versions of applications.

Rules Changes

To reduce the time and effort required to get an application into the Extras archive, the need for manual review of an application's source code will be removed. Instead, a proper application sandbox and associated AppArmor profile will prevent both accidental and intentional abuse of a user's system. A manual review will not be needed to verify the security profile, because the profile will be generated by MyApps using a list of pre-approved policy options.

New Release Process

When a new development release of Ubuntu is started, the Extras archive will not be immediately opened. Instead, to prevent excessive error reports from upstream applications during the development cycle, the Extras archives for the development release will be opened at the Beta milestone. Opening Extras at this point in the development cycle will allow enough time for upstream authors and projects to make any necessary changes to their submission prior to the final release.

Once opened, the binary packages in the archive from the previous release will be copied into the new release. Forward-copying the binary requires no additional action on the part of the upstream nor any new review by the review board. Packages will not be re-built on the development release of Ubuntu, it will be left to the upstream author or project to ensure that their packages remain buildable on the new release.

From the experience gathered on paid-for applications on the Commercial queue of MyApps, the forward-copying process generally works successfully for 95% of the apps. Those apps for which the copying process does not work are usually affected by either multiarch issues or named, numbered libraries they depend on.

Visibility

Ubuntu has a long history of being stable and secure, a reputation that was made possibly by careful and thorough reviews by both Ubuntu and Debian developers. When changing the source and method of delivering applications to users, it will be important to keep the user informed about what they are installing, who is providing it, and the possible side affects it may cause.

Indicate Source

There are several places where an Ubuntu user will see the availability of an application. Before selecting an application to install or version to upgrade, the user will be informed, as much as possible, about the provider of the packages and the level of support provided.

The Web Catalog (apps.ubuntu.com) will also be used to identify the provider of the different available versions of an application. For versions of an application in the distro archives, the web catalog will indicate that the version is provided by the Ubuntu project. For applications in the Extras archive, the web catalog will indicate that the software is provided by the upstream author or project.

When an application crash is detected, and if the application was installed from either the distro archives or the Extras archive, Apport will display a dialog asking the user if they would like to report the crash. If the installed application is from the distro archives, Apport will indicate that the error was from an Ubuntu provided package. If the installed application is from the Extras archive, Apport will indicate that the error was from a package provided by the upstream author or project. In addition, when the version of the application is provided by the upstream author or project, the user will be warned that the crash details being collected will be seen by the upstream author or project if submitted.

Indicate Access

In addition informing the user about the source of an application, they will need to be informed about the level of access being given to the application in its AppArmor profile.

MyApps will provide an API with information about the security policy options for each application published to the Extras archive. This list will be limited to the optional, non-runtime security access options from the 3rd list in the AppArmor Creation section above.

The Ubuntu Software Center will use the MyApps API to present the user with a list of policies being requested by an application they wish to install. If the application is requesting one or more of these options, Software Center will prompt the user to accept them before the application is installed.

If the user has defined a custom security profile for their system, Software Center will check to see whether an application from Extras is requesting access to one or more resources that has been denied in the user's profile. If so, Software Center will inform the user that the application violates their custom profile, and provide an option to make an exception for the application in question in order to allow its installation.

The Update Manager will also use the MyApps API to see if a newer version of a package is going to access one or more resources that the currently installed version did not. When this happens, Update Manager will de-select and disable the package from its upgrade list, and direct the user (with a link or button) to the Software Center so that they can view and approve the new security policy options before upgrading the package.

Work Items

The following tables list the high-level work items that are required to implement this specification.

App Preparation

Work Item

Assignee

Status

Priority

Package Creation

Make Pkgme package Python apps

INPROGRESS

High

Make Pkgme package C apps

TODO

High

Make Pkgme package C++ apps

TODO

High

Make Pkgme package C#/Mono apps

TODO

Low

Make Pkgme package Vala apps

TODO

Low

Make Pkgme package Java apps

TODO

Medium

Make Pkgme package Flash apps

TODO

Low

Make Quickly use correct version number scheme

TODO

Low

Make Quickly produce a correct copyright file from AUTHORS file

TODO

Low

Make Quickly produce a clean changelog on submitubuntu (only one entry)

TODO

Low

Create a python program to create AppArmor profiles from config files

TODO

High

Add Quickly commands to manage the AppArmor config file and generate a profile

TODO

High

Package Verification

Create a new Lintian profile for checking Extras packages

DONE

High

Make Lintian check for the correct version number scheme

DONE

Low

Make Lintian check that the changelog has only one entry

DONE

Low

Make Lintian check the proper copyright file format

DONE

Low

Make Lintian check for = or <= in Depends

INPROGRESS

High

Make Quickly use the new Lintian profile

TODO

High

Make Pkgme use the new Lintian profile

TODO

Low

Package Isolation

Define a whitelist of directories where an app can install files (using regex matches)

TODO

High

Make Quickly install only to whitelisted directories

TODO

High

Make Pkgme install only to whitelisted directories

TODO

Low

Make Lintian check for files installing outside of the whitelisted directories

INPROGRESS

High

Make Lintian check for the existance of maintainer scripts

DONE

High

/opt/ Build Relocation

Work Item

Assignee

Status

Priority

Build system relocation support

Add relocation support to autotools

Unknown

High

Add relocation support to cmake

Unknown

High

Add relocation support to scons

Unknown

High

Add relocation support to setup.py

Unknown

High

debhelper relocation support

debuild flag to target /opt/

Unknown

High

dh_installdocs

Unknown

High

dh_python .rtupdate files

Unknown

High

Lintian needs to accept files in /opt/

DONE

High

Lintian needs to warn on files in /usr/

DONE

High

change Exec and Icon locations in .desktop files

Unknown

High

change Exec locations in .service files

Unknown

High

Discovery from /opt/

Quickly get_file helper functions

Done

High

Unity Dash .lens and .scope files

TODO

High

DBus .service definitions

Unknown

High

AppArmor profiles

Unknown

High

XDG .desktop files

Unknown

High

Apport crash.db files

Unknown

High

Compile glib .schema files

Unknown

High

Online Accounts

Done

Medium

docs/man/info files

Unknown

Medium

Yelp help docs

Unknown

Medium

PolicyKit actions

Unknown

Medium

Mime type definitions

Unknown

Low

Thumbnailers

Unknown

Low

Theme lookup of icon by name

Unknown

Low

Absolute paths to /opt/

Gtk use of Icon/pixmap

Unknown

High

Qt use of Icon/pixmap

Unknown

High

GtkBuilder and python gettext translations

TODO

High

GtkResource files

Unknown

High

GIR files

Unknown

High

Unity Launcher .desktop

Unknown

High

App Indicators icons

Unknown

High

Message Menu .desktop

Unknown

High

Sound Menu .desktop and images

Unknown

High

Fonts

Unknown

Medium

ICC color profiles

Unknown

Low

Developer Request

Work Item

Assignee

Status

Priority

Applying for access

Add data models in MyApps for upload requests, reviews and approvals

TODO

High

Add a form in MyApps where a developer can request upload rights for an app in the Software Center

TODO

High

Add a page to MyApps where a developer can view the status of their request, leave and respond to comments

TODO

Medium

Add an agreement page to MyApps where the developer can record their acceptance of the rules for uploading to Extras

TODO

High

Ask the legal team to check if the terms of service need any changes as a result of the new process

TODO

High

Add a page or pages where members of the review boards can review, comment on, and approve or reject upload requests

TODO

High

Developer Verification

Define criteria a developer must meet in order verify their identity

TODO

High

Define a process of verifying the identity criteria, including which review board or group will be responsible for doing it

TODO

High

Revoking Access

Define the reasons why a developer's upload access may be revoked due to problems with the stability of their packages

TODO

High

Define the reasons why a developer's upload access may be revoked due to problems with the security of their packages

TODO

High

Define the reasons why a developer's upload access may be revoked due to problems with the content of their packages

TODO

High

Define a method for reporting problems with the stability, security or content of a package in Extras

TODO

Medium

Define a process for reviewing whether a reported application should result in the revocation of a developer's upload access

TODO

High

Provide a form on MyApps for reporting abuse or ToS violations

TODO

High

Add a link in the Software Center for Extras package pointing to the MyApps abuse form

TODO

High

Security Policy Preparation

Work Item

Assignee

Status

Priority

AppArmor Creation

Add a form where the developer can select AppArmor policy groups

DONE

High

Add an API for reviews to verify the security policy options being requested by an application

DONE

High

AppArmor Integration

Pre-populate the AppArmor security manifest

DONE

High

Automatically generate an AppArmor profile using aa-easyprof, that enables access only to the requested resources

DONE

High

Make the AppArmor profile testable by the developer

DONE

High

Make sure the packaging ships a security manifest in the uploaded package

DONE

High

Make the new binary package available for download (on MyApps) by the developer

DONE

Medium

App Sandboxing

Restrict an application's access to files and directories to pre-approved locations

Done

High

Restrict an application's access to the network

Done

High

Restrict an application's access to certain network destinations

POSTPONED

Low

Restrict DBus communication so an app can't see another program's data

DONE

High

Restrict an application's access to DBus services to pre-approved services

DONE

High

Restrict an application's ability to send signals

INPROGRESS

High

Restrict display manager/server communication so an app can't see another program's data

DONE

High1

Restrict GSettings/dconf access (blocked by default)

DONE

High

Restrict GSettings/dconf access so an app can't see or change another program's settings

POSTPONED

Medium2

Restrict which external programs an application can call to pre-approved programs

DONE

High

Restrict an application's access to the GConf values for another application (blocked by default)

DONE

Low

Restrict an application's manipulating of the environment when executing programs

INPROGRESS

High

Restrict an application's access to gnome-keyring (blocked by default)

DONE

High

Restrict an application's access to specific gnome-keyring credentials

POSTPONED

Medium2

Restrict an application's access to ubuntu online accounts

DONE

High

Restrict an application's access to specific ubuntu online accounts credentials

DONE

Medium

If needed, restrict an application's ability to load Gtk modules via specially crafted display manager/server settings (eg Xsettings or similar)

DONE

Low1

Run externally called programs under the calling application's restrictions

DONE

High

Helpers

Provide a "helper" application for giving a restricted app access to un-approved files with the user's consent (content-hub)

DONE

High

  1. handled by Mir on Ubuntu Touch. X mediation (ie for Ubuntu Desktop) not scheduled for 13.10
  2. not required for SDK apps in 13.10

App Review

Work Item

Assignee

Status

Priority

Package submission

Update MyApps to allow uploading an initial Debian source package (verifying the package)

TODO

High

Update MyApps to allow specifying an initial package from a PPA (verifying the PPA and the package)

TODO

High

Package checks

Make MyApps check for install file conflicts on newly uploaded packages

TODO

High

Make MyApps check for files installing outside of the whitelisted directories

TODO

High

Make MyApps check for the existance of maintainer scripts

TODO

High

Make MyApps check for the upper-limits on package dependency (= or >= in Depends)

TODO

High

Make MyApps check for conflicting DBus service names in .service files

TODO

High

Make MyApps check for blacklisted DBus service name prefixes

TODO

High

Upload

Work Item

Assignee

Status

Priority

Extras Archive

Update MyApps to include a data model for the final package to be uploaded (separate from the initial package)

TODO

High

Add a page to MyApps where the developer can manage their uploadable packages

TODO

Medium

Give MyApps access to upload packages to the Extras archive

TODO

High

Add a page to MyApps allowing the developer to initiate the upload to Extras

TODO

High

Add a page on MyApps where the developer can monitor the status of their uploading/uploaded packages

TODO

Medium

Add a page to MyApps showing the upload history of a given package

TODO

Low

Add a mechanism to MyApps that would allow a developer to remove their package from Extras

TODO

Low

Extras Archive

Work Item

Assignee

Status

Priority

Rules changes

Add page to developer.ubuntu.com explaining the new submission process and requirements

TODO

High

Remove ARB wiki pages including submission requirements

TODO

High

Add Extras to /etc/apport/native-origins.d/

TODO

Medium

Build debug packages for compiled apps in Extras

TODO

Low

New release process

Add opening Extras at the FeatureFreeze milestone to new release cycle process

TODO

High

Add forward-copying Extras binary packages in to new release cycle process

TODO

High

Visibility

Work Item

Assignee

Status

Priority

Indicate Source

Software Center should identify packages from the regular archives as being provided by Ubuntu

TODO

High

Software Center should identify packages from the Extras or Partners archive as being provided by Upstream

TODO

High

If any other archive contains a newer version of a package than the regular archives, Software Center should indicate this and provide an option for installing the "Ubuntu provided" version

TODO

Medium

If any other archive contains a newer version of a package than the Extras archive, Software Center should indicate this and provide an option for installing the "Upstream provided" version

TODO

Medium

Software Center should identify packages from any other archives as being provided by an unknown source

TODO

Low

If any other archive contains a newer version of a package installed from the regular archives, Update Manager should indicate this and provide an option for keeping the "Ubuntu provided" version

TODO

Medium

If any other archive contains a newer version of a package installed from the Extras archive, Update Manager should indicate this and provide an option for keeping the "Upstream provided" version

TODO

Medium

The Apps Directory should identify packages from the regular archives as being provided by Ubuntu

TODO

Medium

The Apps Directory should identify packages from the Extras or Partners archive as being provided by Upstream

TODO

Medium

Indicate Access

Software Center should indicate the security policies that will be given to an application1

TODO

High

Software Center should prompt the user to accept the requested security policies prior to installing

TODO

High

Software Center should prevent installation of software that doesn't match the user's custom security policy settings

TODO

High

Provide an option to make an exception to the user's custom security policy for a given application

TODO

High

Update Manager should prevent the installation of new package versions that request more access than the installed version

TODO

High

Update Manager should direct the user to Software Center to approve the upgrade of a blocked package

TODO

High

1 This is no longer a requirement

Feedback

If you have got an interest in making Ubuntu a target platform for app developers and would like to contribute to the specification with your input, please give us some feedback >.

In the AppDevUploadProcess page it says for upload rights permission:"Physical verification: as an alternative, developers can be verified at Ubuntu events such as the Ubuntu Developer Summit (UDS), release parties, jams, etc. A set of individuals will be part of a team or review board who will be able to accept developers applications into MyApps directly. "

Can Ubuntu Developer Summit still be a "physical verification" because the Ubuntu Developer Summit is now virtual? Of course there CAN be sessions where people apply for upload permissions. However it would be super difficult to do so, especially when each session lasts a hour only, and per 6 months. Release parties and jams are not a good idea too, when upload approvers might only be able to attend one specific Local Community's event. This would cause a lot of people unable to do so.

I propose letting the Ubuntu Developer Membership Board (DMB) to do such "verification". After all, they are experts when coming up to developer membership applications such as Ubuntu Core Developer, MOTU, Ubuntu Contributing Developer and per-package uploader. We should treat the app uploaders as a per-package uploader.

However there might be an issue when people cram into a meeting (also one hour, but at least held bi-weekly) and causes the DMB to go overworking. Henceforth, having such a team wouldn't really work "physically" or even "virtually".

Also I do propose one thing: Let people in ~ubuntu-dev upload packages themselves, without needing permission to upload. After all, they already gained upload rights, so they must know the proper things they need to do, and upload properly. I think the Developer Membership Board (DMB) does trust them on that.

So probably this "approval process" should really need to be fully considered. -- smartboyhw 2013-04-14 10:34:40

Please note that feedback will be cleared out as soon as it has been addressed and (if necessary) added to the spec and recorded in the Feedback log.

AppDevUploadProcess (last edited 2013-09-18 11:36:42 by jdstrand)