AppDevUploadProcess

Differences between revisions 198 and 200 (spanning 2 versions)
Revision 198 as of 2012-09-05 11:37:48
Size: 58779
Editor: dpm
Comment:
Revision 200 as of 2012-09-06 12:33:07
Size: 59740
Editor: dpm
Comment: Changed milestone when to open Extras, as per didrocks feedback: APIs are often not yet stable on FeatureFreeze, so it's better to aim for Beta to be on the safe side
Deletions are marked like this. Additions are marked like this.
Line 205: Line 205:
  * `net.launchpad.scope.!RemoteVideos`
  * `net.launchpad.scope.image.flickr`
Line 212: Line 210:

These policies will provide the minimal amount of access needed for desktop applications, including the ability to read and write the application's own user data and settings, interact with desktop services like Indicators and Notifications, and use shared platform resources such as system libraries, fonts and standard icons.
Line 285: Line 285:
When the application needs to open a file outside of its sandbox, it will call a service running outside of the sandbox requesting the file. The service will then prompt the user to select the file using a file chooser, or approve the file selected by the sandboxed application. Only after the user makes and approves of the selection will the application will be given the ability to access it. When the application needs to open a file outside of its sandbox, it will call a service running outside of the sandbox. The application can either request access to a specific file or directory, or prompt the user to select a file or directory that the app should use. The helper service will then prompt the user to select the file using a file chooser, or approve the file selected by the sandboxed application. Only after the user makes and approves of the selection will the application will be given the ability to access it.
Line 288: Line 288:

If a request is made to the helper service for a file or resources that the application already has access to, it should return that resource immediately, without needing additional approval from the user, so that the developer doesn't need to use separate APIs for accessing files inside and outside of their application's sandbox. In order to encourage the use of calls to these helper services, our documentation and tutorials should recommend the use of these APIs for all file access needs, and they should be included in and used by the Quickly templates we provide.
Line 356: Line 358:
When a new development release of Ubuntu is started, the Extras archive will not be immediately opened. Instead, to prevent excessive error reports from upstream applications during the development cycle, the Extras archives for the development release will be opened at the FeatureFreeze milestone. Opening Extras at this point in the development cycle will allow enough time for upstream authors and projects to make any necessary changes to their submission prior to the final release. When a new development release of Ubuntu is started, the Extras archive will not be immediately opened. Instead, to prevent excessive error reports from upstream applications during the development cycle, the Extras archives for the development release will be opened at the '''Beta''' milestone. Opening Extras at this point in the development cycle will allow enough time for upstream authors and projects to make any necessary changes to their submission prior to the final release.

App Developer Upload Process

Please note: This is a work in progress specification and it is not complete. There are parts of the current draft that will change based on feedback and research we are currently doing. This spec's creation is being coordinated by Jono Bacon, David Planella, and Michael Hall.

• If you'd like to contribute please add your feedback or participate in the discussion on the ubuntu-devel mailing list
• For any other queries please contact Jono Bacon <jono AT ubuntu DOT com>.

Problem

Today we face a significant challenge in Ubuntu. If an application author wants to get the latest version of their software in Ubuntu they have two options:

  1. An Ubuntu developer packages the software and delivers it in the development release.

  2. The app developer can submit their apps via MyApps in the stable release for review.

There are various challenges with these two approaches:

  • For (1) app developers cannot get new releases of their software on stable releases. This results in versions of their software in Ubuntu being one or more releases behind the current stable version from upstream (we have seen app developers often not want to recommend the Ubuntu Software Center a place to get their software due to this issue).
  • For (1) this requires knowing an Ubuntu developer who can help you or hoping that an Ubuntu developer cares about your software. This is less empowering for app developers who want to deliver their software directly into Ubuntu.
  • For (2), if the developer is selling their software, they are reasonably well catered: the consumer apps team will take care of them delivering their apps into Ubuntu. For Free Software developers who use approach (2) they must do more work themselves. The Application Review Board (ARB) are there to review these apps but the ARB, as a team of volunteers, is not able to scale with increases in the queue (the recent App Developer Showdown was a good example of this), resulting in long delays for the developer. The ARB frequently gets messages from developers asking if something is wrong when they haven't heard back in 24 hours, and the developers often become discouraged after a delay of a week or so, thinking their work was wasted. A bad first experience gives them no motivation to improve their skills and come back for more.
  • For (2), the developer is often required to make changes to their application and packaging in order to comply with the different packaging and installation requirements for MyApps (such as the /opt requirement).

  • Irrespective of whether the app dev uses (1) or (2), they are relying on other people to do work before they can deploy their application. This is less empowering, and for an emerging platform such as Ubuntu, we need to enable app developers to deliver their apps without such delays.

Goal

The goal here is to provide a safe, reliable means in which app developers can upload their stable releases directly into Ubuntu.

With this goal in mind we want to resolve the common bottlenecks that we have experienced with the current process for app developers:

  • We should not rely on manual reviews of software before inclusion. Manual reviews have been found to cause a significant bottleneck in the MyApps queue and they won’t scale effectively as we grow and open up Ubuntu to thousands of apps.

  • Requiring app installation into the /opt directory is significantly challenging to app developers and not well supported in Ubuntu; this should not be a requirement.

We also want to enhance the platform experience for both users and developers who are building applications for Ubuntu:

  • Provide a standard, stable and useful platform for developers, and a rapid path to delivering their apps to Ubuntu users
  • Give users a safer access to the growing ecosystem of independent applications being developed for Ubuntu.
  • Keep the user informed and in control over who is providing the software they install, and what access to their system it will have.

Please note: this specification only focused on applications that are uploaded to the Extras extension repository. Thus, this process does not cover developer uploads of newer versions of software in the existing main archive (which includes the main, restricted, universe and multiverse components).

Proposed Solution

This section outlines the proposed solution for how app developers upload their apps, and how end users can browse a combination of apps delivered by Ubuntu/Debian developers as well as apps uploaded by app developers.

How App Devs Upload Their Apps

Here is a proposed solution to resolve these problems.

App Dev Upload diagram

  1. App Preparation - the developer can use our tools to generate a package to be included in the Ubuntu Software Center. These tools will likely include Quickly and pkgme to generate the package.

  2. Developer Request - in MyApps the developer creates a request to upload their application through MyApps. The developer will be asked to provide proof that they are either (a) the copyright or trademark owner of the application or (b) a key representative of the team that created it (e.g. a core developer for an Open Source project) or (c) has the endorsement of the project to provide packages to Ubuntu. They will not be granted permissions to any package name currently in the Ubuntu archive. The developer will also be required to agree to terms and conditions that will clearly outline what content is acceptable and unacceptable, and that the software can be removed if questionable, illegal, or infringing content is found. Once approved the developer will be provided with upload access for that specific application to extras.

  3. Security Policy Preparation - the developer then fills in a web form that outlines the resource requirements for the application. This will generate an AppArmor policy for the application and provide its access needs to the Software Center.

  4. App Review - When the developer submits a new version of their application, it will go through a series of automated checks to verify that it conforms to the Extras packaging and security rules. The developer will be responsible for ensuring the package works effectively. If it doesn’t, the ratings and reviews will surely reflect this.

  5. Upload - The developer is now able to upload versions of their package to the Ubuntu Software Center without each version requiring a specific review and thus achieving the goals highlighted in this spec. The developer will not be able to upload other packages, just the one they have access to upload. This upload will be performed by MyApps at the developer's request.

  6. Visibility - When this version of the application appears in the Ubuntu Software Center it will be made clear that the application is provided by the developer, who is solely responsible for it. Users will be able to see a combination of applications provided by Ubuntu/Debian developers (which are supported) as well as developer-uploaded applications such as outlined by this process. See the next section for more details.

How End Users Browse Apps

The goal here is to ensure that Ubuntu users can make an informed choice about what applications they want to install. This informed choice should not only be based upon the features of the application but also how support the application is.

There are two types of application that the Ubuntu Software Center will show:

  1. Officially supported apps - these are the apps in main that are supported as part of our support policy. These are likely to be older version but releases with assured integration and security fix support.

  2. Developer uploaded apps - these are the apps that app devs can upload. These are not supported by Ubuntu developers and the app developer is responsible for any quality and integration considerations.

We would need some design input on the best way of presenting this information.

Implementation

App Dev Upload Diagram (II).png

App Preparation

This step deals with how app authors will prepare their app before submitting it to MyApps and starting the app developer upload process. We want to be able to help and support our app developers to submit their applications in a format that can be easily processed for inclusion in the Ubuntu Software Center.

MyApps will accept one of the following formats (please note the priorities listed that indicate the order in which we want to build this support):

  • Debian source package (high priority) - this works out of the box with Quickly today and would be our recommend method of submitting content. On the server side, pkgme would unpack it to integrate the security profile into a newly-generated source package.

  • Source tarball (medium priority) - in this case app developers do not have to worry about packaging at all, as the package is generated automatically on the server side. Requires work on pkgme to implement an initial set of supported backends. We need to support this format for the long tail of app developers who will be bringing their apps to Ubuntu without any prior packaging experience.

  • PPA link (low priority) - this also works out of the box with Quickly and is simply a variation of the source package format format. This provides a useful means in which app developers can run a PPA for where they publish and test their packages and then simply nominate a a PPA package for release in the Ubuntu Software Center.

At the final stage of application development, when the author feels the application is ready to be published, they will be able to execute a Quickly command to prepare their code in a way that can be processed by MyApps. The command will be:

quickly submitubuntu

Alternately, the author will be able to manually submit their application through the MyApps web interface, selecting the source package files from their computer or directing MyApps to download them from a PPA. The goal is to ultimately create a package on the server side that can be distributed in the Software Centre.

Your app tab, upload controls

Your app tab, upload controls

Your app tab, upload controls, version I (click to zoom)

Your app tab, upload controls, version II (click to zoom)
Erratum: should be "As a source package", with three upload fields: "orig.tar.gz", ".dsc" and ".diff.gz/.debian.tar.gz"

We will no longer be requiring applications to be installed to /opt and rely on the sandboxing that the security profile will provide and on file path conflict checking on the archive side. However, for extra security we will only have a set of whitelisted locations where applications will be able to install files. The same or equivalent set of tools will be run on the server side to do the checks.

The whitelist of approved installation locations is:

Whitelisted install location

Notes

/usr/share/${appname}/

/usr/bin/${appname}

/usr/bin/${appname}-*

/usr/lib/${appname}/

/usr/share/unity/lenses/${appname}/

/usr/share/applications/${appname}.desktop

/usr/share/applications/${appname}-*.desktop

/usr/share/dbus-1/services/${appname}.service

DBus service files. The file names are in reverse domain naming notation (e.g. com.ubuntu.OneConf.service)

/usr/share/dbus-1/services/${appname}-*.service

Same as above

/usr/share/glib-2.0/schemas/${appname}.gschema.xml

GSettings schema files. The file names are in reverse domain naming notation (e.g. org.gnome.Evince.gschema.xml)

/usr/share/doc/${appname}/

/usr/share/man/man{3,8}/${appname}*.*.gz

/usr/share/help/<lang>/${appname}/

/usr/share/locale/<lang>/LC_MESSAGES/${appname}.mo

/usr/share/icons/hicolor/<size>/${appname}.{png,svg}

/usr/share/python/runtime.d/${appname}.rtupdate

/etc/apport/crashdb.conf.d/${appname}-crashdb.conf

Apps will only be allowed to install files in these locationsPermalink

On the client side, Quickly and pkgme will integrate a custom Lintian profile to do the checks for whitelisted location (and any additional ones required) and will output warnings or errors to indicate whether the local app might need to be fixed before being submitted.

While this specification is concerned with apps created by the tools that Ubuntu provides (which in this case they are Quickly), app developers that write their apps outside Quickly should be able to get their software prepared in the same form Quickly does.

A further step towards better process automation would be to provide an API to enable Quickly to directly send the application to MyApps. While this would streamline the process, it is considered non-essential for the current revision of the spec.

Developer Request

Applying for access

To ensure that we are giving upload access only to the original author or a proper representative of the upstream project, we will require that person to request upload access for their application. The author or representative must first create an account and user profile in the MyApps portal as it currently exists.

Once their profile is created, they will need to be able to request upload access for a package, providing details about their association with the upstream project. If the submitter is not the owner or representative of the project, they will be required to provide a URL to a webpage, blog post or mailing list archive showing that the owner or representative of the project is endorsing their effort upload the application to the Ubuntu Software Center.

A link will be presented on their user profile page that will point to a new page containing a form for the developer to fill out. The details required by the form should be sufficient for a review board to determine whether or not the requesting user meets the requirements to represent that project. The review board will be responsible for developing the detail requirements, but it should include the following:

  1. The developer's first and last name (e.g. Joe Smith).

  2. A contact email for the developer (e.g. joe.smith AT email DOT com)

  3. The name of the package(s) that the user wishes to upload (e.g. accomplishments-viewer). This could be more than one package if required.

  4. If the developer is the sole owner of the application, they need to tick a box to verify that they are copyright owner, representative, or endorsed package provider for the project. This would be in the form of a textbox in which the user prepares a summary of their involvement in the upstream project with content such as the following recognized:
    • Project information page listing the user as the creator or current owner of the project
    • Links to commit logs from the user.
    • Active participation in the project strategy and discussion (such as mailing list posts).
    • Testimonials from other members of the project.
    • Links to an specific endorsement for uploading the application to the Ubuntu Software Center

App details page, Uploaders tab

Your app tab, with upload request

App details page, Uploaders tab (click to zoom)

Your app tab, with upload request (click to zoom)

Upload rights request page

Upload rights request page (click to zoom)

In addition, the form should include an agreement text stating that the requesting user assumes responsibility for the maintenance and content of the package they are requesting access to upload. This information will be saved to the MyApps database. More than one of these requests may be made by the same MyApps user, if they wish to maintain more than one package in the Extras archive.

Developer Verification

A developer's request will then be sent to the review board as an email message, which will include the name of the developer, the package they are requesting upload access for, and a link to review the rest of their request details on MyApps.

The link will point to a new page on MyApps where members of the review board can review the details submitted by the developer. This page will also provide a form for leaving comments on the developer's request. These comments will be visible to the board members but not the developer making the request. This page will give the board members the ability to mark the request as accepted or rejected and include any final comments to the developer once they have completed their review.

The review board will use the provided information to verify only the identity of the requesting developer. Verification of the developer's role in the project is not needed for approval, but will be saved for future verification in the event that the owner or representative of the project objects to their upload access. An account is considered complete when all available information is present and the review board have confidence in the identity of the developer.

MyApps will perform an automated check to ensure that the package name being requested is not already taken.

Revoking Access

Given the number of developers who are expected to to start using this new upload process, it is likely that there will be times when, for one reason or another, we will need to revoke their access. This will occur when the terms of conditions of the service have been breached. Some potential causes for access revocation include, but are not limited to:

  1. Attempting to upload malicious content.
  2. Uploading content in violation of the Extension Repository Policy

  3. Uploading content in violation of copyright, trademark, or other legal restrictions.
  4. The owner or representative of a project objects to their upload access.

Revoking access is something that will only occur in situations in which the terms and conditions have been breached. Access will be revoked by changing the status of the user's upload request record from "Approved" to "Revoked" in MyApps.

Access will not be revoked if an application is considered poor quality. The ratings and reviews system is a useful approach in which our users can communicate their dissatisfaction (or satisfaction) with an application and help other users to make a judgement call of whether to install it or not.

In the case of content in violation of the Extras archive policies, users can submit an abuse complaint that will be reviewed by the review board indicating the application and what content they feel is objectionable. The review board will then contact the developer to determine whether the application can be resolved so that it complies with the archive policies. If it can not, the review board will meet to determine whether the application should be removed from the archive and the developer's upload access for that application will be revoked.

In the case of a legal violation, such as copyright or trademark, either the review board or Canonical will receive the complaint from the copyright or trademark holder. In this case the review board will immediately remove the package from the archive and suspend the developer's upload access until the issue has been resolved. They will then contact the developer with information about the complaint, and determine whether the application can be fixed so that it is no longer in violation. If it can not, the review board will permanently revoke the developer's upload access for that application.

The Ubuntu Software Center, when viewing a package from the Extras archive, will display an additional link for reporting abuse by the package. The link will direct the reporter to a page on MyApps with a form for detailing the abuse being reported, and optional contact information about the reporter. The form should include a radio option listing the 4 reasons for revoking access listed above, or an optional "other" reason that the reporter must specify.

Security Policy Preparation

Security Profile Creation Tab

Security profile creation tab in MyApps (click to zoom).

Once an application is installed, AppArmor will provide a security sandbox to protect the user from malicious or simply broken code. To accomplish this, each application will need a properly constructed, and minimally permissive profile to restrict what their application can access on the local system. Producing such a profile, and more importantly checking it for correctness, is currently a very manual process.

AppArmor Creation

In order to reduce the complexity of the AppArmor profiles used by applications submitted to Extras, the author will be asked to fill out a form in MyApps describing the access needs of their application. The options presented to him will correspond to the following pre-defined Policy Groups used by the AppArmor Easy Profile (aa-easyprof) script.

Default Permissions

The following AppArmor policy abstractions will be given by default:

  • Base - Basic application access

  • Freedesktop.org - Give read access to standard system resources such as icons, menu and application descriptors

  • XDG-desktop - Gives access to the user's XDG_CONFIG_HOME and XDG_CACHE_HOME directories

  • Unity Integration - Gives access only to those DBus services below, needed to integrate with the Unity desktop

    • com.canonical.Unity*

    • com.canonical.AppMenu.Registrar

    • com.canonical.hud*

    • com.canonical.indicator*

    • net.launchpad.lens.photo

    • net.launchpad.lens.video

    • org.ayatana.bamf

    • org.freedesktop.Notifications

  • Fonts - Gives access to installed system fonts

  • Private-files-strict - Prevents access to the user's private files and directories

  • X - Needed for all X11 applications

These policies will provide the minimal amount of access needed for desktop applications, including the ability to read and write the application's own user data and settings, interact with desktop services like Indicators and Notifications, and use shared platform resources such as system libraries, fonts and standard icons.

Additional Permissions

Any additional permissions chosen (other than those under Default above), will be shown to the user on the Software Center on the app details screen. There will also be a preferences dialog in the Software Center where security-conscious users will be able to choose whether they want to be prompted to install apps that fall outside the security policy they set up in this preferences dialog. They will also be able to override (and remember) the security policy on a single app basis (similarly to what Firefox does when prompting to access sites with invalid SSL certificates).

In addition to the default access, the developer will need to identify what runtime environment their application will need access to (these can be pre-populated based on a scan of the submitted package):

  • GNOME - Adds additional access commonly needed by applications meant to integrate with Gnome

  • KDE - Adds additional access commonly needed by applications meant to integrate with Kde

  • Python - Needed for Python applications to access installed Python libraries

  • Perl - Needed for Perl applications to access installed Perl libraries

  • Ruby - Needed for Ruby application to access installed Ruby libraries

  • PHP - Needed for PHP application to access installed PHP libraries

The author may also choose from the following access options, which will be presented to the user for acceptance before the application will be installed:

  • Network - Group of policies allowing network access

    • Nameservice - Grants access to the network and network services

    • SSL certs - Needed for application that need to verify and decrypt SSL encrypted data

  • Spell Checking - Group of policies allowing access to spell-checking dictionaries

    • Aspell - Needed for application that use the aspell spell-checking dictionary

    • Enchant - Needed for applications that use the enchant spell-checking dictionary

  • Printing - Gives the application access to print

    • CUPS client - Needed for application that wish to access a printer

  • Camera - Gives the application access to the user's webcam

    • Video - Needed for application that wish to use v4l

In the case where there are 'child' abstractions (e.g. on Network and Spell Checking) we might want to assess whether it is worth just showing the main AppArmor abstraction and enabling all children for the sake of simplicity (and unless finer grain control is required).

The chosen options will be stored in the MyApps database, with a ForeignKey to the version of the submitted application they were created for. When the author submits a new version of their application, these values will be used as the default for the new version, but the author will again be presented with the form to either verify the current options or specify different ones.

AppArmor Integration

The developer will not be able to add any other AppArmor abstractions beyond the ones defined above. All of these options are considered safe, and do not require a manual review before being allowed into the Extras archive. The user will be told about any options from the third group that the application needs, and will be asked to allow or reject its installation.

Once the security profile is defined, MyApps will call the aa-easyprof program through an external plugin or service (e.g. pkgme), passing the list of policy groups and abstractions defined by the author, along with a base template used for Extras application, to produce an AppArmor profile. The base template will contain the default set of AppArmor policies that will be needed by most desktop applications, and which won't allow the application to interfere with any other application on the system. The resulting AppArmor profile will then be linked to on the application's page in MyApps, so that the developer can download and test it manually. Quickly should be able to download this profile and re-build a project package using a custom command.

MyApps will unpack the submitted Debian source package, add the generated AppArmor profile, and rebuild the source package, signing it with the MyApps GPG key. The source package will then be uploaded to the staging PPA to be built into a binary package. A link to the resulting binary package will then be displayed on the submission page in MyApps, which the developer can use to install and test their application running sandboxed under the newly created AppArmor profile.

By limiting the number of AppArmor options to a set of common access needs defined by aa-easyprof Policy Groups, the creation of the profile can be fully automated. And by generating it on a trusted system, the integrity and correctness of the profile can be trusted without a manual review.

App Sandboxing

Because the security and stability of the user's system will depend more on the inability of running applications to cause problems than on a manual review of the application identifying them, a robust sandbox will be needed to prevent any un-granted access.

AppArmor is the preferred way of providing mandatory access controls (MAC) on Ubuntu, and it will also serve as the basis for most of the application sandboxing that will be applied to Extras application. AppArmor will provide the following necessary restrictions:

  • Limit filesystem read and write access to a pre-defined whitelist of files and directories
  • Restrict an application's ability to access the network
  • Limit what network destinations an application can request
  • Limit what DBus service the application can call
  • Prevent an application from listening in on other applications' DBus communication
  • Limit what external programs an application is able to call
  • Force any external programs to run under the same restrictions as the calling application
  • X server (requires Xace plugin and AppArmor support)

    • Prevent an application from listening in on another application's keyboard/mouse input and output
    • Restrict an application's ability to perform screenshots outside of its window
    • Restrict an application's ability to access the clipboard
    • Limit/Support 3D in some manner

In addition to AppArmor, sandboxing will be required by these other parts of Ubuntu as well:

  • DConf/GSettings
    • Prevent an application from writing to system-wide or session-wide settings
    • Prevent an application from reading or writing to another application's settings
    • Prevent Gtk from loading and executing additional modules based on an application's settings
  • GNOME Keyring
    • Prevent unauthorized applications from obtaining credentials
  • Ubuntu Online Accounts
    • Prevent unauthorized applications from obtaining credentials

It is expected that MyApps toolkits will need to be patched to gracefully handle sandbox denials. Eg, it is known that some applications will crash when faced with an Xace denial.

Helpers

Because application authors will only be able to choose from a predefined set of access policies, a proxy service will allow the application to extend its access beyond the confines if its AppArmor profile with the user's approval. This service will broker requests for additional access between the application and the user, and either dynamically expand the application's sandbox, put a copy of the content into the application's sandbox, or allow the application to read and write through the proxy service itself.

When the application needs to open a file outside of its sandbox, it will call a service running outside of the sandbox. The application can either request access to a specific file or directory, or prompt the user to select a file or directory that the app should use. The helper service will then prompt the user to select the file using a file chooser, or approve the file selected by the sandboxed application. Only after the user makes and approves of the selection will the application will be given the ability to access it.

Helper services will also provide content guarantees by checking and/or converting the format of what is being saved. This would allow the user to approve writing files of a certain type, such as an png image, to a location outside of the AppArmor sandbox. When the application attempts to save data to this location, the service will verify that the content matches what has been approved, and either convert it (if possible) or reject it.

If a request is made to the helper service for a file or resources that the application already has access to, it should return that resource immediately, without needing additional approval from the user, so that the developer doesn't need to use separate APIs for accessing files inside and outside of their application's sandbox. In order to encourage the use of calls to these helper services, our documentation and tutorials should recommend the use of these APIs for all file access needs, and they should be included in and used by the Quickly templates we provide.

App Review

Submission complete, package errors already found

Submission complete, package errors already found (click to zoom)

Package errors shown on standalone page

Package errors shown on standalone page (click to zoom)

As part of this process, the successful applicants will be given upload rights to their software. This will enable them to upload their packages without any further review, given that they conform to the rules set out in the App Preparation section. In addition to those rules, MyApps will perform a series of automated checks against the submitted source package to further ensure the safety and stability to a user's system.

If an app does not pass the automated checks in MyApps, it will remain in Draft status and the app author won't be able to submit it until any warnings or errors have been addressed. This will require tools like Lintian to be run on the submitted source package, which is a process that can take in the order of minutes to be finished. Thus there should be a mechanism to asynchronously show any status messages in the MyApps UI and to send a notification e-mail to the uploader. The checks can either be triggered when pressing the 'Submit' button at the end of the process, or already when the source package is uploaded.

DBus Service Restrictions

If a submitted application will be installing a DBus session bus service, a further check will be performed to prevent service name conflicts. To accomplish this, a complete list of service names must be maintained and used to verify any new service names.

To create the list of all service names, a one time scan of the archives will be performed at the point in each development cycle where the Extras archive is opened for the new release. This scan will perform the following:

  1. Find packages with a .service file

  2. Download the package
  3. Extract any .service files from the package

  4. Parse the DBus service name from the .service file

  5. Append both the source package name and service name to the list

Whenever a new package is submitted to MyApps, it will be scanned for the presence of a .service file. If one is found, it will be extracted from the package, have its DBus service name parsed and checked against the list. If no match is found, the new package and service name will be appended to the list, and the submission will be allowed to proceed. If a match is found, the submission will be rejected. MyApps will display this conflict on a review page for the developer, showing both the conflicting service name and any existing packages that are using it.

In addition, a blacklist of DBus service name prefixes will be checked. Services using one of the following will not be allowed:

Blacklisted DBus service

Notes

com.canonical.*

net.launchpad.*

org.ayatana.*

org.gnome.*

org.gtk.*

org.freedesktop.*

Apps will not be allowed to install DBus services in these namespacesPermalink

Packaging Restrictions

The Security Team raised the concern that packaging dependencies should also be checked to ensure they cannot introduce security regressions or vulnerabilities. These will only deal with dependencies and not other packaging fields or aspects. This should be an automated check that in practical terms will be coded in a Lintian profile to be run on the server and that will reject the submission if the checks raise a warning. MyApps will display this warning on a review page for the developer.

The current list of rules to check for in the new Lintian profile is:

  • Only reject Depends relationships with = or <= in the Debian control fields of the package

Upload

Once all the app itself, its metadata and security profile have been integrated into a source package, the server will initiate the process of uploading, building and publishing a distributable binary package.

pkgme will be the piece of infrastructure that will do most of the work at this this stage. The main goal is again automation and not having to rely on human intervetion for the upload step. Following this premise, the steps will be:

  1. Pkgme uploads the final source package containing the AppArmor profile to the ARB PPA

  2. The ARB PPA is synced automatically (hourly) to extras.ubuntu.com
  3. The final binary package appears on extras.ubuntu.com and is distributable at that point

Extras Archive

The Extras archive is central to providing an improved process for uploading applications to the Ubuntu Software Center. This archive will combine the features of both the distro archives and Launchpad PPAs in a way that provides both developers and users with reliable access to the latest versions of applications.

Rules Changes

To reduce the time and effort required to get an application into the Extras archive, the need for manual review of an application's source code will be removed. Instead, a proper application sandbox and associated AppArmor profile will prevent both accidental and intentional abuse of a user's system. A manual review will not be needed to verify the security profile, because the profile will be generated by MyApps using a list of pre-approved policy options.

In order take better advantage of Ubuntu's development tools and runtime configurations, applications will no longer be required to install into the /opt/extras.ubuntu.com/ directory. The original requirement for installing applications into /opt/ was intended to prevent file conflicts during installation, and also to prevent changes that affect other parts of the user's system such as overriding libraries or startup scripts. This will now be accomplished by restricting install locations to a whitelist of files and directories that are both specific to the installing application, and won't affect other parts of the user's system.

New Release Process

When a new development release of Ubuntu is started, the Extras archive will not be immediately opened. Instead, to prevent excessive error reports from upstream applications during the development cycle, the Extras archives for the development release will be opened at the Beta milestone. Opening Extras at this point in the development cycle will allow enough time for upstream authors and projects to make any necessary changes to their submission prior to the final release.

Once opened, the binary packages in the archive from the previous release will be copied into the new release. Forward-copying the binary requires no additional action on the part of the upstream nor any new review by the review board. Packages will not be re-built on the development release of Ubuntu, it will be left to the upstream author or project to ensure that their packages remain buildable on the new release.

From the experience gathered on paid-for applications on the Commercial queue of MyApps, the forward-copying process generally works successfully for 95% of the apps. Those apps for which the copying process does not work are usually affected by either multiarch issues or named, numbered libraries they depend on.

Visibility

Ubuntu has a long history of being stable and secure, a reputation that was made possibly by careful and thorough reviews by both Ubuntu and Debian developers. When changing the source and method of delivering applications to users, it will be important to keep the user informed about what they are installing, who is providing it, and the possible side affects it may cause.

Indicate Source

There are several places where an Ubuntu user will see the availability of an application. Before selecting an application to install or version to upgrade, the user will be informed, as much as possible, about the provider of the packages and the level of support provided.

The Web Catalog (apps.ubuntu.com) will also be used to identify the provider of the different available versions of an application. For versions of an application in the distro archives, the web catalog will indicate that the version is provided by the Ubuntu project. For applications in the Extras archive, the web catalog will indicate that the software is provided by the upstream author or project.

When an application crash is detected, and if the application was installed from either the distro archives or the Extras archive, Apport will display a dialog asking the user if they would like to report the crash. If the installed application is from the distro archives, Apport will indicate that the error was from an Ubuntu provided package. If the installed application is from the Extras archive, Apport will indicate that the error was from a package provided by the upstream author or project. In addition, when the version of the application is provided by the upstream author or project, the user will be warned that the crash details being collected will be seen by the upstream author or project if submitted.

Indicate Access

In addition informing the user about the source of an application, they will need to be informed about the level of access being given to the application in its AppArmor profile.

MyApps will provide an API with information about the security policy options for each application published to the Extras archive. This list will be limited to the optional, non-runtime security access options from the 3rd list in the AppArmor Creation section above.

The Ubuntu Software Center will use the MyApps API to present the user with a list of policies being requested by an application they wish to install. If the application is requesting one or more of these options, Software Center will prompt the user to accept them before the application is installed.

If the user has defined a custom security profile for their system, Software Center will check to see whether an application from Extras is requesting access to one or more resources that has been denied in the user's profile. If so, Software Center will inform the user that the application violates their custom profile, and provide an option to make an exception for the application in question in order to allow its installation.

The Update Manager will also use the MyApps API to see if a newer version of a package is going to access one or more resources that the currently installed version did not. When this happens, Update Manager will de-select and disable the package from its upgrade list, and direct the user (with a link or button) to the Software Center so that they can view and approve the new security policy options before upgrading the package.

Work Items

The following tables list the high-level work items that are required to implement this specification.

App Preparation

Work Item

Assignee

Status

Priority

Package Creation

Make Pkgme package Python apps

INPROGRESS

High

Make Pkgme package C apps

TODO

High

Make Pkgme package C++ apps

TODO

High

Make Pkgme package C#/Mono apps

TODO

Low

Make Pkgme package Vala apps

TODO

Low

Make Pkgme package Java apps

TODO

Medium

Make Pkgme package Flash apps

TODO

Low

Make Quickly use correct version number scheme

TODO

Low

Make Quickly produce a correct copyright file from AUTHORS file

TODO

Low

Make Quickly produce a clean changelog on submitubuntu (only one entry)

TODO

Low

Package Verification

Create a new Lintian profile for checking Extras packages

TODO

High

Make Lintian check for the correct version number scheme

TODO

Low

Make Lintian check that the changelog has only one entry

TODO

Low

Make Lintian check the proper copyright file format

TODO

Low

Make Lintian check for = or <= in Depends

TODO

High

Make Quickly use the new Lintian profile

TODO

High

Make Pkgme use the new Lintian profile

TODO

Low

Package Isolation

Define a whitelist of directories where an app can install files (using regex matches)

TODO

High

Make Quickly install only to whitelisted directories

TODO

High

Make Pkgme install only to whitelisted directories

TODO

Low

Make Lintian check for files installing outside of the whitelisted directories

TODO

High

Make Lintian check for the existance of maintainer scripts

TODO

High

Developer Request

Work Item

Assignee

Status

Priority

Applying for access

Add data models in MyApps for upload requests, reviews and approvals

TODO

High

Add a form in MyApps where a developer can request upload rights for an app in the Software Center

TODO

High

Add a page to MyApps where a developer can view the status of their request, leave and respond to comments

TODO

Medium

Add an agreement page to MyApps where the developer can record their acceptance of the rules for uploading to Extras

TODO

High

Ask the legal team to check if the terms of service need any changes as a result of the new process

TODO

High

Add a page or pages where members of the review boards can review, comment on, and approve or reject upload requests

TODO

High

Developer Verification

Define criteria a developer must meet in order verify their identity

TODO

High

Define a process of verifying the identity criteria, including which review board or group will be responsible for doing it

TODO

High

Revoking Access

Define the reasons why a developer's upload access may be revoked due to problems with the stability of their packages

TODO

High

Define the reasons why a developer's upload access may be revoked due to problems with the security of their packages

TODO

High

Define the reasons why a developer's upload access may be revoked due to problems with the content of their packages

TODO

High

Define a method for reporting problems with the stability, security or content of a package in Extras

TODO

Medium

Define a process for reviewing whether a reported application should result in the revocation of a developer's upload access

TODO

High

Provide a form on MyApps for reporting abuse or ToS violations

TODO

High

Add a link in the Software Center for Extras package pointing to the MyApps abuse form

TODO

High

Security Policy Preparation

Work Item

Assignee

Status

Priority

AppArmor Creation

Add a form to MyApps where the developer can select what system resources their application will need to access

TODO

High

Add an API to MyApps for querying the security policy options being requested by an application

TODO

High

AppArmor Integration

Make MyApps generate an AppArmor profile, based off aa_easyprof, that enables access only to the requested resources

TODO

High

Make the new AppArmor profile available for download (on MyApps) by the developer

TODO

High

Add a Quickly command to automatically download and integrate the generated profile with the local project

TODO

Low

Make MyApps build a new binary package, using the uploaded source package and the generated AppArmor profile

TODO

High

Make the new binary package available for download (on MyApps) by the developer

TODO

Medium

App Sandboxing

Restrict an application's access to files and directories to pre-approved locations

Done

High

Restrict an application's access to the network

Done

High

Restrict an application's access to certain network destinations

TODO

Low

Restrict DBus communication so an app can't see another program's data

INPROGRESS

High

Restrict an application's access to DBus services to pre-approved services

INPROGRESS

High

Restrict X11 communication so an app can't see another program's data

TODO

High

Restrict dconf access so an app can't see or change another program's settings

TODO

High

Restrict gnome-keyring access so an app can't see or change another program's keys

TODO

Medium

Restrict which external programs an application can call to pre-approved programs

DONE

High

Restrict an application's access to the GSettings/GConf values for another application

TODO

Low

Restrict an application's manipulating of the environment when executing programs

TODO

High

Restrict an application's access to gnome-keyring credentials

TODO

High

Restrict an application's access to ubuntu online accounts credentials

TODO

High

Restrict an application's ability to load Gtk modules via specially crafted GSettings

TODO

Low

Run externally called programs under the calling application's restrictions

DONE

High

Helpers

Provide a "helper" application for giving a restricted app access to un-approved files with the user's consent

TODO

High

App Review

Work Item

Assignee

Status

Priority

Package submission

Update MyApps to allow uploading an initial Debian source package (verifying the package)

TODO

High

Update MyApps to allow specifying an initial package from a PPA (verifying the PPA and the package)

TODO

High

Package checks

Make MyApps check for install file conflicts on newly uploaded packages

TODO

High

Make MyApps check for files installing outside of the whitelisted directories

TODO

High

Make MyApps check for the existance of maintainer scripts

TODO

High

Make MyApps check for the upper-limits on package dependency (= or >= in Depends)

TODO

High

Make MyApps check for conflicting DBus service names in .service files

TODO

High

Make MyApps check for blacklisted DBus service name prefixes

TODO

High

Upload

Work Item

Assignee

Status

Priority

Extras Archive

Update MyApps to include a data model for the final package to be uploaded (separate from the initial package)

TODO

High

Add a page to MyApps where the developer can manage their uploadable packages

TODO

Medium

Give MyApps access to upload packages to the Extras archive

TODO

High

Add a page to MyApps allowing the developer to initiate the upload to Extras

TODO

High

Add a page on MyApps where the developer can monitor the status of their uploading/uploaded packages

TODO

Medium

Add a page to MyApps showing the upload history of a given package

TODO

Low

Add a mechanism to MyApps that would allow a developer to remove their package from Extras

TODO

Low

Extras Archive

Work Item

Assignee

Status

Priority

Rules changes

Add page to developer.ubuntu.com explaining the new submission process and requirements

TODO

High

Remove ARB wiki pages including submission requirements

TODO

High

Add Extras to /etc/apport/native-origins.d/

TODO

Medium

Build debug packages for compiled apps in Extras

TODO

Low

New release process

Add opening Extras at the FeatureFreeze milestone to new release cycle process

TODO

High

Add forward-copying Extras binary packages in to new release cycle process

TODO

High

Visibility

Work Item

Assignee

Status

Priority

Indicate Source

Software Center should identify packages from the regular archives as being provided by Ubuntu

TODO

High

Software Center should identify packages from the Extras or Partners archive as being provided by Upstream

TODO

High

If any other archive contains a newer version of a package than the regular archives, Software Center should indicate this and provide an option for installing the "Ubuntu provided" version

TODO

Medium

If any other archive contains a newer version of a package than the Extras archive, Software Center should indicate this and provide an option for installing the "Upstream provided" version

TODO

Medium

Software Center should identify packages from any other archives as being provided by an unknown source

TODO

Low

If any other archive contains a newer version of a package installed from the regular archives, Update Manager should indicate this and provide an option for keeping the "Ubuntu provided" version

TODO

Medium

If any other archive contains a newer version of a package installed from the Extras archive, Update Manager should indicate this and provide an option for keeping the "Upstream provided" version

TODO

Medium

The Apps Directory should identify packages from the regular archives as being provided by Ubuntu

TODO

Medium

The Apps Directory should identify packages from the Extras or Partners archive as being provided by Upstream

TODO

Medium

Indicate Access

Software Center should indicate the security policies that will be given to an application

TODO

High

Software Center should prompt the user to accept the requested security policies prior to installing

TODO

High

Software Center should prevent installation of software that doesn't match the user's custom security policy settings

TODO

High

Provide an option to make an exception to the user's custom security policy for a given application

TODO

High

Update Manager should prevent the installation of new package versions that request more access than the installed version

TODO

High

Update Manager should direct the user to Software Center to approve the upgrade of a blocked package

TODO

High

Feedback

If you have got an interest in making Ubuntu a target platform for app developers and would like to contribute to the specification with your input, please give us some feedback >.

In the AppDevUploadProcess page it says for upload rights permission:"Physical verification: as an alternative, developers can be verified at Ubuntu events such as the Ubuntu Developer Summit (UDS), release parties, jams, etc. A set of individuals will be part of a team or review board who will be able to accept developers applications into MyApps directly. "

Can Ubuntu Developer Summit still be a "physical verification" because the Ubuntu Developer Summit is now virtual? Of course there CAN be sessions where people apply for upload permissions. However it would be super difficult to do so, especially when each session lasts a hour only, and per 6 months. Release parties and jams are not a good idea too, when upload approvers might only be able to attend one specific Local Community's event. This would cause a lot of people unable to do so.

I propose letting the Ubuntu Developer Membership Board (DMB) to do such "verification". After all, they are experts when coming up to developer membership applications such as Ubuntu Core Developer, MOTU, Ubuntu Contributing Developer and per-package uploader. We should treat the app uploaders as a per-package uploader.

However there might be an issue when people cram into a meeting (also one hour, but at least held bi-weekly) and causes the DMB to go overworking. Henceforth, having such a team wouldn't really work "physically" or even "virtually".

Also I do propose one thing: Let people in ~ubuntu-dev upload packages themselves, without needing permission to upload. After all, they already gained upload rights, so they must know the proper things they need to do, and upload properly. I think the Developer Membership Board (DMB) does trust them on that.

So probably this "approval process" should really need to be fully considered. -- smartboyhw 2013-04-14 10:34:40

Please note that feedback will be cleared out as soon as it has been addressed and (if necessary) added to the spec and recorded in the Feedback log.

AppDevUploadProcess (last edited 2013-09-18 11:36:42 by jdstrand)