App Review Board Optimization

Problem

We have made significant in-roads recently in improving the Ubuntu application developer experience (e.g. building developer.ubuntu.com, creating the MyApps submission process, creating documentation and APIs, and providing support resources). This work has made building and delivering applications on Ubuntu significantly easier for developers. Please note: when we say 'developers' here, we are not referring to Ubuntu developers or packagers, but application authors who only care about Ubuntu as a platform (and not how it is built) and being able to deliver their applications to that platform.

An important part of this process is how a developer gets their application into the Ubuntu Software Center. Currently all developers submit their apps using the same interface (MyApps), but the resulting process is quite different:

The problem we have seen recently is that the number of applications requiring review is signficantly overwhelming the ARB. This is resulting in a sub-optimal application developer experience: the developer can wait weeks and often months for their application to get in, if ever. This reflects poorly on Ubuntu as a platform of opportunity for application developers.

Creating a comprehensive platform that delights both our users and our developers who want to deliver content to that platform is critical for Ubuntu to be successful. In a perfect world we want to ensure that we can capture the creativity and imagination of developers and have them be able to deliver this creativity easily and effectively to Ubuntu. We have a tremendous Free Software platform, and today most of that picture is complete, but this ARB review process is unfortunately one of the bottlenecks, and this page seeks to capture the challenges so we as a community can work together to resolve them.

The ARB is a critical component in Ubuntu and many of it's members have performed admirable and exhaustive work. To be clear: these challenges are not a reflection on the current ARB members, but an opportunity to make the ARB experience more pleasureable and efficient for both developers and members of the ARB. Resolving these issues will be essential for us to be able to offer Ubuntu as an attractive platform for application developers.

The Goal of This Page

The goal of this page is to gather the challenges together into one cohesive place so we can brainstorm and plan solutions to these challenges. Many of these topics will form the basis of discussions at the up-coming Ubuntu Developer Summit in Oakland in May 2012.

Statistics

To illustrate some of these challenges, here are some statistics from the 8th February until the 31st March. In this period the board reviewed and published 4 applications:

When we add these to the previously published apps in the 12.04 cycle, this makes a total of 7 applications published for Ubuntu 11.10. 5 applications were rejected in this period (BooruPy Loadr, nSnake, fcitx, Gestimedia, and TerraView 4.2).

The current ARB application queue is 70 submissions. The queue has grown by 19 apps (+35%) since the beginning of this period (since 8th Feb). [This is not accurate, the current queue is 20 apps. Where are these numbers coming from? -Allison]

The conclusion I am drawing here is that despite the admirable contributions of some members of the ARB, the level of demand for application authors to have their applications reviewed and published to the Ubuntu Software Center is far outstripping the throughput of these contributing ARB members. This is resulting in applications languishing in the queue for weeks and months and delivering a sub-optimal experience for app developers.

Potential Issues

This section lists some potential inefficiencies that we can use as a basis for discussion.

Application Prioritization

The board does not have any kind of prioritization facility in MyApps that indicates that App A may be of more interest/more important/more valuable (or however else we define priority) than App B. I think given there are more apps than time, some kind of prioritization would be helpful to ensure that the apps of most interest to Ubuntu users get to the top of the list.

Potential Solutions

Regular Queue Review

There is no regular cadence around queue reviews. This has meant that the ARB have been taking a more ad-hoc approach of picking apps whenever each member has time as opposed to regular meetings and reviewing and moving the submissions along.

Potential Solutions

Defining Scope

I didn't see this myself but a member of my team observed that some ARB members have fixed bugs in some of the apps in the interests of resolving the issues to get them through the queue. While an admirable contribution, time spent fixing bugs in one app takes time away from other apps getting a review.

Potential Solutions

Being Stricter With Needs-Information

Applications that are waiting upon feedback from a developer languish in the queue when a developer fails to respond. This causes the queue to be cluttered with content that has no practical means of moving forward as it is blocking on the developer. We have experienced this elsewhere in the project with bugs and we generally auto-close reports with no feedback within a certain time-frame.

Potential Solutions

Review Assessment Criteria

We should discuss what kind of criteria the ARB are using to review apps in the queue. I suspect there are areas in which the ARB could provide a lighter-weight review while still providing the assurances they provide around quality and security. As an example, in my (humble) opinion, if an application installs, can be removed, and does not pose a security risk, it should get through. If the application quality itself is poor, the ratings and reviews will tell that story to our users.

Potential Solutions

Expectations Of ARB Members

While some members of the ARB have contributed extensively to reviewing content in the queue, some members have contributed little and not had the time to participate. There are two issues here (a) a strong Ubuntu value has always been that our governers step down if they don't have the time to contribute within the expectations of a particular governance board; how do we ensure that we are maintaining a board of participants who can fulfill their responsibilities, and (b) we should revisit the expectations of the time required as an ARB member as I suspect this has changed since the inception of the board a few years back.

Potential Solutions

ARB Capacity is too low

The current ARB members are unable to keep on top of the queue as it continues to grow (the statistics demonstrated this). Aside from some of the efficiencies that are discussed elsewhere on this page, I suspect we simply need a bigger ARB who can provide further hands on deck.

Potential Solutions

Review Packaging Requirements

The initial idea of the app developer process was to provide a streamlined way to submit applications, which was less strict than the workflow for submitting apps to the archive, and that would empower app authors to easily submit regular updates of their software. Some of the current technical packaging requirements have defeated that purpose, be it on the side of the app developers because our tools don't support well these requirements, be it on the side of the ARB, as they are burdened with additional work to fix untested submissions.

Potential Solutions

AppReviewBoard/PreciseReview (last edited 2012-04-19 15:27:33 by static-50-46-157-62)