PreciseReview
App Review Board Optimization
Problem
We have made significant in-roads recently in improving the Ubuntu application developer experience (e.g. building developer.ubuntu.com, creating the MyApps submission process, creating documentation and APIs, and providing support resources). This work has made building and delivering applications on Ubuntu significantly easier for developers. Please note: when we say 'developers' here, we are not referring to Ubuntu developers or packagers, but application authors who only care about Ubuntu as a platform (and not how it is built) and being able to deliver their applications to that platform.
An important part of this process is how a developer gets their application into the Ubuntu Software Center. Currently all developers submit their apps using the same interface (MyApps), but the resulting process is quite different:
- For commercial applications the Canonical consumer applications team review those applications and upload them.
For Free Software applications the Application Review Board (ARB) review these applications and upload them.
The problem we have seen recently is that the number of applications requiring review is signficantly overwhelming the ARB. This is resulting in a sub-optimal application developer experience: the developer can wait weeks and often months for their application to get in, if ever. This reflects poorly on Ubuntu as a platform of opportunity for application developers.
Creating a comprehensive platform that delights both our users and our developers who want to deliver content to that platform is critical for Ubuntu to be successful. In a perfect world we want to ensure that we can capture the creativity and imagination of developers and have them be able to deliver this creativity easily and effectively to Ubuntu. We have a tremendous Free Software platform, and today most of that picture is complete, but this ARB review process is unfortunately one of the bottlenecks, and this page seeks to capture the challenges so we as a community can work together to resolve them.
The ARB is a critical component in Ubuntu and many of it's members have performed admirable and exhaustive work. To be clear: these challenges are not a reflection on the current ARB members, but an opportunity to make the ARB experience more pleasureable and efficient for both developers and members of the ARB. Resolving these issues will be essential for us to be able to offer Ubuntu as an attractive platform for application developers.
The Goal of This Page
The goal of this page is to gather the challenges together into one cohesive place so we can brainstorm and plan solutions to these challenges. Many of these topics will form the basis of discussions at the up-coming Ubuntu Developer Summit in Oakland in May 2012.
Statistics
To illustrate some of these challenges, here are some statistics from the 8th February until the 31st March. In this period the board reviewed and published 4 applications:
- Framingham Calculator
- Unity Graphic Design lens
- harmonySEQ
- LEDs
When we add these to the previously published apps in the 12.04 cycle, this makes a total of 7 applications published for Ubuntu 11.10. 5 applications were rejected in this period (BooruPy Loadr, nSnake, fcitx, Gestimedia, and TerraView 4.2).
The current ARB application queue is 70 submissions. The queue has grown by 19 apps (+35%) since the beginning of this period (since 8th Feb). [This is not accurate, the current queue is 20 apps. Where are these numbers coming from? -Allison]
The conclusion I am drawing here is that despite the admirable contributions of some members of the ARB, the level of demand for application authors to have their applications reviewed and published to the Ubuntu Software Center is far outstripping the throughput of these contributing ARB members. This is resulting in applications languishing in the queue for weeks and months and delivering a sub-optimal experience for app developers.
Potential Issues
This section lists some potential inefficiencies that we can use as a basis for discussion.
Application Prioritization
The board does not have any kind of prioritization facility in MyApps that indicates that App A may be of more interest/more important/more valuable (or however else we define priority) than App B. I think given there are more apps than time, some kind of prioritization would be helpful to ensure that the apps of most interest to Ubuntu users get to the top of the list.
Potential Solutions
Make the queue open and allow the community to go in and upvote/downvote and add comments and for the ARB to take this content into considerable. Of course it could be gamed, but I suspect it would provide at least indications of interest and popularity. -- JonoBacon
Regular Queue Review
There is no regular cadence around queue reviews. This has meant that the ARB have been taking a more ad-hoc approach of picking apps whenever each member has time as opposed to regular meetings and reviewing and moving the submissions along.
Potential Solutions
A regular queue review (either on a mailing list or on IRC) that assessing the current queue (and any priorities) and assigns the work across different ARB members. -- JonoBacon
Defining Scope
I didn't see this myself but a member of my team observed that some ARB members have fixed bugs in some of the apps in the interests of resolving the issues to get them through the queue. While an admirable contribution, time spent fixing bugs in one app takes time away from other apps getting a review.
Potential Solutions
Define the scope of the ARB reviews to be purely based upon assessing the submitted content and not (a) resolving bugs/issues or (b) resolving packaging issues. -- JonoBacon
Being Stricter With Needs-Information
Applications that are waiting upon feedback from a developer languish in the queue when a developer fails to respond. This causes the queue to be cluttered with content that has no practical means of moving forward as it is blocking on the developer. We have experienced this elsewhere in the project with bugs and we generally auto-close reports with no feedback within a certain time-frame.
Potential Solutions
I think we should put a strong "if there is no response within a week, it gets rejected". This will put the responsibility in the developers hands to be timely, which I think is reasonable given that the ARB is a community team offering their spare time to provide these reviews. -- JonoBacon
Review Assessment Criteria
We should discuss what kind of criteria the ARB are using to review apps in the queue. I suspect there are areas in which the ARB could provide a lighter-weight review while still providing the assurances they provide around quality and security. As an example, in my (humble) opinion, if an application installs, can be removed, and does not pose a security risk, it should get through. If the application quality itself is poor, the ratings and reviews will tell that story to our users.
Potential Solutions
Review https://wiki.ubuntu.com/AppReviewBoard/Review/Guidelines again to see what efficiencies can be made there. -- JonoBacon
Expectations Of ARB Members
While some members of the ARB have contributed extensively to reviewing content in the queue, some members have contributed little and not had the time to participate. There are two issues here (a) a strong Ubuntu value has always been that our governers step down if they don't have the time to contribute within the expectations of a particular governance board; how do we ensure that we are maintaining a board of participants who can fulfill their responsibilities, and (b) we should revisit the expectations of the time required as an ARB member as I suspect this has changed since the inception of the board a few years back.
Potential Solutions
In terms of (a) we should re-enforce this core value with the ARB and ask those who don't have the time to gracefully step down so another community member can step in and help. As with any board, we should also take a regular review to ensure everyone is able to provide the time to participate and check in with those who are not able to commit the time to discuss their level of participation. -- JonoBacon
In terms of (b), we should review the experience of the existing ARB and the typical rate of apps coming into the queue and adjust our expectations for a typical ARB member. -- JonoBacon
ARB Capacity is too low
The current ARB members are unable to keep on top of the queue as it continues to grow (the statistics demonstrated this). Aside from some of the efficiencies that are discussed elsewhere on this page, I suspect we simply need a bigger ARB who can provide further hands on deck.
Potential Solutions
Have no upper limit on ARB members and have an outreach campaign to get more members participating. -- JonoBacon
Promote, grow and enable the Ubuntu App Review Contributors team to help with reviews. -- DavidPlanella
Review Packaging Requirements
The initial idea of the app developer process was to provide a streamlined way to submit applications, which was less strict than the workflow for submitting apps to the archive, and that would empower app authors to easily submit regular updates of their software. Some of the current technical packaging requirements have defeated that purpose, be it on the side of the app developers because our tools don't support well these requirements, be it on the side of the ARB, as they are burdened with additional work to fix untested submissions.
Potential Solutions
Start enforcing the requirement of accepting only PPA submissions, to put less work on the ARB and reduce the back and forth communication traffic when a submission fails to build. -- DavidPlanella
As a follow-up to the previous point, escalate the MyApps bug to add support for choosing a PPA for Free Software Submissions -- DavidPlanella
Drop the /opt requirement, as it is a significant technical barrier both for the submitter and the ARB. -- DavidPlanella
AppReviewBoard/PreciseReview (last edited 2012-04-19 15:27:33 by static-50-46-157-62)