RatingsAndReviews

Revision 25 as of 2010-11-26 17:59:25

Clear message

This is a living specification for ratings and reviews in Ubuntu Software Center. A branch implementing this feature should not be merged to trunk until the server side is implemented and reliably maintained.

Rating and reviewing software involves three software components:

  1. (1) Ubuntu Software Center itself

  2. (2) the review service, an Internet server that receives, stores, and publishes reviews

  3. (3) the Ubuntu Single Sign-On service.

Use cases not specified yet

  • Displaying reviews
    • when some of them are for an older or newer version
    • when the connection is interrupted while downloading reviews
    • on an Ubuntu Web site
    • on the software vendor’s Web site
  • Removing a review
    • because of its content
    • because of its age

Use cases currently specified

Submitting a review

This process begins when you activate the “Review…” button on an “Installed Software” item screen.

If the sign-on details have not been remembered from a previous review or purchase, (1) Ubuntu Software Center should open an alert for the (3) Ubuntu Single Sign-On authentication/registration process, using the summary text “To review software, you need an Ubuntu Single Sign-On account.”.

This alert should eventually be specified and implemented as a separate library. Compare with the Ubuntu One control panel specification.


Erratum: “Remember password” should be “Sign on automatically next time”

The “Password:” field should be insensitive whenever “I have an Ubuntu Single Sign-On account” is not selected. The “Continue” button should be insensitive whenever the “E-mail address:” field does not contain a valid e-mail address, or when “I have an Ubuntu Single Sign-On account” is selected and the “Password:” field is empty. In addition, whenever there is not Internet connection, the “Continue” button should be insensitive and the bottom leading corner of the alert should feature a mini error icon and the text “Not connected to the Internet.”. (The appearance or disappearance of the error message should not resize the alert; it should always have been large enough to show the whole message.)

Test case: In a session where you have not signed in, choose “Write a review”; the sign-in dialog should appear. Disconnect from the Internet; the “Not connected to the Internet” error message should appear, and “Continue” should become insensitive. Reconnect to the Internet; the error message should disappear, and “Continue” should become sensitive.

When you activate “Continue”, the “Continue” button should become insensitive, and:

  • If you chose “I want to register for an account now” or “I’ve forgotten my password”, USC should launch your preferred Web browser to the appropriate Web page, while a spinner and the text “Opening browser…” appears in the bottom leading corner of the alert for five seconds (providing feedback at least until the browser appears). Once the five seconds are up, the spinner and text should disappear, and the alert should reselect “I have an Ubuntu Single Sign-On account”, ready for you to enter your details once you have closed the browser window.
  • If you chose “I have an Ubuntu Single Sign-On account”, a spinner and the text “Signing in…” should appear at the bottom leading corner, and the “Cancel” button should become “Stop”. (The change in label should not change the size of the button; it should always have been large enough for either label.)



    If the sign-in process stops unsuccessfully, either because you activated “Stop” or because of an error, the spinner and progress text should disappear, “Stop” should change back to “Cancel”, and “Continue” should become sensitive (unless there is no Internet connection). Then:

    • If you activated “Stop”, “Cancel” should also become insensitive for one second (in case you double-clicked on “Stop” by mistake).
    • If the e-mail address or password was incorrect, the error sound (if any) should play, the primary text of the alert should change to “Your Ubuntu Single Sign-On details were incorrect. Try entering them again.”, the “Stop” button should change back to “Cancel”, the “Continue” button should become sensitive, and the “E-mail address:” field should be focused.
    • If there was any HTTP error while signing in, the error sound (if any) should play, and a mini error icon and the text “Couldn’t connect to the sign-on service.” should appear in the bottom leading corner of the alert.

Once you are signed in, the alert should close, and a dialog without a parent window should appear for submitting the review.

This dialog should be resizable; resizing it should resize only the review text field. That field should be focused by default.

The dialog’s title should be of the form “Review {Item Title}”.

Within the dialog, first should be displayed the icon, title, and version number of the software.

Next should be a “Review by:” item showing your name as registered with the Single Sign-On service (as a subtle hint that you are accountable for your words).

The review text field should accept multiple paragraphs, and should use automatic spell-checking. It should be unlabelled, but have the accessible name “Review text”, and the visible caption: “Be brief and informative. Don’t post bug reports.”

The “Summary:” field should accept a single line of text.

The “Rating:” field should consist of five stars plus a caption. By default, no stars should be filled. Whenever the field is focused, the Left and Right keys should increase or decrease the rating in one-star increments, and the 1 through 5 keys should change the rating to that many stars. The caption should reflect the star you are mousing over, if you are, or otherwise the rating you have currently set: “” (no rating yet), “Awful” (1), “Poor” (2), “Adequate” (3), “Good” (4), or “Excellent” (5). Setting a rating level, whether using a pointing device or a key equivalent, should highlight the new rating in a similar visual style as when clicking a checkbox or radio button.

The “Cancel” button should have Esc as its keyboard equivalent.

The “Publish” button should have Enter as its keyboard equivalent. Activating “Publish” should submit the review.

When you activate the “Publish” button, progress should be shown using a spinner and the text “Submitting review…” in the bottom leading corner, exactly as for the sign-on dialog.

review-submit-progress.jpg

Stopping and errors should similarly be handled exactly as for the sign-on dialog.

review-submit-error.jpg

Submitting a review should send this information to the review server:

  • via URL, the name of the repository, package, and version number
  • your Single Sign-On credentials
  • the language of your locale.

Once a review is submitted successfully, the review dialog should close, USC should automatically navigate to display the relevant software item screen (if it was not being shown already), that screen should scroll smoothly to reveal all of your review, and the background color of the review should fade from a “new” color to the standard color.

in another language

(1) For now, Ubuntu Software Center should assume that you are writing the review in the same language Ubuntu is using in its interface, and it should specify that language when submitting the review.

(2) The review service should store the language specified for every review.

Can we auto-detect the language on the server side, like Google Translate does?

with empty review text or summary, or no rating

(1) The “Publish” button in should be sensitive only when there is text in both the review field and the Summary field and you have set a rating.

(2) The review service should reject with an error any review that does not have all of review text, summary, and a rating.

containing too much text

Review text should have at most 5000 characters, and summaries should have at most 80 characters.

(1) In Ubuntu Software Center:

  • Whenever the number of characters in the review field is 4900 or more, the maximum number of characters remaining should appear as a caption aligned with the trailing side of the field. Numbers from 100 to 50 should use the default text color; 0 or less should use the alert text color; and 50 to 0 should be a proportionate mixture of the two colors.
  • Whenever the number of characters in the review field is 60 or more, the maximum number of characters remaining should appear as a caption aligned with the trailing side of the field. Numbers from 20 to 10 should use the default text color; 0 or less should use the alert text color; and 10 to 0 should be a proportionate mixture of the two colors.

    review-submit-length.jpg

    Test case: Paste 5027 characters into the review text field. The number “-27” should appear in red. Hold down Backspace. The number should steadily increase. After it passes zero, it should steadily get blacker.

(2) On the server side, the review service should reject with an error any review that has too many characters in either field.

containing a very long word

There should be no limits on the length of the words used in a review. Any extra word breaking needed should be done when displaying reviews.

containing inappropriate whitespace

There should be no limits on the whitespace used in a review. Any whitespace collapsing needed should be done when displaying reviews.

when there is no Internet connection

(1) Whenever there is no Internet connection, the “Publish” button in any review dialog should be insensitive, and at the opposite side of the dialog should be a mini error icon and the text “Not connected to the Internet.”.

If there is a connection error of any sort while submitting a review, the “Publish” button should become sensitive again (unless there is no Internet connection), and at the opposite side of the dialog should be a mini error icon and the text “Couldn’t connect to the review service.”. That text should disappear when you next use any control in the dialog (for example, change the contents of any of the fields, or click “Publish” again).

for many programs in quick succession

Ubuntu Software Center should not attempt to limit the number or speed of your reviews. (Any attempt at flooding will likely be done outside USC anyway.)

(2) The review service should handle flooding appropriately — for example, by silently ignoring excessive numbers of reviews from the same person, and/or by automatically flagging them for review and possible mass deletion.

for the same software a second time

If you try to review an item and (1) Ubuntu Software Center knows that you have already reviewed that version of that item, it should behave as if you are correcting the review.

This might change in future, if we let people submit multiple reviews in different languages.

Correcting a review

How do you choose to correct a review in the first place?

If you choose to correct one of your reviews, (1) Ubuntu Software Center should pre-fill the review dialog with the review text, summary, and rating from that review. Whenever all of them are exactly the same as they were originally, the “Publish” button should be insensitive. If a revision is submitted, (2) the review service should treat it as a replacement for the previous review.

If someone has flagged a review for moderation, editing it should not cause the review service to clear those flags, but it should record the fact that the review was edited after the flagging. Should the review service record every version?

Presenting reviews

Throughout the “Get Software” section, a list view item should have at its trailing end a hyperlink consisting of stars for the median rating for the item, plus text of the form “17 reviews”. Activating the hyperlink should navigate to the software item screen and scroll to its “Reviews” section. In addition, text of the form “ (X stars from Y reviews)” should be present at the end of the item’s accessible description — for example, “An application for managing botanical collections (4 stars from 3 reviews)”.

item-row-available-unselected.jpg

When an item is being installed or removed, the progress bar in the list view should replace any rating/reviews link.

Similarly a software item screen should have in its top trailing corner a hyperlink of exactly the same form, which does the same thing. In the “Installed Software” section, this area should instead have a “Rate & Review…” button that begins the review process.

The software item screen in the “Get Software” section should also contain a “Reviews” section.

  • If there are no reviews for the item, it should consist of a single paragraph with the sentence “None yet.” If you have installed the current version of the software now or in the past, this should be followed by the link “Be the first to review it.”

  • Otherwise, it should consist of the five most relevant reviews for the item, followed by a “Write a review” link. “Most relevant” in this sense means the most recent reviews for the current version, with any remainder being the most recent reviews of the next most recent version, then the next most recent, and so on.

get-software-item-reviews.jpg

(Omitting reviews in the “Installed Software” section should discourage, without preventing, people from biasing their ratings and reviews based on the ratings and reviews already submitted. They can still refer to the same item in the “Get Software” section if they want to see the current ratings and reviews.)

In the “Get Software” section, an item that has any reviews in a software list view should have a link consisting of (a) stars representing the average rating and (b) light text of the form “7 reviews”. Activating the link should navigate to the software item screen and scroll to its Reviews section.

containing very long words

If any word in a review is longer than a whole line, (1) Ubuntu Software Center should break the word at the end of the line. The main pane should not scroll horizontally.

containing inappropriate whitespace

When (1) Ubuntu Software Center presents a review:

  • Any series of consecutive whitespace characters that contains at least two line breaks of any sort (e.g. U+00A, U+000C, or U+000D) should be treated as a blank line and nothing else.

  • Any series of consecutive whitespace characters that contains one line break of any sort (e.g. U+00A, U+000C, or U+000D) should be treated as a line break and nothing else.

  • Any series of consecutive whitespace characters that does not contain a line break of any sort should be treated as a single space and nothing else.

Test case: Submit a test review that contains two paragraphs separated by 50 carriage returns and a few tab characters. When displayed, the two paragraphs should be separated only by a blank line.

in different languages

(1) Opposite the “Reviews” heading, in the software item screen, should be an unobtrusive option menu. The menu should contain items for any languages, language families, and language variants that have any reviews for this software item. The order of the items should be:

  • “All languages”;
  • your locale’s language family, e.g. “English”;

  • your locale’s language variant, if any, e.g. “English (United Kingdom)”;

  • a separator, if there are items both above and below it to separate;
  • any other languages/families/variants, in alphabetical order.

For example, if your locale is pt_BR, and the current item has four reviews, three pt_PT and one en_AU, the menu should have items “Todas as línguas”, “Português”, a separator, “Inglesa”, “Inglês (Austrália)”, and “Português (Brasil)”.

The initial setting for the menu should be “All languages”. If you change the setting, a spinner should appear next to the menu, until the reviews in the newly selected language/family/variant are loaded and replace those currently displayed. If the replacement reviews fail to load, the menu should return automatically to its previous setting. The menu selection should persist across items and across USC sessions.

When there are many reviews

When there are more than ten reviews, in your current language selection, for the same version of an item as is available to you, (1) Ubuntu Software Center should display only the most recent ten. Below them should be a button, as wide as the whole reviews column but with subtle background and border, labelled “Show more reviews”. Activating the button should load and fade in the next batch of reviews, below the ones that are already displayed. If there are even more reviews available, the button should slide down to make room for the new reviews; otherwise it should fade out before the reviews fade in.

review-present-batch.jpg

when there is no Internet connection

(1) Ubuntu Software Center should update and cache the median rating, and the number of reviews, for all items in a software source whenever it updates the list of software in that source. It should update and cache the median rating, the number of reviews in each language, and the batch of most recent reviews for your current language setting, for an individual item whenever you visit the screen for that item and there is an Internet connection. The cache should have a maximum size.

If there is no Internet connection when you navigate to a software item screen for which there are reviews but none of them are cached, the part of the screen that would normally display reviews should instead say simply “Connect to the Internet to see {number of} reviews.”. If an Internet connection starts working while still displaying that screen, the reviews should load and appear in-place without moving the rest of the screen.

Similarly, when (but only when) there is no Internet connection while USC is presenting one batch of reviews and the next oldest batch is not cached, the “Older” link should be insensitive, and next to it should be the text “Connect to the Internet to see more reviews.”.

Flagging reviews as inappropriate

Below each review should be a “Flag” link. Activating the link should invoke the Ubuntu Single Sign-On authentication/registration process, if necessary, using the summary text “To flag a review as inappropriate, you need an Ubuntu Single Sign-On account.”. The dialog should also feature a “Just Hide It” button; activating this button should cancel the process, but Ubuntu Software Center should then also immediately hide the review and remember indefinitely that you do not want to see that review again.

review-report-sign-on.jpg review-report.jpg

Once you are signed in, a parent-less dialog should appear for flagging the review.

It should have a “Why is this review inappropriate?” menu with choices “Unspecified”, “Offensive language”, “Infringes copyright”, “Not about this software”, and “Other”.

The “Cancel” button should have Esc as its keyboard equivalent.

The “Flag” button should have Enter as its keyboard equivalent. It should be sensitive only if the reason is not “Unspecified” and you have entered text in the details field.

This dialog should handle progress and HTTP errors in the same way as the review dialog.


Old notes

This is a means for users to communicate with other users about software. It is not specifically a means of communication with Ubuntu developers.

User stories

  • Mikaela has just finished playing Briquolo and wants to review it. She opens the Ubuntu Software Center and finds Briquolo, and clicks the "Rate & Review..." button. A dialog appears for her to sign in to her Ubuntu Single Sign-On account. She doesn't have one, so she goes through the registration process. (What does this look like?) She is then presented with a dialog to enter her rating and her review comments.

  • Sandro is on a dial-up connection. He is looking through the games available for Ubuntu, and comes across Briquolo. When he navigates to the Briquolo information screen, the basic information appears immediately, with Mikaela's review appearing at the bottom of the page a few seconds later.
  • Tillie is browsing the available software in the Ubuntu Software Center when she comes across a review that uses inappropriate language. She clicks the "Report as Inappropriate" button. The review is immediately and persistently hidden for her, and her report is also sent to Launchpad for a moderator to review.

Launchpad registry

* Single signon to review automatically giving user a LP account

  • email address and user name
  • SSO will validate email address
  • insert LP account w/validated email so we don't try to validate twice
  • how to autogenerate a user name?
  • how to tell user he has an LP account?

* Do we need web ui in LP to input reviews via LP?

  • exposed only through API?
  • enter only via desktop?
  • email address state: validated but not yet used?

* Model work to support R&R in LP for an almost pure desktop experience (user doesn't know LP exists) * UI work in LP to manage R&R in the LP web ui * Expand the existing concept of PersonalStanding to R&R?

  • allows us to hide a swath of reviews tied to a spammer

* Since users will abuse R&R to report bugs, maybe we can expose to the driver a "turn this review into a bug report" (or link, or dup it, etc). * quickly bug# (via jdo, this is the awesomeness) * Architectural ideas

  • we already have couchdb. throw reviews into there and let launchpad process it offline; allows for offline reviews w/automatic sync
  • rabbitmq?
  • may not work for downloading because we don't want to force every desktop to have the whole review database
  • authentication data

* Radical idea: this data doesn't live in LP. LP is a client of it

  • Potentially, users wouldn't need an LP account to interact with it (SSO yes, LP no)

* R&R database are static (i.e. the same for everybody asking about the same app/language/distro)

  • No need for personalization, so why hit LP for read access?
  • could be generated every 5 minutes & pushed to an http (anonymous) url

  • no anonymous access to the LP api

* I want to see my reviews immediately but I can't see yours for 3 hours can be solved on the client (again, couchdb fits in nicely)

  • ratings.tgz generated every 6 hours. use archive mirrors
  • reviews.tgz separate? reviews are much larger

* Pretty safe bet that this will be huge * A user can only review once? Supercede?

  • don't bake review once into the data model
  • user can edit their review
  • keep data model open for multiple reviews

* Couch farm to aggregate as necessary to handle load, but LP sees one thing (the root of the tree) * Notifications - we need to think about it

  • by default, no notifications at all

* We happen to be the magic conduit for users to be talking to other users * Reviewing the reviews - not for Lucid

What a review contains

  • Review text
    • Is it a good idea to require text along with a rating?
      • Gives an idea of the person giving the rating
      • but would also make it more likely to have reviews with very little content ("good.")
      • option: make comments non-mandatory, use the stars for average rating, but hide the "empty" comments
      • If comments were mandatory, that would make approval/disapproval of other reviews more important
      • Comparisons: Android Marketplace, Get Satisfaction, Amazon
  • What language the review was in
    • Maybe prioritize reviews for the same language, but still show reviews for other languages
  • Who reviewed it
  • What version they had installed
    • Maybe prioritize reviews for current version, but still show reviews for previous versions
    • Maybe someone can replace their review of a package with a review for a newer version
    • Someone might review the versions in 10.04 LTS and in 11.10 separately
  • Rating (stars)
  • Maybe ratings abusers could share the Launchpad idea of "standing"
  • Maybe add helpful/unhelpful ratings to reviews themselves? (Lucid+n)
  • Link a review to a bug report
    • or possibly "convert to bug", like with questions
    • developers could comment on particular reviews (and have their comments highlighted)
    • look at brainstorm/amazon, they have similar issues to solve
  • "The issue you raised in this review, I've fixed it in this PPA I've just made"
  • Language used
    • Application average ratings may differ legitimately based on language (if an application is poorly localized, or has issues with e.g. metric system vs. other system)
    • Tag the language used, show it first.

Sending the data from Ubuntu Software Center

  • Use couchdb to store ratings and reviews offline, and they get sent to Launchpad when you're online
  • could have webkit frame do the signup process for launchpad within software center
  • One rating per user, but make it editable (or supercedable), especially when user gets new version of software

Getting the data into Ubuntu Software Center

  • New API needed
  • What to do without internet connection?
  • Cached?
    • Could cache the ratings (very small amount of data) but download the text reviews on demand
    • could put the ratings in the packages file and give them with apt-get update
      • or in the archive metadata index
  • This may increase Web requests to Launchpad, a lot
    • How will launchpad deal with millions of queries?
      • If we don't care about instantaneous reviews
    • Maybe Launchpad is not necessarily the right home for this information in the first place?
      • CouchDB + desktopcouch for review content (desktopcouch would easily give us offline reviewing)
        • scalable, allows local version to be more up to date than global one (my review I just made gets up immediately, a review I just flagged gets hidden immediately)
      • Launchpad could be a client of that standalone database
      • If the data doesn't live in Launchpad, that solves the problem of needing a Launchpad account
    • Data doesn't need to be up to date, but does need to include reviews you just did

    • If ratings are stored in the archive, they'll get mirrored by Ubuntu mirrors
    • Show the latest n reviews by default, require another action to get more

Other issues

  • Maybe we should allow some packages to be immune from review?
    • analogous to YouTube's "ratings have been disabled for this video"

  • Default packages pose specific problems
    • Less incentive to review
    • Some people use software without knowing what it is.
    • Harder to gauge "real" popularity
  • "Help" > "Review This Application..."

  • How does this relate to popcon?
    • Show popularity as well as ratings?

    • Maybe we don't need popcon any more?
    • but popcon doesn't tell lies about installation
    • Maybe we could ask about popcon when someone is submitting their first review.
    • Would be much more useful if we knew how often people had been using the program.

  • When would people review software?
    • Suggest that you review it when you're uninstalling it?
      • May catch them at a bad time.
  • What e-mail, if any, would a review submitter receive as a result?
    • Developer comments on their review?

What the Launchpad team needs

  • To know whether Launchpad should be used for this at all
  • Data model for Lucid
  • Data model for the future
  • API requirements

What the Ubuntu Software Center engineers need

  • Structured data that maps a package name to a rating (ratings in archive index?)
  • Reviews in given languages (not all languages)
  • Anonymous access to the Launchpad API! ... or an export of the data somewhere
  • Estimates of how big the data will be (per archive/component/etc)
  • have a LP API to be able to add new reviews
  • export Reviews-$lang.gz with the latest 5 comments and add "click here for
    • more" button in the s-center - maybe depends on the size of the db ???

Further work

  • research couchdb and if it can be used for the ratings
    • (couchdb-karmic-main-$lang)

Discussion topics

• what to allow

  • 1 ratings (1-5 stars) 2 multi-language reviews (like amazon, multiple ones?, just one wiki like? 3 comments? review the reviews
    • Merge ratings and comments?

• where allow changes?

  • ∘ software-center ∘ LP web ∘ both? what first?

• do we need a new LP page or can/should we re-use https://launchpad.net/ubuntu/lucid/+package/update-manager ? • data needs to be per-binary package, per-distro release (per arch?) • how should s-c get the read-only data?

  • ∘ ratings are small -> can go into a index file (small) ∘ reviews can be potentially very big if we allow multiple ones

    • ‣ query LP via launchpadlib for every page? ‣ make LP export a static page/data cache that is updated every n-minutes? ‣ export them all as a index file ‣ cache on the client? ‣ use clever JS on the client with webkit (cross domain JS issues?)
    • couchdb/desktopcouch - maybe not LP

• offline capability? for cdrom only media ?

Implementation design

mvo and barry worked out an implementation design.