RatingsAndReviews

Differences between revisions 36 and 37
Revision 36 as of 2010-11-29 14:47:29
Size: 32058
Editor: eth0
Comment: more details of what the moderation interface should contain
Revision 37 as of 2010-11-30 14:58:48
Size: 28386
Editor: eth0
Comment: sign-on dialog moved to https://wiki.ubuntu.com/SingleSignOn/SignOnDialog
Deletions are marked like this. Additions are marked like this.
Line 35: Line 35:
''This alert should eventually be specified and implemented as a separate library. Compare with the [[http://docs.google.com/View?id=dfkkjjcj_82fb6d42hk|Ubuntu One control panel specification]].''

{{attachment:review-single-sign-on.jpg}}<<BR>>
''Erratum: “Remember password” should be “Sign on automatically next time”''

The “Password:” field should be insensitive whenever “I have an Ubuntu Single Sign-On account” is not selected. The “Continue” button should be insensitive whenever the “E-mail address:” field does not contain a valid e-mail address, or when “I have an Ubuntu Single Sign-On account” is selected and the “Password:” field is empty. In addition, whenever there is not Internet connection, the “Continue” button should be insensitive and the bottom leading corner of the alert should feature a mini error icon and the text “Not connected to the Internet.”. (The appearance or disappearance of the error message should not resize the alert; it should always have been large enough to show the whole message.)

{{attachment:review-single-sign-on-offline.jpg}}

'''Test case:''' In a session where you have not signed in, choose “Write a review”; the sign-in dialog should appear. Disconnect from the Internet; the “Not connected to the Internet” error message should appear, and “Continue” should become insensitive. Reconnect to the Internet; the error message should disappear, and “Continue” should become sensitive.

When you activate “Continue”, the “Continue” button should become insensitive, and:

 * If you chose “I want to register for an account now” or “I’ve forgotten my password”, USC should launch your preferred Web browser to the appropriate Web page, while a spinner and the text “Opening browser…” appears in the bottom leading corner of the alert for five seconds (providing feedback at least until the browser appears). Once the five seconds are up, the spinner and text should disappear, and the alert should reselect “I have an Ubuntu Single Sign-On account”, ready for you to enter your details once you have closed the browser window.

 * If you chose “I have an Ubuntu Single Sign-On account”, a spinner and the text “Signing in…” should appear at the bottom leading corner, and the “Cancel” button should become “Stop”. (The change in label should not change the size of the button; it should always have been large enough for either label.)

 {{attachment:review-single-sign-on-progress.jpg}}<<BR>><<BR>>

 If the sign-in process stops unsuccessfully, either because you activated “Stop” or because of an error, the spinner and progress text should disappear, “Stop” should change back to “Cancel”, and “Continue” should become sensitive (unless there is no Internet connection). Then:

  * If you activated “Stop”, “Cancel” should also become insensitive for one second (in case you double-clicked on “Stop” by mistake).

  * If the e-mail address or password was incorrect, the error sound (if any) should play, the primary text of the alert should change to “Your Ubuntu Single Sign-On details were incorrect. Try entering them again.”, the “Stop” button should change back to “Cancel”, the “Continue” button should become sensitive, and the “E-mail address:” field should be focused.
 
  * If there was any HTTP error while signing in, the error sound (if any) should play, and a mini error icon and the text “Couldn’t connect to the sign-on service.” should appear in the bottom leading corner of the alert.

Once you are signed in, the alert should close, and a [[SoftwareCenter#def-dialog|dialog]] ''without'' a parent window should appear for submitting the review.
Once you are signed in, a [[SoftwareCenter#def-dialog|dialog]] ''without'' a parent window should appear for submitting the review.

This is a living specification for ratings and reviews in Ubuntu Software Center. A branch implementing this feature should not be merged to trunk until the server side is implemented and reliably maintained.

Ratings and reviews are a way for Ubuntu users to communicate with other Ubuntu users about software. They are not specifically a means of communication with Ubuntu developers or application developers.

Rating and reviewing software involves three software components:

  1. (1) Ubuntu Software Center itself

  2. (2) the review service, an Internet server that receives, stores, and publishes reviews

  3. (3) the Ubuntu Single Sign-On service.

Use cases not specified yet

  • Displaying reviews
    • when some of them are for an older or newer version
    • on an Ubuntu Web site
    • on the software vendor’s Web site
  • Removing a review
    • because of its content
    • because of its age

Use cases currently specified

Submitting a review

This process begins when you activate the “Review…” button on an “Installed Software” item screen.

If the sign-on details have not been remembered from a previous review or purchase, (1) Ubuntu Software Center should open an alert for the (3) Ubuntu Single Sign-On authentication/registration process, using the summary text “To review software, you need an Ubuntu Single Sign-On account.”.

Once you are signed in, a dialog without a parent window should appear for submitting the review.

This dialog should be resizable; resizing it should resize only the review text field. That field should be focused by default.

The dialog’s title should be of the form “Review {Item Title}”.

Within the dialog, first should be displayed the icon, title, and version number of the software.

Next should be a “Review by:” item showing your name as registered with the Single Sign-On service (as a subtle hint that you are accountable for your words).

The review text field should accept multiple paragraphs, and should use automatic spell-checking. It should be unlabelled, but have the accessible name “Review text”, and the visible caption: “Be brief and informative. Don’t post bug reports.”

The “Summary:” field should accept a single line of text.

The “Rating:” field should consist of five stars plus a caption. By default, no stars should be filled. Whenever the field is focused, the Left and Right keys should increase or decrease the rating in one-star increments, and the 1 through 5 keys should change the rating to that many stars. The caption should reflect the star you are mousing over, if you are, or otherwise the rating you have currently set: “” (no rating yet), “Awful” (1), “Poor” (2), “Adequate” (3), “Good” (4), or “Excellent” (5). Setting a rating level, whether using a pointing device or a key equivalent, should highlight the new rating in a similar visual style as when clicking a checkbox or radio button.

Probably there will be a caption here explaining that you give permission for your review to be republished.

The “Cancel” button should have Esc as its keyboard equivalent.

The “Publish” button should have Enter as its keyboard equivalent. Activating “Publish” should submit the review.

When you activate the “Publish” button, progress should be shown using a spinner and the text “Submitting review…” in the bottom leading corner, exactly as for the sign-on dialog.

review-submit-progress.jpg

Stopping and errors should similarly be handled exactly as for the sign-on dialog.

review-submit-error.jpg

Submitting a review should send this information to the review server:

  • via URL, the name of the repository, package, and version number
  • your Single Sign-On credentials
  • the language of your locale.

Once a review is submitted successfully, the review dialog should close, USC should automatically navigate to display the relevant software item screen (if it was not being shown already), that screen should scroll smoothly to reveal all of your review, and the background color of the review should fade from a “new” color to the standard color.

in another language

(1) For now, Ubuntu Software Center should assume that you are writing the review in the same language Ubuntu is using in its interface, and it should specify that language when submitting the review.

(2) The review service should store the language specified for every review.

Can we auto-detect the language on the server side, like Google Translate does?

with empty review text or summary, or no rating

(1) The “Publish” button in should be sensitive only when there is text in both the review field and the Summary field and you have set a rating.

(2) The review service should reject with an error any review that does not have all of review text, summary, and a rating.

containing too much text

Review text should have at most 5000 characters, and summaries should have at most 80 characters.

(1) In Ubuntu Software Center:

  • Whenever the number of characters in the review field is 4900 or more, the maximum number of characters remaining should appear as a caption aligned with the trailing side of the field. Numbers from 100 to 50 should use the default text color; 0 or less should use the alert text color; and 50 to 0 should be a proportionate mixture of the two colors.
  • Whenever the number of characters in the review field is 60 or more, the maximum number of characters remaining should appear as a caption aligned with the trailing side of the field. Numbers from 20 to 10 should use the default text color; 0 or less should use the alert text color; and 10 to 0 should be a proportionate mixture of the two colors.

    review-submit-length.jpg

    Test case: Paste 5027 characters into the review text field. The number “-27” should appear in red. Hold down Backspace. The number should steadily increase. After it passes zero, it should steadily get blacker.

(2) On the server side, the review service should reject with an error any review that has too many characters in either field.

containing a very long word

There should be no limits on the length of the words used in a review. Any extra word breaking needed should be done when displaying reviews.

containing inappropriate whitespace

There should be no limits on the whitespace used in a review. Any whitespace collapsing needed should be done when displaying reviews.

when there is no Internet connection

(1) Whenever there is no Internet connection, the “Publish” button in any review dialog should be insensitive, and at the opposite side of the dialog should be a mini error icon and the text “Not connected to the Internet.”.

If there is a connection error of any sort while submitting a review, the “Publish” button should become sensitive again (unless there is no Internet connection), and at the opposite side of the dialog should be a mini error icon and the text “Couldn’t connect to the review service.”. That text should disappear when you next use any control in the dialog (for example, change the contents of any of the fields, or click “Publish” again).

for many programs in quick succession

Ubuntu Software Center should not attempt to limit the number or speed of your reviews. (Any attempt at flooding will likely be done outside USC anyway.)

(2) The review service should handle flooding appropriately — for example, by silently ignoring excessive numbers of reviews from the same person, and/or by automatically flagging them for review and possible mass deletion.

for the same software a second time

If you try to review an item and (1) Ubuntu Software Center knows that you have already reviewed that version of that item, it should behave as if you are correcting the review.

This might change in future, if we let people submit multiple reviews in different languages.

Correcting a review

How do you choose to correct a review in the first place?

If you choose to correct one of your reviews, (1) Ubuntu Software Center should pre-fill the review dialog with the review text, summary, and rating from that review. Whenever all of them are exactly the same as they were originally, the “Publish” button should be insensitive. If a revision is submitted, (2) the review service should treat it as a replacement for the previous review.

If someone has flagged a review for moderation, editing it should not cause the review service to clear those flags, but it should record the fact that the review was edited after the flagging. Should the review service record every version?

Presenting reviews

Throughout the “Get Software” section, a list view item should have at its trailing end a hyperlink consisting of stars for the median rating for the item, plus text of the form “17 reviews”. Activating the hyperlink should navigate to the software item screen and scroll to its “Reviews” section. In addition, text of the form “ (X stars from Y reviews)” should be present at the end of the item’s accessible description — for example, “An application for managing botanical collections (4 stars from 3 reviews)”.

item-row-available-unselected.jpg

When an item is being installed or removed, the progress bar in the list view should replace any rating/reviews link.

Similarly a software item screen should have in its top trailing corner a hyperlink of exactly the same form, which does the same thing. In the “Installed Software” section, this area should instead have a “Rate & Review…” button that begins the review process.

The software item screen in the “Get Software” section should also contain a “Reviews” section.

  • If there are no reviews for the item, it should consist of a single paragraph with the sentence “None yet.” If you have installed the current version of the software now or in the past, this should be followed by the link “Be the first to review it.”

  • Otherwise, it should consist of the five most relevant reviews for the item, followed by a “Write a review” link. “Most relevant” in this sense means the most recent reviews for the current version, with any remainder being the most recent reviews of the next most recent version, then the next most recent, and so on.

get-software-item-reviews.jpg

(Omitting reviews in the “Installed Software” section should discourage, without preventing, people from biasing their ratings and reviews based on the ratings and reviews already submitted. They can still refer to the same item in the “Get Software” section if they want to see the current ratings and reviews.)

In the “Get Software” section, an item that has any reviews in a software list view should have a link consisting of (a) stars representing the average rating and (b) light text of the form “7 reviews”. Activating the link should navigate to the software item screen and scroll to its Reviews section.

containing very long words

If any word in a review is longer than a whole line, (1) Ubuntu Software Center should break the word at the end of the line. The main pane should not scroll horizontally.

containing inappropriate whitespace

When (1) Ubuntu Software Center presents a review, any series of consecutive whitespace characters that:

  • contains at least two line breaks of any sort (e.g. U+00A, U+000C, or U+000D) should be treated as a blank line and nothing else.

  • contains one line break of any sort should be treated as a line break and nothing else.
  • does not contain any line breaks should be treated as a single space and nothing else.

Test case: Submit a test review that contains two paragraphs separated by 50 carriage returns and a few tab characters. When displayed, the two paragraphs should be separated only by a blank line.

containing other-directional text

(1) Ubuntu Software Center should present each review using the direction customary for the language it is in. For example, an Arabic review should be presented from right to left.

in different languages

(1) Opposite the “Reviews” heading, in the software item screen, should be an unobtrusive option menu. The menu should contain items for any languages and language families that have any reviews for this software item. The order of the items should be:

  • “All languages”;
  • your locale’s language/family, e.g. “English”;

  • a separator, if there are items both above and below it to separate;
  • any other languages/families, in alphabetical order.

For example, if your locale is pt_BR, and the current item has four reviews, three pt_PT and one en_AU, the menu should have items “Todas as línguas”, “Português”, a separator, and “Inglês”.

The initial setting for the menu should be your language or language family. If you change the setting, a spinner should appear next to the menu, until the reviews in the newly selected language/family are loaded and replace those currently displayed. If the replacement reviews fail to load, the menu should return automatically to its previous setting. The menu selection should persist across items and across USC sessions.

when there are many reviews

When there are more than ten reviews, in your current language selection, for the same version of an item as is available to you, (1) Ubuntu Software Center should display only the most recent ten. Below them should be a button, as wide as the whole reviews column but with subtle background and border, labelled “Show more reviews”. Activating the button should make it insensitive, change its label to “Loading…”, and load and fade in the next batch of reviews, below the ones that are already displayed. If there are even more reviews available, the button should slide down to make room for the new reviews, change its label back to “Show more reviews”, and resume sensitivity. Otherwise, the button should fade away completely before the reviews fade in.

review-present-batch.jpg

when there is an error downloading them

If (1) Ubuntu Software Center tries to download a batch of reviews and fails (because the connection is interrupted, or it times out, or whatever), the label for loading more reviews for that item should change to read “There was a problem downloading reviews. Try again?”

when there is no Internet connection

(1) Ubuntu Software Center should update and cache the median rating, and the number of reviews, for all items in a software source whenever it updates the list of software in that source. It should update and cache the median rating, the number of reviews in each language, and the batch of most recent reviews for your current language setting, for an individual item whenever you visit the screen for that item and there is an Internet connection. The cache should have a maximum size.

If there is no Internet connection when you navigate to a software item screen for which there are reviews but none of them are cached, the part of the screen that would normally display reviews should instead say simply “Connect to the Internet to see {number of} reviews.”. If an Internet connection starts working while still displaying that screen, the reviews should load and appear in-place without moving the rest of the screen.

Similarly, when (but only when) there is no Internet connection while USC is presenting one batch of reviews and the next oldest batch is not cached, the button to load more reviews should be insensitive, and its label should be “Connect to the Internet to see more reviews.”.

Flagging reviews as inappropriate

Below each review should be a “Flag” link. Activating the link should invoke the Ubuntu Single Sign-On authentication/registration process, if necessary, using the summary text “To flag a review as inappropriate, you need an Ubuntu Single Sign-On account.”. The dialog should also feature a “Just Hide It” button; activating this button should cancel the process, but Ubuntu Software Center should then also immediately hide the review and remember indefinitely that you do not want to see that review again.

review-report-sign-on.jpg review-report.jpg

You won’t always need to sign in, so there needs to be a way of hiding reviews from the flag dialog too.

Once you are signed in, a parent-less dialog should appear for flagging the review.

It should have a “Why is this review inappropriate?” menu with choices “Unspecified”, “Offensive language”, “Infringes copyright”, “Not about this software”, and “Other”.

The “Cancel” button should have Esc as its keyboard equivalent.

The “Flag” button should have Enter as its keyboard equivalent. It should be sensitive only if the reason is not “Unspecified” and you have entered text in the details field.

This dialog should handle progress and HTTP errors in the same way as the review dialog.

If you choose “Just Hide It” or “Flag”, the review in question should fade away indefinitely, and anything below it should slide up into place.

Moderating reviews

At any time, (2) the review service should be able to operate in either of two modes: active or passive moderation. In active moderation, every review must be approved manually before appearing to other users, but may still be flagged after that. In passive moderation, we rely on flagging to tell us if a review should be removed. Initially, we should use active moderation.

review-moderating-initial.jpg

https://developer.ubuntu.com/reviews should be a page with access restricted to the moderation team. The page should present a batched list, containing:

  1. any unmoderated flagged reviews, in descending order of number of flags
  2. any other unmoderated reviews submitted while active moderation was in effect.

Why on developer.ubuntu.com? Because it will be the central place for application developers. They’ll sign in to see the status of their applications, both before and after they are accepted into the archives — including any reviews.

In the initial version, each item in the list should show:

  • the item being reviewed
  • the name and e-mail address of the reviewer
  • the date of the review
  • the rating and the review summary
  • the review text
  • the name and e-mail address of the person who flagged the review
  • the date of the flag
  • the basic reason given for flagging the review
  • the full explanation
  • buttons for removing or approving the review.

Later, the moderation system should be enhanced as necessary to deal with the number of reviews:

  • The name of the item should link to a batched list of other reviews for that item.
  • The name of the item should be followed by the number of reviews that:
    • are currently flagged for that item
    • have ever been removed for the item
    • have ever been submitted for the item.
  • The name of the reviewer should link to a batched list of their other reviews.
  • The name of the reviewer should be followed by the number of their reviews that:
    • are currently flagged
    • have ever been removed
    • have ever been submitted.
  • The name of the person who flagged the review should link to a batched list of their other flags.
  • The name of the reviewer should be followed by the number of reviews that they have:
    • currently flagged
    • ever flagged that were removed
    • ever flagged at all.
  • The “Delete” button should be followed by a “Delete and Ban” button for silently dropping any future reviews from that reviewer.

Future work

Rating a review itself

Was this review helpful? Yes/No

Right of reply for developers

Presenting reviews on an external Web site

Atom feed? JSON?

What extra legal requirements are there?


Old notes

Launchpad registry

* Expand the existing concept of PersonalStanding to R&R?

  • allows us to hide a swath of reviews tied to a spammer

* Since users will abuse R&R to report bugs, maybe we can expose to the driver a "turn this review into a bug report" (or link, or dup it, etc). * quickly bug# (via jdo, this is the awesomeness) * Architectural ideas

  • we already have couchdb. throw reviews into there and let launchpad process it offline; allows for offline reviews w/automatic sync
  • rabbitmq?
  • may not work for downloading because we don't want to force every desktop to have the whole review database
  • authentication data

* I want to see my reviews immediately but I can't see yours for 3 hours can be solved on the client (again, couchdb fits in nicely)

  • ratings.tgz generated every 6 hours. use archive mirrors
  • reviews.tgz separate? reviews are much larger

* Pretty safe bet that this will be huge * A user can only review once? Supercede?

  • don't bake review once into the data model
  • user can edit their review
  • keep data model open for multiple reviews

* Couch farm to aggregate as necessary to handle load, but LP sees one thing (the root of the tree) * Notifications - we need to think about it

  • by default, no notifications at all

* We happen to be the magic conduit for users to be talking to other users * Reviewing the reviews - not for Lucid

What a review contains

  • What version they had installed
    • Maybe prioritize reviews for current version, but still show reviews for previous versions
    • Maybe someone can replace their review of a package with a review for a newer version
    • Someone might review the versions in 10.04 LTS and in 11.10 separately
  • Rating (stars)
  • Maybe ratings abusers could share the Launchpad idea of "standing"
  • Maybe add helpful/unhelpful ratings to reviews themselves? (Lucid+n)
  • Link a review to a bug report
    • or possibly "convert to bug", like with questions
    • developers could comment on particular reviews (and have their comments highlighted)
    • look at brainstorm/amazon, they have similar issues to solve
  • "The issue you raised in this review, I've fixed it in this PPA I've just made"
  • Language used
    • Application average ratings may differ legitimately based on language (if an application is poorly localized, or has issues with e.g. metric system vs. other system)
    • Tag the language used, show it first.

Sending the data from Ubuntu Software Center

  • Use couchdb to store ratings and reviews offline, and they get sent to Launchpad when you're online
  • could have webkit frame do the signup process for launchpad within software center
  • One rating per user, but make it editable (or supercedable), especially when user gets new version of software

Getting the data into Ubuntu Software Center

  • New API needed
  • What to do without internet connection?
  • Cached?
    • Could cache the ratings (very small amount of data) but download the text reviews on demand
    • could put the ratings in the packages file and give them with apt-get update
      • or in the archive metadata index
    • CouchDB + desktopcouch for review content (desktopcouch would easily give us offline reviewing)
      • scalable, allows local version to be more up to date than global one (my review I just made gets up immediately, a review I just flagged gets hidden immediately)
    • Data doesn't need to be up to date, but does need to include reviews you just did

    • If ratings are stored in the archive, they'll get mirrored by Ubuntu mirrors
    • Show the latest n reviews by default, require another action to get more

Other issues

  • Maybe we should allow some packages to be immune from review?
    • analogous to YouTube's "ratings have been disabled for this video"

  • Default packages pose specific problems
    • Less incentive to review
    • Some people use software without knowing what it is.
    • Harder to gauge "real" popularity
  • "Help" > "Review This Application..."

  • How does this relate to popcon?
    • Show popularity as well as ratings?

    • Maybe we don't need popcon any more?
    • but popcon doesn't tell lies about installation
    • Maybe we could ask about popcon when someone is submitting their first review.
    • Would be much more useful if we knew how often people had been using the program.

  • When would people review software?
    • Suggest that you review it when you're uninstalling it?
      • May catch them at a bad time.
  • What e-mail, if any, would a review submitter receive as a result?
    • Developer comments on their review?

What the Launchpad team needs

  • To know whether Launchpad should be used for this at all
  • Data model for Lucid
  • Data model for the future
  • API requirements

What the Ubuntu Software Center engineers need

  • Structured data that maps a package name to a rating (ratings in archive index?)
  • Reviews in given languages (not all languages)
  • Estimates of how big the data will be (per archive/component/etc)
  • have a LP API to be able to add new reviews
  • export Reviews-$lang.gz with the latest 5 comments and add "click here for
    • more" button in the s-center - maybe depends on the size of the db ???

Further work

  • research couchdb and if it can be used for the ratings
    • (couchdb-karmic-main-$lang)

Discussion topics

• what to allow

  • 1 ratings (1-5 stars) 2 multi-language reviews (like amazon, multiple ones?, just one wiki like? 3 comments? review the reviews
    • Merge ratings and comments?

• where allow changes?

  • ∘ software-center ∘ LP web ∘ both? what first?

• do we need a new LP page or can/should we re-use https://launchpad.net/ubuntu/lucid/+package/update-manager ? • data needs to be per-binary package, per-distro release (per arch?) • how should s-c get the read-only data?

  • ∘ ratings are small -> can go into a index file (small) ∘ reviews can be potentially very big if we allow multiple ones

    • ‣ query LP via launchpadlib for every page? ‣ make LP export a static page/data cache that is updated every n-minutes? ‣ export them all as a index file ‣ cache on the client? ‣ use clever JS on the client with webkit (cross domain JS issues?)
    • couchdb/desktopcouch - maybe not LP

• offline capability? for cdrom only media ?

Implementation design

mvo and barry worked out an implementation design.

SoftwareCenter/RatingsAndReviews (last edited 2015-04-13 08:09:55 by mpt)