Meeting started by brendand at 15:02:08 UTC. The full logs are available at http://ubottu.com/meetingology/logs/ubuntu-meeting/2011/ubuntu-meeting.2011-10-24-15.02.log.html .
Jedimike explained changes that are to be made to the Ubuntu Friendly scoring system so that systems that would be awarded 1 star due to tests that weren't run will be rejected by the system. This will avoid the large number of systems currently on the site that were not tested properly (over 250).
Everyone agreed this was a good idea. Some issues were raised around how this change will be communicated to the user. Making tests which are essential unskippable in Checkbox would be the best solution but would be a large change to SRU into Oneiric. Guidance on the UF website seems to be the agreed way to do it at the moment, but brendand and akgraner concurred that some users don't necessarily make first contact with Ubuntu Friendly through the website so might miss this memo.
ACTION: cr3 to add gremlins/detect to checkbox (brendand, 15:47:42) ACTION: jedimike to prepare a small usability survey about the site (brendand, 16:01:24)
Meeting ended at 16:01:59 UTC.
- cr3 to add gremlins/detect to checkbox
- jedimike to prepare a small usability survey about the site
Action items, by person
- * cr3 to add gremlins/detect to checkbox
- * jedimike to prepare a small usability survey about the site
People present (lines said)
- brendand (75)
- jedimike (32)
- cr3 (27)
- ara (22)
- roadmr (19)
- akgraner (15)
- meetingology (5)
15:02:08 <brendand> #startmeeting Ubuntu Friendly Squad
15:02:08 <meetingology> Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired
15:02:33 <ara> hello!
15:02:36 <jedimike> hi
15:02:36 <roadmr> hey
15:02:45 <brendand> The agenda for today is:
15:02:47 <brendand> Scoring system - jedimike
15:02:47 <brendand> AOB
15:02:58 <brendand> #topic Scoring system - jedimike
15:03:10 <jedimike> o/
15:03:11 <brendand> jedimike - proceed
15:03:29 <jedimike> currently, the rules for rejection and failure of submissions are this:
15:03:29 <jedimike> * If you skip all tests in a core category, we reject the submission
15:03:30 <jedimike> * If you skip an individual test in a core category that required external hardware that not everyone might have (writable media, for example), we ignore that test and apply a penalty to the score, but still allow that category to pass if all other tests pass
15:03:30 <jedimike> * If you skip an individual test in a core category that you should have ran, we fail the category and score the system 1 star
15:03:30 <jedimike> That last rule seems far too harsh. I think that skipping an individual test in a core category that we don't allow you to skip should result in the submission being rejected, as we have a lot of 1 star systems that are only 1 star because someone skipped a test.
15:03:43 <jedimike> 251 systems, to be precise
15:04:03 <ara> o/
15:04:06 <jedimike> ..
15:04:13 <brendand> ara - go ahead
15:04:48 <ara> I agree that we should change 3rd rule to reject the submission
15:04:56 <ara> if a non-skippable test was skipped
15:05:04 <jedimike> o/
15:05:14 <ara> and I would put something in the particpate page to make things clearer
15:05:27 <ara> ..
15:05:41 <brendand> back to you jedimike
15:06:34 <jedimike> I'd also put something in that intermediate "report a problem" page that says, "Can't find your results?" and explains why we reject submissions, and directs them towards either results tracker, or where we need them to look, to view their submission
15:06:36 <jedimike> ..
15:06:54 <ara> o/
15:07:09 <brendand> ara - shoot
15:07:41 <ara> yes, I agree. I think that we don't need to be really fancy on letting people know exactly why it was rejected
15:07:53 <ara> a nice explanation of the basics should be fine
15:08:07 <ara> and yes ,that explanation should be at Participate and Report a problem
15:08:28 <brendand> o/
15:08:29 <ara> not sure if pointing to the submissions is needed, though
15:08:29 <ara> ..
15:09:01 <roadmr> o/
15:09:04 <brendand> just a quick piece, to say that in the long term if we could enforce running of the tests through the checkbox UI that would be nice
15:09:05 <brendand> ..
15:09:21 <brendand> roadmr - go ahead
15:09:29 <roadmr> we should make sure that reasons for rejecting submissions are made abundantly clear
15:09:31 <cr3> o/
15:09:43 <roadmr> we already see people wondering why their submissions take so long to appear in UF
15:09:54 <roadmr> so if we just start "eating" them there's bound to be some complaints
15:10:00 <roadmr> it goes to being as transparent as possible
15:10:16 <jedimike> o/
15:10:29 <roadmr> ideally I'd like to see checkbox say "this submission is not UF-complete so it'll be rejected" or something to the point (i.e. instant feedback) but it may not be possible to do
15:10:37 <roadmr> still this is something to be looked into, for transparency's sake.
15:10:38 <roadmr> ..
15:10:40 <ara> o/
15:10:55 <brendand> cr3 - your turn
15:10:57 <cr3> brendand: dude, I want to write a testing game! seriously though, it doesn't really need to contribute to UF though but might be nice
15:11:00 <cr3> ..
15:11:20 <brendand> ok, after that, back to ara
15:11:34 <cr3> not jedimike?
15:11:45 <ara> jedimike it is
15:11:47 <brendand> cr3 - you're right
15:11:57 <brendand> jedimike?
15:12:02 <cr3> brendand: /unignore jedimike
15:12:18 <jedimike> I agree with roadmr that we do need something to tell the users at least which of their submissions made it and which didn't
15:12:25 <jedimike> we don't need to break down scores or anything
15:12:36 <jedimike> but it might improve the quality of submissions
15:12:54 <jedimike> if we were able to say "X submission didn't get accepted because you skipped audio/xyz"
15:13:17 <jedimike> and it would cut out the "why didnt my results get listed" questions totally"
15:13:18 <jedimike> ..
15:13:40 <brendand> ara, your turn (and apologies to jedimike)
15:14:19 <ara> so, I agree with roadmr, but those are ideas for 12.04 LTS, when we improve the UI
15:14:32 <ara> but we need to take them into account, of course
15:15:12 <ara> with the UI changes we can make a much better work in letting people know what they have to run for the submission to be accepted in UF
15:15:33 <jedimike> o/
15:15:44 <ara> for now, an explanation of what is a core category and how it works might help
15:15:44 <ara> ..
15:15:58 <brendand> jedimike, go ahead
15:16:45 <jedimike> if we're not going to make the submissions available to the users through UF, can I put in a feature request for results tracker so that a user's test runs are linked to from their page on there?
15:16:45 <jedimike> ..
15:17:03 <brendand> o/
15:18:17 <brendand> we need to remember that even though ubuntu-friendly is in beta, if we don't keep things transparent we risk creating frustration for users
15:18:53 <jedimike> o/
15:19:39 <brendand> one challenge i think we have is that we are able to update UF pretty much as we want, but not the source of the tests (i.e. checkbox)
15:20:10 <brendand> so we need to avoid making changes in UF which really require support from checkbox
15:20:26 <brendand> this is one of those i think
15:20:27 <brendand> ...
15:20:37 <brendand> jedimike - you can go now
15:20:46 <akgraner> oops sorry I am late
15:21:33 <brendand> akgraner - no problem. we're discussing a possible change to the scoring system. 15:21:50 * akgraner catches up
15:22:00 <brendand> akgraner - thanks
15:22:03 <brendand> jedimike?
15:22:05 <jedimike> just to say, transparency is good and at the moment it's not clear if your submission has made it or not, and I think if it's possible to make that change on results track to link the user's page to their test submissions it would help (even helps me respons to bug reports!)
15:22:11 <jedimike> ..
15:22:41 <brendand> o/
15:22:45 <akgraner> o/
15:23:02 <brendand> akgraner - you go first
15:23:47 <akgraner> Is ther anyway to say - not let the skipped question count against the over scoring and not somewhere that these scores don't include skipped questions
15:24:06 <akgraner> overall scoring I meant
15:24:11 <akgraner> ...
15:25:35 <akgraner> I mean I might have skipped a test just b/c I didn't have a USB stick handy that doesn't mean it didn't pass
15:25:43 <brendand> akgraner - the issue is that, at the moment tests like the audio ones are skippable. and it seems a lot of people are skipping them. but if we only have one submission for a system then we need to make a call about what that means.
15:25:58 <ara> me needs to leave now
15:26:02 <brendand> bye ara 15:26:06 * ara will read the minutes tomorrow
15:26:07 <ara> cheers 15:26:19 * brendand continues
15:26:21 <akgraner> brendand, I go back and re-do the test for the ones I skip once I find all my stuff
15:26:22 <jedimike> o/
15:27:04 <brendand> we don't want to say that a skipped test means that the component must work, but neither can we say for sure that it means it doesn't
15:27:20 <brendand> so the best thing to do is probably to not accept these submissions at all
15:27:44 <akgraner> but some people don't have external monitors
15:27:57 <akgraner> so you would disregard their test on that one point
15:28:00 <brendand> akgraner - but for tests which need special equipment we already have a different rule, so your system can still get a good score if you didn't test external monitor or usb
15:28:14 <cr3> o/
15:28:21 <akgraner> ah ok
15:28:30 <jedimike> akgraner: if a test requires special equipment, like a USB stick or external monitor, we allow people to skip it without failing that component
15:29:06 <brendand> ..
15:29:19 <brendand> jedimike - you can go ahead now
15:29:28 <akgraner> gotcha - sorry I didn't know that. /me is quiet now
15:29:46 <jedimike> was just going to say that and that we need to make that clear on the participate page
15:30:21 <jedimike> and implement what ara said about making it clear that if you skip tests that don't require external equipment
15:30:28 <jedimike> your submission may not be included in the site
15:30:29 <jedimike> ..
15:31:10 <brendand> akgraner - an example of a test you *can't* skip is audio/alsa_record_playback_internal, since it doesn't require extra equipment
15:31:18 <brendand> cr3, your turn
15:31:20 <cr3> if I understand correctly, skippable tests may affect scoring between 3-5, but non-skippable tests are those that may affect scoring between 1-3
15:31:23 <cr3> I think someone made a point that non-skippable tests should be enforced in the checkbox UI so that people don't get surprised with a crappy score between 1-3
15:31:26 <cr3> ..
15:31:34 <brendand> cr3 - that was me
15:31:49 <brendand> cr3 - but the problem is we'd need to SRU that change in
15:32:22 <brendand> o/
15:32:55 <cr3> brendand: oneiric is beta, it could be argued that we're really targeting precise with the ultimate ninja solution
15:33:27 <brendand> actually, if i could expand on this. i'm not sure i feel too awesome about the 'skippableness of tests being encoded in the u-f site itself
15:33:42 <brendand> it should really be encoded in checkbox
15:34:25 <brendand> ..
15:35:43 <brendand> it seems that most people agree we need to reject submissions that don't have all the tests run that must be run
15:36:03 <brendand> the question is to what extent do we guide the users about this?
15:36:36 <brendand> ideal would be to enforce it in checkbox, but in the short term it needs to be stated on the UF website
15:36:40 <brendand> but...
15:37:19 <brendand> it's important to remember that the assumption that everyone will engage with UF directly through the site is probably wrong
15:37:46 <brendand> so some users may not even read the participate page
15:37:47 <brendand> ...
15:38:32 <akgraner> o/
15:38:46 <brendand> akgraner - go ahead
15:39:22 <akgraner> brendand, your assumption is right - many people find out about system testing on their computer, run the test, *then* find out about the site
15:40:03 <akgraner> and I know a people who never read documentation (sadly) but it does happen...
15:40:22 <cr3> o/ 15:40:32 * brendand points to self
15:40:38 <brendand> cr3 - your turn
15:41:13 <cr3> it would be nice to see the number of submissions to launchpad before and after ubuntu friendly was announced, there was already a large number of submissions coming in before probably from people just discovering checkbox and running it for the heck of it
15:41:17 <cr3> ..
15:42:39 <brendand> cr3 - indeed
15:43:44 <cr3> brendand: modulo hardware certification submissions, of course
15:45:14 <cr3> brendand: is it time for aob?
15:45:32 <brendand> seems like everyone is done on this topic
15:45:36 <brendand> #topic AOB
15:46:17 <brendand> AOB?
15:46:31 <cr3> o/
15:47:12 <brendand> cr3 - yep
15:47:13 <cr3> checkbox needs tests for gremlins because they've been misbehaving in my computer lately
15:47:16 <cr3> ..
15:47:42 <brendand> #action cr3 to add gremlins/detect to checkbox 15:47:42 * meetingology cr3 to add gremlins/detect to checkbox
15:48:10 <cr3> "Does your gremlin have a shiny coat, and I don't mean a pimp coat!"
15:49:12 <roadmr> o/
15:49:51 <brendand> roadmr - is this still about gremlins?
15:49:58 <roadmr> nope
15:50:07 <brendand> roadmr - then please, go ahead
15:50:28 <roadmr> heheh, just wanted to get a feel for how useful people think the number of "raters" is in the UF front page
15:50:58 <cr3> between 1 and 10, I'd say 11
15:51:11 <roadmr> ... basically just that, should that information be available at a glance, or is it ok to move it to the system's detail page for instance?
15:51:14 <roadmr> ..
15:52:31 <brendand> roadmr - i think it depends on whether people are using it the way we imagine they would (i.e. to get an idea of how 'reliable' the results are)
15:53:54 <roadmr> hmm maybe at some point we could conduct a poll on how useful people visiting UF think each bit of information is
15:54:06 <brendand> maybe at UDS?
15:54:39 <roadmr> I was thinking something like those "would you like to answer a poll to help improve our site?" - to get a feel from normal, average users
15:54:57 <roadmr> I think the UDS crowd may be too biased towards preferring a lot of information
15:55:15 <cr3> roadmr: maybe it can be used as an excuse to make people aware of the site in the first place
15:55:31 <brendand> roadmr - i hate those things but yeah, you're right it should be from normal site users
15:55:34 <cr3> roadmr: although, since ara will be making a presentation, everyeone will innevitably know about it. maybe ask her to announce the poll?
15:55:36 <roadmr> cr3: that too! and to encourage a bit more participation, which is always good
15:56:28 <cr3> I think that deserves an action item for ara if we all agree a poll would be useful. I don't see how it could hurt
15:57:08 <cr3> from the little usability testing I've done, it's been tremendously useful
15:57:35 <cr3> if anyone intends to prepare some usability testing sessions, you might like to read: Rocket Surgery Made Easy
15:58:21 <roadmr> we'd have to have the poll ready before UDS (i.e. in about 10 days)
15:58:21 <cr3> ... and don't let jedimike run the session otherwise he'll jedi mind trick everyone to say what he wants to hear
15:59:04 <brendand> who'd like to prepare some questions?
16:00:07 <brendand> considering we're nearly out of time, i propose cr3
16:00:29 <cr3> brendand: I thought jedimike would be better placed for writing the questions, no?
16:00:43 <brendand> jedimike - do you want to do that?
16:01:01 <jedimike> brendand: yeah
16:01:24 <brendand> #action jedimike to prepare a small usability survey about the site 16:01:24 * meetingology jedimike to prepare a small usability survey about the site
16:01:37 <brendand> ok, i think our time here is up
16:01:49 <brendand> thanks everyone for your participation
16:01:59 <brendand> #endmeeting