20130206

Summary

LINK: https://wiki.ubuntu.com/UbuntuFriendly/Meetings (SergioMeneses, 01:03:56) LINK: https://launchpad.net/hexr (cprofitt, 01:05:12) LINK: https://launchpad.net/~hexr-dev/+members#active (cprofitt, 01:07:21)

  • wiki clean up

LINK: https://wiki.ubuntu.com/UbuntuFriendly/ (cprofitt, 01:09:58) ACTION: SergioMeneses will review the Ubuntu Friendly wiki (cprofitt, 01:11:30)

  • improvements
    • submissions (cprofitt, 01:14:28)
    • checkbox (cprofitt, 01:24:58)

ACTION: roadmr will produce a flow chart of the current process (cprofitt, 01:49:23) ACTION: next meeting February 19th UTC 17:00 - cprofitt will send a message to the list to get a chair (cprofitt, 01:58:36)

Votes

  • agree that improving feedback on submissions is important in moving foward
    • For: 5 Against: 0 Abstained: 0

Action items

  • SergioMeneses will review the Ubuntu Friendly wiki

  • roadmr will produce a flow chart of the current process
  • next meeting February 19th UTC 17:00 - cprofitt will send a message to the list to get a chair

Action items, by person

  • cprofitt
    • next meeting February 19th UTC 17:00 - cprofitt will send a message to the list to get a chair
  • roadmr
    • roadmr will produce a flow chart of the current process
  • SergioMeneses

People present (lines said)

  • cprofitt (113)
  • roadmr (50)
  • meetingology (16)
  • SergioMeneses (12)

  • pleia2 (12)
  • phillw (11)
  • chilicuil (2)

log

01:00:17 <cprofitt> #startmeeting ubuntu-friendly

  • 01:00:17 <meetingology> Meeting started Wed Feb 6 01:00:17 2013 UTC. The chair is cprofitt. Information about MeetBot at http://wiki.ubuntu.com/meetingology.

    01:00:17 <meetingology>

    01:00:17 <meetingology> Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired

    01:00:35 <cprofitt> #meetingtopic welcome back

    01:00:46 <cprofitt> hello all -- and welcome to the Ubuntu Friendly meeting

    01:01:00 <cprofitt> can everyone that is here for the meeting just let me know you are in the room

    01:02:09 <SergioMeneses> o/

    01:02:12 <phillw> o/

    01:02:40 <cprofitt> thanks for coming everyone

    01:02:51 <cprofitt> our agenda is:

    01:02:59 <pleia2> o/

    01:03:15 <cprofitt> hexr, wiki cleanup, how to improve parts of the Ubuntu Friendly system

    01:03:24 <cprofitt> your questions and any advice/ideas you all have

    01:03:49 <cprofitt> baloons are you here?

    01:03:56 <SergioMeneses> #link https://wiki.ubuntu.com/UbuntuFriendly/Meetings

    01:04:29 <cprofitt> alright... I might have baloons take over if he comes in, but let me tell you a bit about hexr

    01:04:51 <cprofitt> #meetingtopic hexr

    01:05:12 <cprofitt> #link https://launchpad.net/hexr

    01:06:30 <cprofitt> hexr, from my understanding, will potentially make use of the same data Ubuntu Friendly does, but it will do so to see what harware components have been tested

    01:06:50 <cprofitt> so it will seek to find out of a specific Nvidia card has been tested or a specific wireless card

    01:07:06 <cprofitt> and not look at systems, but components

    01:07:21 <cprofitt> #link https://launchpad.net/~hexr-dev/+members#active

    01:07:36 <cprofitt> the developers are listed on that page

    01:08:24 <cprofitt> I think knowing that could help Ubuntu Friendly add some more functionality to the site -- where people might be able to sort systems based on a desired video card or wireless card...

    01:08:43 <cprofitt> for now it is just something for us to know as we move forward

    01:09:22 <cprofitt> #topic wiki clean up

    01:09:41 <cprofitt> how many of you have had time to look at the wiki page for UF?

    01:09:58 <cprofitt> #link https://wiki.ubuntu.com/UbuntuFriendly/

    01:10:53 <cprofitt> would anyone be willing to take on reviewing the pages to ensure that they are up-to-date?

    01:11:00 <SergioMeneses> cprofitt, we can put some icons

    01:11:04 <SergioMeneses> cprofitt, I can do it

    01:11:30 <cprofitt> #action SergioMeneses will review the Ubuntu Friendly wiki

    01:11:30 * meetingology SergioMeneses will review the Ubuntu Friendly wiki

    01:11:34 <cprofitt> thanks SergioMeneses

    01:11:45 <cprofitt> please feel free to ask for help from others as you do that

    01:12:05 <cprofitt> if anyone else knows of pages that need updates please let SergioMeneses or me know

    01:12:14 <SergioMeneses> cprofitt, ok

    01:12:26 <chilicuil> ok =)

    01:12:36 <cprofitt> #topic improvements

    01:13:12 <cprofitt> I know from personal experience we have some things to improve moving forward

    01:13:21 <cprofitt> phillw: did you want to discuss some of the thoughts you had?

    01:13:56 <cprofitt> anyone else?

    01:14:28 <cprofitt> #subtopic submissions

    01:14:34 <phillw> cprofitt: just to mention that you and laptop team are quite similar and I hope there is a good flow of informations between you both Smile :)

    01:14:47 <cprofitt> phillw: I would hope so as well.

    01:15:01 <cprofitt> One of the things that got me involved was submissions

    01:15:29 <cprofitt> right now it is my understanding that submissions are not getting in to UF

    01:15:35 <cprofitt> but I am not sure why that is

    01:15:40 <pleia2> phillw: hm, I thought the laptop testing team was for testing Ubuntu on laptops for upcoming releases, not reporting on how well supported laptops are in current releases

    01:16:31 <SergioMeneses> Im totally agree with pleia2,

    01:17:10 <phillw> pleia2: ergo, one a release is out, that information is available to what is then the 'current' release?

    01:17:21 <phillw> *once*

    01:18:00 <cprofitt> I would like to see two things done to improve in regards to submissions

    01:18:05 <pleia2> phillw: but it would be results for when it was in development, so maybe the graphics card for that one laptop finally did end up working once the release happened, the results would be from when it wasn't working Confused :\

    01:18:15 <pleia2> for example

    01:18:39 <phillw> i see, I'll have a chat with the laptop team Smile :)

    01:18:49 <cprofitt> pleia2: I agree.. the results from pre-release could would not be invluded in Ubuntu Friendly

    01:19:19 <pleia2> they're both valueable and needed projects, for sure, I just thing they provide different data to different audiences

    01:19:25 <pleia2> s/thing/think

    01:19:28 <cprofitt> but the laptop testing team could, upon release, be a group of people who ran the tests for UF

    01:19:36 <pleia2> cprofitt: indeed!

    01:20:05 <cprofitt> and from my understanding the database we are pulling from might actually be the same -- but with different criteria for what information is included

    01:20:13 <cprofitt> and also differences in how it gets displayed

    01:20:21 <pleia2> it makes sense for both projects to test the same things

    01:20:27 <cprofitt> but I would need one of the developers to give me more information on that

    01:20:52 <cprofitt> I would like to see two things done to improve in regards to submissions

    01:21:19 <cprofitt> 1. some system be put in place to make it easy for those submitting results to know that the results were processed

    01:21:31 <cprofitt> if the result was invalid - let them know why

    01:22:05 <cprofitt> as a person who has submitted results and never saw my results in UF I know how that can cause someone to lose interest in contributing

    01:22:52 <pleia2> +1

    01:23:13 <cprofitt> 2. I would also want to have a group of people that could look at invalid results to ensure that how the data is being processed is accurate -- that the automated system is not makring valid results as invalid

    01:23:30 <cprofitt> this is particularly important as hardware changes...

    01:23:52 <cprofitt> failure to test a card reader when there is no such device present should not invalidate results

    01:24:04 <cprofitt> any other comments in regards to submissions?

    01:24:58 <cprofitt> #subtopic checkbox

    01:25:10 <cprofitt> that brings me to one more idea

    01:25:24 <cprofitt> this one is not mine, but one that has been discussed in the qa channel

    01:25:52 <cprofitt> there may be a need to refine checkbox a bit to allow for a 'simple' test

    01:26:06 <cprofitt> or in the case of hexr a test of specific hardware

    01:26:36 <cprofitt> the simple test would be an abbreviated test to make sure that basic things like wireless work etc.

    01:27:07 <cprofitt> I also had cases when I tested where the automated test 'failed' itself, but I knew the device was working

    01:27:39 <cprofitt> I think there should be some sort of mechanism for making a note of that issue built in to checkbox so that changes can be addressed with those issues

    01:28:03 <cprofitt> having devices 'fail' a test when they are actually working undercuts people's faith in the process and the results.

    01:28:20 <cprofitt> any thoughts in that area?

    01:28:32 <roadmr> o/

    01:28:43 <cprofitt> yes, roadmr

    01:29:02 <roadmr> hello! first, if an automated test fails when the device is working, that means a bug that we'd like to get fixed

    01:29:21 <roadmr> I understand this adds a lot of friction to things, so maybe thinking of ways to make this more effortless may be worthwhile 01:29:31 * cprofitt nods

    01:29:42 <roadmr> second, the automated tests are in general more reliable but they are also quite valuable in high-volume environments

    01:30:07 <roadmr> if we assume that people can be trusted, we can potentially use manual test cases instead of automated ones that are most unreliable

    01:30:30 <roadmr> even so, it would be good to collect some information that can be automatically analyzed, maybe server-side

    01:30:46 <roadmr> this way we can potentially re-score a submission given the raw data, even after-the-fact

    01:30:51 <pleia2> admittedly there is some trust to an extent already in the testing (you can hear a sound, etc)

    01:30:59 <cprofitt> I too, would prefer to get the automated tests working... I think human interaction always introduces some potential for error

    01:31:08 <roadmr> this would however introduce a "we're collecting stuff and sending it up to canonical" factor that some people don't like

    01:31:19 <cprofitt> I do like rescoring a submission after the fact

    01:31:33 <roadmr> so this is an area for improvement but we always have the option of doing this manually

    01:31:47 <cprofitt> roadmr: in this case I think the 'collecting data' part is voluntary

    01:31:54 <cprofitt> at least with checkbox

    01:32:15 <roadmr> cprofitt: yes, well in order to determine what's going on, some extensive logs would need to be collected 01:32:22 * cprofitt nods

    01:32:30 <roadmr> cprofitt: but keeping the user in control would be the best

    01:32:48 <roadmr> pleia2: hehe, we sought to remove the human factor from that by adding an automated audio_test Smile :) so the computer listens to itself

    01:32:56 <cprofitt> could that be an optional part of checkbox -- that a user would opt-in to collecting additional data?

    01:32:57 <roadmr> anyway, those are my ideas on this

    01:33:32 <pleia2> roadmr: does it work? I guess I haven't run it in a bit Smile :)

    01:33:35 <cprofitt> I appreciate the feedback and information roadmr

    01:33:35 <roadmr> cprofitt: sure, with the current checkbox architecture it feels a bit complex to do, but we can always change that, we should bend the tool to the needs, not the other way around

    01:33:51 <roadmr> pleia2: as long as you have a microphone and speakers, it works and it's very reliable

    01:33:59 <pleia2> nice

    01:34:15 <cprofitt> roadmr: +1 we want to make it easy for people to collect data for this type of use -- improving user experience

    01:34:58 <cprofitt> roadmr: do you have any insight as to why results are not getting included in to Ubuntu Friendly or is that outside of your scope?

    01:35:42 <roadmr> cprofitt: I have a pretty good idea why, yes

    01:35:52 <cprofitt> could you go a bit in to that for us?

    01:36:13 <roadmr> cprofitt: two components are involved in this, the first is checkbox, which runs the tests, assembles a report and pushes that to launchpad

    01:36:45 <roadmr> the second is the UF website/application, which pulls the reports (submissions) from launchpad and adds the results to the friendly database (what you see in the website) - does the scoring, aggregation and such

    01:37:14 <roadmr> for a submission to appear at all, it needs to contain (and pass) all tests identified as "core"

    01:37:30 <roadmr> the basics for a computer to be usable, if any core tests are failed the submission will not appear in UF

    01:37:52 <roadmr> if it passes the "core" tests, it gets 3 stars, and any additional tests it passes increase its rating, up to 5 stars

    01:38:06 <roadmr> a 5-star rating means that all the components we test work perfectly

    01:38:14 <roadmr> so why would a submission not appear?

    01:38:39 <roadmr> some of the core tests are automated, and checkbox is not very good at 1) telling you one of them failed, and 2) letting you rerun/amend the results

    01:39:19 <roadmr> checkbox itself has no idea of core tests, so it can't tell you "hey, your card reader didn't work and that means you won't make it into UF - please rerun this before submitting"

    01:39:42 <cprofitt> is there currenlty anyway for a person who has submitted the test to see their submissions on launchpad?

    01:40:16 <roadmr> that's one reason, the second is that since the UF website needs some love, there may have been a bit of "drift" into the tests checkbox submits and performs, the UF site has not been updated as checkbox whitelists (sets of tests to run) change

    01:40:23 <roadmr> those would be the two main reasons

    01:40:36 <roadmr> cprofitt: there's a cryptic url that you an access to see your submissions, let me find it

    01:40:53 <cprofitt> so, it sounds like the key is giving the user some feedback as to if their device failed a core test... and also allowing for them to resubmit

    01:40:53 <roadmr> cprofitt: https://launchpad.net/~/+hwdb-submissions

    01:41:37 <roadmr> cprofitt: yep, again, something that's a bit difficult to do with the current checkbox architecture which is very linear Sad :( think of the difference between linear and non-linear video editing

    01:41:47 <roadmr> in a sense, with checkbox we can only ffwd, rewind, and look at the results at the very end 01:41:58 * cprofitt nods

    01:42:00 <roadmr> with no/little chance of going back to a specific test.

    01:42:08 <cprofitt> exactly...

    01:42:28 <cprofitt> it is also my understanding there is no ability to 'pause' a test and come back and pick up where you left off

    01:42:49 <roadmr> there is, but it's very awkward heheh

    01:42:51 <cprofitt> roadmr: is there currently any team that reviews the failed submissions?

    01:43:07 <roadmr> what you do basically is murder checkbox, when you restart it does its best to pick up where it left off

    01:43:30 <roadmr> it gives you three choices: rerun the last test, skip to the next one (in case the last test outright crashed the system), or start anew 01:43:42 * cprofitt nods

    01:43:50 <roadmr> the message is quite terse and the choices are somewhat ambiguous, so most people are/will be confused by this

    01:44:36 <roadmr> cprofitt: reviewing failed submissions, not really, no. It can be done but pretty much the only person who can and knows how to do it is jedimike, by looking at how the UF results importer processed a submission

    01:44:48 <roadmr> he'd have more specifics on how to do it, but it's done on an ad-hoc basis

    01:45:00 <cprofitt> roadmr: thanks

    01:45:07 <roadmr> np Smile :)

    01:45:14 <cprofitt> I asked because I saw that as part of the launchpad team description

    01:45:27 <cprofitt> does anyone else have any questions for roadmr?

    01:46:19 <cprofitt> I would like to set an initial priority for the team moving forward

    01:46:33 <cprofitt> I think that we have two high value items

    01:47:12 <cprofitt> 1. Improving the UF website process so that there is some feedback to the person who submitted the system as to the fact the results were processed but failed

    01:47:35 <cprofitt> 2. Documenting the current process - so others can understand it as we start to look at improving it

    01:47:42 <cprofitt> are there any other items people see?

    01:48:47 <cprofitt> roadmr: would you be willing to procude a flow chart of what you described that we could them use on the wiki?

    01:48:57 <cprofitt> just a rough outline of how the process currently flows?

    01:49:08 <roadmr> cprofitt: sure

    01:49:22 <roadmr> cprofitt: there's some documentation (an outline plus more detailed links) here: https://friendly.ubuntu.com/what-is-ubuntu-friendly/

    01:49:23 <cprofitt> #action roadmr will produce a flow chart of the current process 01:49:23 * meetingology roadmr will produce a flow chart of the current process

    01:49:53 <cprofitt> #vote agree that improving feedback on submissions is important in moving foward

    01:49:53 <meetingology> Please vote on: agree that improving feedback on submissions is important in moving foward

    01:49:53 <meetingology> Public votes can be registered by saying +1, +0 or -1 in channel, (private votes don't work yet, but when they do it will be by messaging the channel followed by +1/-1/+0 to me)

    01:49:57 <cprofitt> +1

    01:49:57 <meetingology> +1 received from cprofitt

    01:50:12 <roadmr> +1 definitely heh

    01:50:12 <meetingology> +1 definitely heh received from roadmr

    01:50:33 <chilicuil> +1

    01:50:33 <meetingology> +1 received from chilicuil

    01:50:35 <SergioMeneses> +1

    01:50:35 <meetingology> +1 received from SergioMeneses

    01:50:44 <cprofitt> pleia2: ?

    01:50:48 <cprofitt> phillw: ?

    01:50:56 <phillw> +1

    01:50:56 <meetingology> +1 received from phillw

    01:51:05 <phillw> sorry wasnm't sure if i was to vote Smile :)

    01:51:23 <cprofitt> #endvote

    01:51:23 <meetingology> Voting ended on: agree that improving feedback on submissions is important in moving foward

    01:51:23 <meetingology> Votes for:5 Votes against:0 Abstentions:0

    01:51:23 <meetingology> Motion carried

    01:52:01 <cprofitt> I think for now we should move towards that; after we get that feedback improved we need to look to make some improvements to checkbox as roadmr mentioned

    01:52:16 <cprofitt> any other topics, questions or thoughts?

    01:52:37 <roadmr> o/

    01:52:42 <cprofitt> yes, roadmr

    01:53:00 <SergioMeneses> when is our next meeting?

    01:53:10 <roadmr> hehe, one thing that we *can* potentially do with current checkbox is make it submit directly to a (possibly updated) UF application, so feedback is quicker

    01:53:33 <roadmr> this needs writing a "plugin" and an accompanying web application, so it would need some coordination

    01:53:33 <cprofitt> +1 roadmr that should be explored

    01:53:51 <roadmr> we're still on time to make that happen before Ubuntu Feature Freeze, but we'd have to start working soon 01:54:13 * cprofitt nods

    01:54:24 <roadmr> ideally we sould ask jedimike if there's a way to submit directly to the UF application, because we may not have a lot of manpower to come up with a new one

    01:54:31 <roadmr> sould/should

    01:54:46 <cprofitt> SergioMeneses: I would like us to meet again in two weeks -- I think to be fair to some of the UTC

    01:54:49 <roadmr> that may be a quick way to reduce the decoupling between submission and actual result processing

    01:55:22 <cprofitt> folks we should have the meeting earlier; though that would require a different person to chair the meeting

    01:55:36 <SergioMeneses> perfect

    01:55:41 <cprofitt> roadmr: I know jedimike has spoken to me about that so I will try to touch bases with him about that

    01:55:56 <cprofitt> are any of you in the UTC 0/+1 TZ?

    01:57:01 <phillw> 'im on utc

    01:57:01 <cprofitt> alright... I will send a message to the list and see if I can get a person in that TZ to chair a meeting.

    01:57:02 <roadmr> cprofitt: I'm EST (UTC-5)

    01:57:19 <cprofitt> phillw: would you be willing to chair a meeting on the 19th?

    01:57:27 <cprofitt> roadmr: I am on your TZ as well

    01:57:33 <phillw> I can if no one steps forward Smile :)

    01:57:49 <SergioMeneses> Im UTC-5

    01:57:55 <cprofitt> phillw: ok, I will send the request to the list and you will be the fall back

    01:58:36 <cprofitt> #action next meeting February 19th UTC 17:00 - cprofitt will send a message to the list to get a chair 01:58:36 * meetingology next meeting February 19th UTC 17:00 - cprofitt will send a message to the list to get a chair

    01:58:36 <phillw> what time on 19th are you thinking of?

    01:59:05 <phillw> yeah, 1700 utc would be okay if no one steps forward.

    01:59:09 <SergioMeneses> 17 utc 01:59:12 * cprofitt nods

    01:59:20 <cprofitt> yes, SergioMeneses so noon our time

    01:59:52 <SergioMeneses> cprofitt, it's not problem for me

    01:59:58 <cprofitt> thanks SergioMeneses phillw pleia2 chilicuil roadmr for attending. I think this has been very productive for an initial start

    02:00:04 <cprofitt> #endmeeting

full logs

UbuntuFriendly/Meetings/20130206 (last edited 2013-02-06 02:11:13 by cprofitt)