Summary

Test Descriptions

• We discussed if it would be more benefitial to have a more structured format for test cases.
• Example:

Purpose:
   To check that the systems wired network connection functions correctly.

Test Steps:
  1.) Plug an ethernet cable into the ethernet socket on your system.
  2.) Wait for the network indicator to indicate connecting is complete.

Verification:
  1.) Click on the network indicator.
  2.) Under 'Wired Network' does it show a connection name (such as 
'Auto eth0') and not 'disconnected'?

• One of the concerns is that it may be to strict for people, but in the other hand it gives a clear view on what to test and what to expect.
• There were some interesting ideas for future developments (please, refer to the log for those).
• We will continue the conversation on the mailing list.

FAQ

• Daniel wanted to know if it make sense to put more FAQ for contributing
• It turned out that we need better documentation on how to contribute to Ubuntu Friendly
• It was asked to use LP answers for the FAQ, but it was agreed that LP answers is normally more used for technical questions, and the wiki version still makes sense.

Log

 #startmeeting
 [AGENDA] Wording of test descriptions (brendand)
 New FAQs (roadmr) 
* ara loves to see people adding topics to the meeting
<ara> [TOPIC]  Wording of test descriptions (brendand)
 Motbook is lost!
 mootbot, even
 davmor2 Daviey
 Daviey, can you get us the mootbot again, please?
* brendand waiting for mootbot
<brendand> or shall i just go?
<ara> let's wait 2 minutes, to see if it comes back
<brendand> come back mootbot, we didn't mean it !
 davmor2 Daviey
<ara> mmm, seems like Daviey is not around... OK, let's get started without bot :(
 brendand, all yours
<brendand> ok
 Other QA teams in Canonical use a formal test description format for their tests:
<Daviey> ara: I think mootobt-uk has been superseeded now.
<brendand> Something like
<Daviey> ara: Will follow it up.
<-- brendand (~brendand@188-223-22-218.zone14.bethere.co.uk) has left #ubuntu-quality ("Leaving")
<ara> Daviey, thanks :)
 now we lose brendand :)
--> brendand (~brendand@188-223-22-218.zone14.bethere.co.uk) has joined #ubuntu-quality
<ara> brendand, welcome back :)
<cr3> he probalby pasted /quit
<roadmr> awesome
<brendand> no
<cr3> so format test descriptions should start with a slash :)
<brendand> did you get anything from me just now?
<cr3> brendand: nada
<ara> brendand, niente
<brendand> i wanted to say:
 Other QA teams in Canonical use a formal test description format for their tests:
 e.g.
  PURPOSE:
      1. Wired network connection
  STEPS:
      1. Plug in ethernet cable in to rj45 port
      2. Select Test to verify that it's possible to establish both HTTP
 and FTP connections
  VERIFICATION:
      1. System should connect to the wired network automatically
      2. OSD notify should confirm that the connection has been established
      3. The Network Manager icon on the panel should have 2 arrows.
      4. Was the FTP and HTTP connection correctly established?
 Most Checkbox tests use a more informal format:
 e.g.
 Disconnect any external microphones that you have plugged in.  Select Test, then speak into your internal microphone.  After a few seconds, your speech will be played back to you.
  .
  Did you hear your speech played back?
 The question is:
 whether the first format is too formal and would scare potential community testers away from doing UF testing.
<bladernr_> o/
<brendand> especially looking for opinions from non-developers/QA specialists
 ..
<cr3> o/
<ara> bladernr_, you go first
<bladernr_> First, I actually like the idea of the more formal description. It IS more terse, but strictly defines "This is how you run the test. This is exactly what you should see happen"
 Second, I see that being a positive, however, to implement, it would be nice to ensure the GUI doesn't produce scrollbars for every test definition
 rather, enlarge the UI to present a uniform appearance
 ..
<ara> thanks bladernr_
 cr3, your turn
<cr3> 1. The purpose can be implied from the requires line in checkbox, so formalising that as part of a manual process is just additional overhead.
<-- jfunk has quit (Quit: Leaving)
<ara> o/
<bladernr_> o/
<cr3> 2. Each verification test is an actual test, so that formal description is conflating multiple tests in a single one.
<jedimike> o/
<brendand> o/
<roadmr> o/
<cr3> 3. Steps are nice instead of having a long sentence though, but they should only be followed by a single question.
 ..
<ara> OK, I think I go first
 replying to cr3
 1. People have no idea what  a requires line is. Something like network/wired is not user friendly
<cr3> o/
<ara> 2. This is actually one test, as it will be an overhead to create a new one where the steps are the same, but different verification
 ..
 bladernr_, you go
<bladernr_> heh... +1 on ara's comment about 1.  IMO, it's not a good idea to ever think end users (which is what the target is here, not developers) can grasp implications of anything in the actual code.  I doubt many will even ever look in /usr/share/checkbox/jobs to know what the format actually is.
<bladernr_> regarding 2: I think brendan was just using that as an example of a more formal structure, not suggesting conflation, but I agree, we don't want to conflate multiple tests
 and 3: big +1, the long sentence thing has always been a bit of an annoyance :-)
 ..
<akgraner> o/
<ara> jedimike, your turn :)
<jedimike> I want to make sure i'm on the right track here, first
 this is what we're talking about presenting to end users to get them to test their systems
 if so, there are a couple of things that don't really sit right with me
 first, "purpose"
 what we say there isn't a purpose
 in the English language sense
 secondly, I know we don't like long sentences
 and want to make things as simple as possible
 and we're letting things like OSD creep in - is that going to be understood by the people submitting this - the average end user?
 and lastly, something like "Did the network connect automatically" would be great for someone who knew how to check that
 but not for new users who were seeing things for the first time
 I guess what I'm saying is
 we need to be friendly, and often terse things aren't
 perhaps we need headings as in the example, with a "how do I check this" or similar next to them
 so people could see a line that said "Did the network connect automatically"
 and click an icon that says "tell me how I check this happened"
 and perhaps get a screenshot that shows them exactly how to check
 ok, braindump over, sorry if I missed the point ;)
 ..
<ara> thanks jedimike :)
 brendand
<brendand> hey :)
 got a bit of talking to do
 1.) I think it would be better for everyone to ignore the specific contents of the steps
 2.) This is an actual test written probably by someone for whom English might not be their first language
 3.) The question is more about the style of language used.
 4.) I like jedimike's suggestion of being able to get assistance on verifying a test result, but it seems impracticable
 I guess that's it, but the main point is not get hung up on some of the mistakes we're noticing in this test :)
 ..
<ara> thanks brendand, good point
 by turns I guess the next one is roadmr!
<roadmr> yay!
 If we use the requires line as indicated by cr3, we have to make sure that we're not using it for something else, because we also use it to skip the test if a required package is not installed, but we're not testing the package per se, so that would be confusing.
 Also, some requires lines have python code that may be unintelligible to some people.
 As for the formal test description I'm worried that end users will find it intimidating and/or not know what to do. I'm asking 4 questions on the test description, what should the user do if only one fails? mark as failed and describe everything in the comments field?
 Automating complex tests as much as possible should be our goal, "did the network connect automatically" is something we can auto-check. "does the display work correctly?" is much easier to confirm and also much harder to automate, so we should focus on those.
 Finally for the UI, I'd also like to avoid use of long text which leads to scrollbars which lead to madness, but we need to be careful with screen sizes to avoid making checkbox unusable for netbook users (a design parameter should be for it to be usable in 800x600 resolution).
 (yes, I had that all prepared)
 ..
<ara> :)
 thanks roadmr
 cr3, yo!
<cr3> ara, bladernr: about my point #1, we would not stupidly display "required: device.category == 'NETWORK'" in the interface but that requires line would enable us to display "Testing your Intel Corporation 82567LM Gigabit Network Connection" which is way superior to "Wired network connection"
 ara: maybe the point to #2 is that checkbox might benefit from supporting the use case where steps are the same but multiple questions may be asked, which is a manual test pattern we might like to consider
<cr3> jedimike: OSD means Office of the Secretary of Defense, to anyone searching google :)
 ..
<ara> akgraner, your turn
 (cr3, thanks!)
 akgraner, ?
<akgraner> I just wanted to +1 your 1. comment - echo what jedimike said about keeping things as simple as possible about terse things often not being...I like the idea of a link or something if someone wants to know more about the specific test..
 (sorry had to edit what I already wrote everyone already said it)
 ...
<-- jcollado (~javi@canonical/oem/jcollado) has left #ubuntu-quality
<ara> thanks akgraner
 any other comments?
<cr3> o/
<ara> cr3, go ahead
--> jfunk (~jfunk@pool-96-246-63-52.nycmny.east.verizon.net) has joined #ubuntu-quality
<cr3> brendand: to answer your point #4, that was the idea of the matchmaker presented a year ago where the tester could get in contact with the developer interested in having something tested when they are both online
 ..
<ara> o/
 I think that it is important to focus on what we can achieve for the first release of UF to make things simpler, and leave future improvements for UDS
<brendand> o/
 brianchidester brendand
<ara> brendand, go ahead
<brendand> assuming that everyone understands that we're not getting hung up on the specific flaws of *this particular test case*, can we have a vote?
 to terse or not to terse
 ..
<cr3> o/
* ara thinks is a bit early to vote, as we have no idea what the options are
<ara> cr3, go ahead
<cr3> to second ara's point, I'd present a revised version of the test description presented in this channel to the ubuntu-friendly-squad mailing list for iterative review
<cr3> ..
<ara> o/
<cr3> ara: go ahead :)
<ara> cr3, instead of that, if you could, present different alternatives, please (including the ones you don't like ;-) )
--> silicongirl_ (44e85a9f@gateway/web/freenode/ip.68.232.90.159) has joined #ubuntu-quality
<ara> and once discussed we can vote on the next meeting
<jedimike> o/
<ara> for example, same test, with three different syntaxes
<cr3> ara: +1
<ara> ..
 jedimike, go ahead
 [ACTION] cr3 to present different test syntax to the UF mailing list for a particular chosen test
<jedimike> just wanted to voice my agreement with brendand that it's not so much about the problems with this specific test example, it's getting to an agreement on the style of writing. If we hit the right balance of terse and explained well enough for an end user to carry out, we're on a winner
<bladernr_> o/
<akgraner> o/
* ara waits a bit to see if jedimike has finished or not
<jedimike> ..
 (whoops)
<ara> ::)
 bladernr_, go ahead
<-- DigitalFlux has quit (Quit: No Ping reply in 180 seconds.)
<bladernr_> I'd like to append to the action the following: If there are any community volunteers who are willing to also provide sample test descriptions, that would be fabulous. It's one thing for developers to guess at what would make a test easy for a home user, but it's another thing all together for a non-developer to say "This is what I understand, this is somehting I can easily figure out when performing these tests."
 heh... maybe not so wordy, but I'd be interested in seeing non-developer ideas on this as well
 ..
<ara> akgraner, your turn
--> sbeattie (~steve@208-151-246-43.dq1sn.easystreet.com) has joined #ubuntu-quality
<akgraner> I just wanted to know if I use the system test and give comments to my pain points would that be helpful
 I started going through it this weekend but wanted to ask before I spent a lot of time on it
 ...
<brendand> o/
<ara> brendand, go ahead
<brendand> gah, the thought has left my head
<cr3> akgraner: if you could give your comments either on the ubuntu-friendly-squad mailing list or as bugs against the checkbox project on launchpad, either would certainly be helpful
<brendand> oh, i remember
<ara> brendand, you can go
<brendand> i think we should come to agreement on the level of user that will be doing UF testing. writing for the lowest common denominator can cause test description to become bloated, and become less understandable in general
<ara> o/
<cr3> o/
<ara> brendand, ..?
<brendand> after all, we are doing hardware testing - users will need to know at least basic terms to get somewhere with it
 ..
<ara> (sorry)
 OK, I think the level of user of ubuntu friendly should be someone that has a minimum level of geekness
 LoCo teams, Windows power users, Ubuntu members, etc
 ..
 cgregan czajkowski cyphermox ChanServ ChrisGagnon charlie-tca cking cr3
 cr3, go ahead
<cr3> brendand: each test should be a youtube video where people can see how to run the test, which would be recursive if there was ever a test to make sure youtube works :) seriously though, if the test is too hard for an average user, lets cross that bridge when we reach it because I'm confident we can come up with clever solutions
 ..
<ara> thanks cr3
 any other comments?
 OK; let's follow up in the mailing list once cr3 sends the alternatives
 [TOPIC]  New FAQs (roadmr) 
 roadmr, you can go
<roadmr> yay, so I have one new possible question for the FAQ (do I need to be a  member of UF Squad to contribute?), and I think it would be useful to have some links to relevant places where people can start contributing
<cr3> o/
<roadmr> either I or the rest of the team could do this but I wanted some feedback on whether this is useful or worthwhile
 I'd hate to se the FAQ become a dumping ground for *all* questions as opposed to frequent ones :)
 ..
<ara> thanks roadmr
 cr3, yes?
<cr3> just a quick question: is there any reason why the FAQ isn't generated from launchpad answers?
 ..
 everyone doesn't need to rush to answer that :)
<ara> cr3, not really, you can move it there if you want
 but make sure to link it from the wiki page
<cr3> would it help solve roadmr's concern to determine which questions are actually frequent?
<ara> (the landing wiki page)
<roadmr> o/
<ara> cr3, well, most questions there were asked by people on my blog :)
 roadmr, you go
<akgraner> o/
* ara remembers that we have 5 minutes left
<roadmr> I find most questions on LP answers are more technical and support-oriented, and the current FAQ has more generic stuff, I think that's a good distinction
 answers is good for automatically handling that kind of questions
<cr3> o/
<roadmr> but for "meta-questions" about the project in general I think the current wiki page is good
 linking to related questions in LP answers could be worthwhile though
 ..
<cr3> o\
<ara> cr3, you go
 cr3, sorry, akgraner go
<cr3> o\ was meant to be hand down :)
<akgraner> +1 on what roadmr said..
 I'll just email the ml on the other stuff
 ...
<ara> roadmr, akgraner: OK, we would leave it there ;-)
 any other topics?
 or comments on this one?
<cr3> the faq on the wiki page also allows for pretty pictures
 we need pictures of checkbox... and turtles
<ara> OK, last chance for any other topics
 #endmeeting
 Thanks all!
<akgraner> Thanks ara
* ara saves this log as this channel is not even logged :(