TouchTesting

Differences between revisions 2 and 5 (spanning 3 versions)
Revision 2 as of 2013-09-16 12:25:58
Size: 2580
Editor: jibel
Comment:
Revision 5 as of 2013-09-16 15:49:21
Size: 5206
Editor: adsl-98-70-43-80
Comment:
Deletions are marked like this. Additions are marked like this.
Line 13: Line 13:
This describes the daily process of testing. We will flash our devices, look for bugs, then look to write autopilot tests for any fixed bugs to prevent that bug from re-occurring.
Line 14: Line 15:
TODO === Daily Image Testing ===
==== Install the current stable image ====
Flash your phone to the latest stable image (note: we want the "current" image of touch_ro). If you have not yet bootstrapped or unlocked your phone's bootloader, [[https://wiki.ubuntu.com/Touch/Install|see the install page for more information]].
{{{
  $ phablet-flash ubuntu-system
}}}

Alternatively, once you are running the ubuntu image you may perform a system update from the device itself.

==== Check to see what's changed ====
A list of changes between stable images can be [[http://people.canonical.com/~ogra/touch-image-stats/current/|found here]]. The list contains the list of changing packages.

''Need more detailed list of what is actually changing''

==== Test ====
* Run through the list of new features and changed areas of the phone, paying specific attention to spot regressions and verify the features are working.

* Perform smoke testing on a specific packageset of the phone.

* File bugs against any issues encountered

* Remember this is exploratory testing, so try and break things!

==== Report Bugs ====
See the [[https://wiki.ubuntu.com/QATeam/TouchTesting#Reporting_a_bug|Reporting a Bug section below]] for details.

==== Add a testcase for reported bugs ====
Once testing is completed, review the list of reported bugs with the needs-autopilot-test tag. [[https://wiki.ubuntu.com/Touch/CoreApps/Testing/ContributeAutopilotTestcase|Automate a testcase using autopilot]] that fixes the bug to ensure the same bug doesn't reappear.
Line 17: Line 45:
All bugs filed should use the ???? (need to decide) tag. Omer has made available a tool to help with the process of filing a bug report
Line 18: Line 47:
== There is a crash file in /var/crash == == What package should I file against? ==
File against the ubuntu source package for the failed component.

== Filing the Bug ==

=== Using Omer's tool ===
http://paste.ubuntu.com/6098563/

=== There is a crash file in /var/crash ===
Line 61: Line 98:
== There is no crash file == === There is no crash file ===
Line 73: Line 110:
}}}
Line 74: Line 112:
3. Wait until data collection is done.

4. At the question:

{{{
Line 76: Line 119:
  V: View report
[...]
Line 80: Line 124:
3. On your local machine, click on the link or copy/paste it into your browser. Press 'S' or 'K' if you want to copy the report to another machine (for example if networking is not enabled on the device).

3. If you're reporting from the device, on your local machine, click on the link or copy/paste it into your browser.
Line 89: Line 135:

* Add link and information on omer's testing tool

* See if popey's scripts might be useful and add them

* Need better list of what's changing between images

* Add list of who's testing which feature sets / areas

* Need to expand how to file bugs against which packages

Warning /!\ W.I.P. Warning /!\

Touch Test SWAT

Welcome to the testing page of the Touch Testing SWAT. These pages provide up to date information of current testing activities for Touch until the release of Saucy, check back frequently.

Test coordination

  • Mailing List - The quality mailing list ubuntu-quality@lists.ubuntu.com is used to coordinate testing activities with all participants.

  • IRC - join the Ubuntu QA in #ubuntu-quality and and Ubuntu Touch #ubuntu-touch IRC channels on freenode.

Test process

This describes the daily process of testing. We will flash our devices, look for bugs, then look to write autopilot tests for any fixed bugs to prevent that bug from re-occurring.

Daily Image Testing

Install the current stable image

Flash your phone to the latest stable image (note: we want the "current" image of touch_ro). If you have not yet bootstrapped or unlocked your phone's bootloader, see the install page for more information.

  $ phablet-flash ubuntu-system

Alternatively, once you are running the ubuntu image you may perform a system update from the device itself.

Check to see what's changed

A list of changes between stable images can be found here. The list contains the list of changing packages.

Need more detailed list of what is actually changing

Test

* Run through the list of new features and changed areas of the phone, paying specific attention to spot regressions and verify the features are working.

* Perform smoke testing on a specific packageset of the phone.

* File bugs against any issues encountered

* Remember this is exploratory testing, so try and break things!

Report Bugs

See the Reporting a Bug section below for details.

Add a testcase for reported bugs

Once testing is completed, review the list of reported bugs with the needs-autopilot-test tag. Automate a testcase using autopilot that fixes the bug to ensure the same bug doesn't reappear.

Reporting a bug

All bugs filed should use the ???? (need to decide) tag. Omer has made available a tool to help with the process of filing a bug report

What package should I file against?

File against the ubuntu source package for the failed component.

Filing the Bug

Using Omer's tool

http://paste.ubuntu.com/6098563/

There is a crash file in /var/crash

If the bug is a crash and there is a crash file in /var/crash, you can report it with apport.

1. Connect to the device with adb

  $ adb shell

2. Check for any crash file in /var/crash

3. Report a bug with apport:

  $ apport-cli /var/crash/name_of_crash_file.crash

4. Follow the instructions

5. To the question "What would you like to do? Your options are:", answer "S: Send report"

6. Wait for the upload to proceed

7. Upon upload you'll be presented with a prompt like:

*** To continue, you must visit the following URL:

  https://bugs.launchpad.net/ubuntu/+source/upstart-app-launch/+filebug/<UUID>

You can launch a browser now, or copy this URL into a browser on another computer.


Choices:
  1: Launch a browser now
  C: Cancel

8. Copy the URL to a browser running on your desktop/laptop, it is more convenient than running a browser on the phone to report bug and proceed with the bug reporting process in launchpad

9. Then follow the usual tagging plan and triaging process

There is no crash file

1. Connect to the device with adb

  $ adb shell

2. Find the right package (TODO: Document difference between preinstalled dpkg based packages and click package)

  $ ubuntu-bug maliit-framework

3. Wait until data collection is done.

4. At the question:

What would you like to do? Your options are:
  S: Send report (4.3 KB)
  V: View report
[...]
https://bugs.launchpad.net/ubuntu/+source/maliit-framework/+filebug/<UUID>?

Press 'S' or 'K' if you want to copy the report to another machine (for example if networking is not enabled on the device).

3. If you're reporting from the device, on your local machine, click on the link or copy/paste it into your browser.

Test Activities

Describe here any specific test activities.

TODO

* Add link to reference documentation (how to flash, specific testing like MIR, how to troubleshoot, ...)

* Add link and information on omer's testing tool

* See if popey's scripts might be useful and add them

* Need better list of what's changing between images

* Add list of who's testing which feature sets / areas

* Need to expand how to file bugs against which packages

QATeam/TouchTesting (last edited 2015-01-22 17:18:23 by xdsl-83-150-81-40)