TouchTesting

Differences between revisions 14 and 65 (spanning 51 versions)
Revision 14 as of 2013-09-17 16:32:41
Size: 6575
Editor: nskaggs
Comment:
Revision 65 as of 2015-01-22 17:18:23
Size: 4135
Editor: knome
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
<<Include(QATeam/Header)>>
<<BR>>
Line 3: Line 5:
/!\ '''W.I.P.''' /!\ = Testing Ubuntu Touch =
Welcome! This page contains the information you need to know to test ubuntu touch. Follow the steps outlined to keep your device updated and your testing relevant.
Line 5: Line 8:
= Touch Test SWAT =
Welcome to the testing page of the Touch Testing SWAT. These pages provide up to date information of current testing activities for Touch until the release of Saucy, check back frequently.
== Setup ==
You should subscribe to the phone mailing list, and keep in mind the following IRC channels for help.
Line 8: Line 11:
== Test coordination ==
 * '''Mailing List''' - The quality mailing list [[https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality|ubuntu-quality@lists.ubuntu.com]] is used to coordinate testing activities with all participants.
 * '''IRC''' - join the Ubuntu QA in `#ubuntu-quality` and and Ubuntu Touch `#ubuntu-touch` IRC channels on freenode.
 * '''Mailing Lists''' - The [[https://launchpad.net/~ubuntu-phone|ubuntu phone mailing list]] provides constant updates about what is happening on the phone images.
 * '''IRC''' - join the Ubuntu QA in `#ubuntu-quality` and and Ubuntu Touch `#ubuntu-touch` IRC channels on freenode if you need realtime help.

You also need to flash your phablet device with the latest proposed image.

=== Install the current development image ===
Flash your phone to the latest proposed image. If you have not yet bootstrapped or unlocked your phone's bootloader, [[http://developer.ubuntu.com/start/ubuntu-for-devices/installing-ubuntu-for-devices/|see the install page for more information]].

To test with the proposed image:
{{{
  $ ubuntu-device-flash --channel=devel-proposed
}}}
Line 13: Line 25:
This describes the daily process of testing. We will flash our devices, look for bugs, then look to write autopilot tests for any fixed bugs to prevent that bug from re-occurring. This describes the daily process of testing. We will flash our devices, read what's changed, and look for bugs.
Line 15: Line 27:
== Daily Image Testing ==
=== Install the current stable image ===
Flash your phone to the latest stable image (note: we want the "current" image of touch_ro). If you have not yet bootstrapped or unlocked your phone's bootloader, [[https://wiki.ubuntu.com/Touch/Install|see the install page for more information]].
{{{
  $ phablet-flash ubuntu-system
}}}
== Step 1: Update ==
You need a phone or tablet with the latest development proposed image. If needed, install the current development image now by flashing or using the OTA update feature.
Line 22: Line 30:
==== Perform a system update ====
Alternatively, once you are running the ubuntu image you may perform a system update from the device itself. To do this, select 'system-settings' application, then click update. Press the download button, and once the download is completed, the install button.
=== Perform a system update ===
Once you are running the ubuntu image you may perform a system update from the device itself. The images now prompt and even autoupdate if allowed. To manually update, select the 'system-settings' application, then click update. Once the download completes, press the install and restart button.
Line 25: Line 33:
=== Check to see what's changed ===
A list of changes between stable images can be [[http://people.canonical.com/~ogra/touch-image-stats/current/|found here]]. The list contains the list of changing packages.
[[attachment:update.gif|Visual Walkthrough of updating]]
Line 28: Line 35:
 * Changes in current: http://people.canonical.com/~j-lallement/touch/changes/current/ [[attachment:noupdate.png|Example of no updates found]]
Line 30: Line 37:
 * Changes in pending: http://people.canonical.com/~j-lallement/touch/changes/ == Step 2: Check to see what's changed ==
'''This step is optional.''' A list of changes between stable images can be [[http://people.canonical.com/~lzemczak/landing-team/|found here]]. The list contains the list of changing packages.
Line 32: Line 40:
Use these changes to drive exploratory testing below. Use these changes to help you understand what is new so you can check it for bugs.
Line 34: Line 42:
=== Test === == Step 3: Test ==
Line 37: Line 45:
* Perform smoke testing on a specific packageset of the phone.

* File bugs against any issues encountered
* Use the device as you normally would as you go about daily activities.
Line 43: Line 49:
=== Report Bugs ===
See the [[https://wiki.ubuntu.com/QATeam/TouchTesting#Reporting_a_bug|Reporting a Bug section below]] for details.
== Step 4: Report Bugs ==
Report any bugs you found while testing. See the [[QATeam/TouchTesting/BugReporting|Reporting a Bug page]] for details.

== Step 5: Review Bugs ==
'''This step is optional.''' Review bugs found. If you did not find a new bug, seek to confirm existing bugs via triaging. You can also flag bugs as good candidates for tests to prevent future regressions.

=== Confirm an existing bug ===
Not finding any bugs yourself? Look at [[QATeam/TouchTesting/BugReporting#Triaging|Triaging]] and confirm bugs others have found.
Line 47: Line 59:
Once testing is completed, review the list of reported bugs with the [[https://bugs.launchpad.net/bugs/+bugs?field.tag=needs-autopilot-test| needs-autopilot-test tag]]. [[https://wiki.ubuntu.com/Touch/CoreApps/Testing/ContributeAutopilotTestcase|Automate a testcase using autopilot]] that fixes the bug to ensure the same bug doesn't reappear. Once testing is completed, review the list of reported bugs with the [[https://bugs.launchpad.net/bugs/+bugs?field.tag=touch-needs-autopilot| touch-needs-autopilot tag]]. Pick a testcase to [[https://wiki.ubuntu.com/Touch/CoreApps/Testing/ContributeAutopilotTestcase|automate using autopilot]] that tests the bug to ensure the same bug doesn't reappear. So we can track work happening on these bugs, follow this process while writing the test;
Line 49: Line 61:
= Reporting a bug =
All bugs filed should use the touch-swat tag. Omer has made available a tool to help with the process of filing a bug report.

== What package should I file against? ==
File against the ubuntu source package for the failed component. [[https://wiki.ubuntu.com/QATeam/TouchTesting/BugFiling|See this page]] for a full broken out list of components.

== List of Reported Bugs ==
[[https://bugs.launchpad.net/bugs/+bugs?field.tag=touch-swat|All touch-swat bug reports]]

== Filing the Bug ==

=== Using Omer's tool ===
''' This is the recommended way to file bugs '''

Omer has created a tool to allow you to report a bug from your desktop, provided the phone is connected via usb cable. [[http://paste.ubuntu.com/6098563/plain/|Download and save this file]]. To run, pass the package name as the only argument. The tool will connect and grab logs from your device, and open a new bug in your desktop browser to file the bug.

=== There is a crash file in /var/crash ===

If the bug is a crash and there is a crash file in /var/crash, you can report it with apport.

1. Connect to the device with adb

{{{
  $ adb shell
}}}

2. Check for any crash file in /var/crash

3. Report a bug with apport:

{{{
  $ apport-cli /var/crash/name_of_crash_file.crash
}}}

4. Follow the instructions

5. To the question "What would you like to do? Your options are:", answer "S: Send report"

6. Wait for the upload to proceed

7. Upon upload you'll be presented with a prompt like:

{{{
*** To continue, you must visit the following URL:

  https://bugs.launchpad.net/ubuntu/+source/upstart-app-launch/+filebug/<UUID>

You can launch a browser now, or copy this URL into a browser on another computer.


Choices:
  1: Launch a browser now
  C: Cancel
}}}

8. Copy the URL to a browser running on your desktop/laptop, it is more convenient than running a browser on the phone to report bug and proceed with the bug reporting process in launchpad

9. Then follow the usual tagging plan and triaging process

=== There is no crash file ===

1. Connect to the device with adb

{{{
  $ adb shell
}}}

2. Find the right package (TODO: Document difference between preinstalled dpkg based packages and click package)

{{{
  $ ubuntu-bug maliit-framework
}}}

3. Wait until data collection is done.

4. At the question:

{{{
What would you like to do? Your options are:
  S: Send report (4.3 KB)
  V: View report
[...]
https://bugs.launchpad.net/ubuntu/+source/maliit-framework/+filebug/<UUID>?
}}}

Press 'S' or 'K' if you want to copy the report to another machine (for example if networking is not enabled on the device).

3. If you're reporting from the device, on your local machine, click on the link or copy/paste it into your browser.
 1. Remove the '''touch-needs-autopilot''' tag from the original bug
 1. Open a new bug against the same project, tagged with '''qa-touch-automated'''
 1. Assign yourself to the bug and update progress as usual. When you release the test, close the bug with fix released.
Line 141: Line 67:
Here are some helpful hints that will come in handy while testing; feel free to add more!

 * Switch from RO to RW (to install additional packages for example):
    * Remount RW
{{{
    $ adb shell
    $ mount /dev/loop0 / -o remount,rw
}}}
    * or touch writable_image
{{{
    $ adb shell touch /userdata/.writable_image
    $ adb reboot
}}}

 * Take a screenshot (helpful for bug reporting)
{{{
   $ adb shell /system/bin/screencap /data/screenshot.png
   $ adb pull /data/screenshot.png ~/screenshot.png
}}}

 * Find out what build you are running
{{{
   $ adb shell cat /etc/media-info
}}}

= TODO =

* Add link to reference documentation (how to flash, specific testing like MIR, how to troubleshoot, ...)

* Add list of who's testing which feature sets / areas
Check out these [[QATeam/TouchTesting/Tips|handy tips]] to help you while testing.


Testing Ubuntu Touch

Welcome! This page contains the information you need to know to test ubuntu touch. Follow the steps outlined to keep your device updated and your testing relevant.

Setup

You should subscribe to the phone mailing list, and keep in mind the following IRC channels for help.

  • Mailing Lists - The ubuntu phone mailing list provides constant updates about what is happening on the phone images.

  • IRC - join the Ubuntu QA in #ubuntu-quality and and Ubuntu Touch #ubuntu-touch IRC channels on freenode if you need realtime help.

You also need to flash your phablet device with the latest proposed image.

Install the current development image

Flash your phone to the latest proposed image. If you have not yet bootstrapped or unlocked your phone's bootloader, see the install page for more information.

To test with the proposed image:

  $ ubuntu-device-flash  --channel=devel-proposed

Test process

This describes the daily process of testing. We will flash our devices, read what's changed, and look for bugs.

Step 1: Update

You need a phone or tablet with the latest development proposed image. If needed, install the current development image now by flashing or using the OTA update feature.

Perform a system update

Once you are running the ubuntu image you may perform a system update from the device itself. The images now prompt and even autoupdate if allowed. To manually update, select the 'system-settings' application, then click update. Once the download completes, press the install and restart button.

Visual Walkthrough of updating

Example of no updates found

Step 2: Check to see what's changed

This step is optional. A list of changes between stable images can be found here. The list contains the list of changing packages.

Use these changes to help you understand what is new so you can check it for bugs.

Step 3: Test

* Run through the list of new features and changed areas of the phone, paying specific attention to spot regressions and verify the features are working.

* Use the device as you normally would as you go about daily activities.

* Remember this is exploratory testing, so try and break things!

Step 4: Report Bugs

Report any bugs you found while testing. See the Reporting a Bug page for details.

Step 5: Review Bugs

This step is optional. Review bugs found. If you did not find a new bug, seek to confirm existing bugs via triaging. You can also flag bugs as good candidates for tests to prevent future regressions.

Confirm an existing bug

Not finding any bugs yourself? Look at Triaging and confirm bugs others have found.

Add a testcase for reported bugs

Once testing is completed, review the list of reported bugs with the touch-needs-autopilot tag. Pick a testcase to automate using autopilot that tests the bug to ensure the same bug doesn't reappear. So we can track work happening on these bugs, follow this process while writing the test;

  1. Remove the touch-needs-autopilot tag from the original bug

  2. Open a new bug against the same project, tagged with qa-touch-automated

  3. Assign yourself to the bug and update progress as usual. When you release the test, close the bug with fix released.

Tips and Tricks

Check out these handy tips to help you while testing.

QATeam/TouchTesting (last edited 2015-01-22 17:18:23 by knome)