TouchTesting

Differences between revisions 23 and 24
Revision 23 as of 2013-09-18 14:29:24
Size: 8275
Editor: adsl-98-70-43-80
Comment:
Revision 24 as of 2013-09-18 14:56:30
Size: 13319
Editor: adsl-98-70-43-80
Comment:
Deletions are marked like this. Additions are marked like this.
Line 50: Line 50:
Once testing is completed, review the list of reported bugs with the [[https://bugs.launchpad.net/bugs/+bugs?field.tag=needs-autopilot-test| needs-autopilot-test tag]]. Pick a testcase to [[https://wiki.ubuntu.com/Touch/CoreApps/Testing/ContributeAutopilotTestcase|automate using autopilot]] that tests the bug to ensure the same bug doesn't reappear. So we can track work happening on these bugs, follow this process while writing the test;

 1. Remove the '''needs-autopilot-test''' tag from the original bug
Once testing is completed, review the list of reported bugs with the [[https://bugs.launchpad.net/bugs/+bugs?field.tag=touch-needs-autopilot| touch-needs-autopilot tag]]. Pick a testcase to [[https://wiki.ubuntu.com/Touch/CoreApps/Testing/ContributeAutopilotTestcase|automate using autopilot]] that tests the bug to ensure the same bug doesn't reappear. So we can track work happening on these bugs, follow this process while writing the test;

 1. Remove the '''touch-needs-autopilot''' tag from the original bug
Line 56: Line 56:
=== Confirm an existing bug ===
Not finding any bugs yourself? Look at the [[https://wiki.ubuntu.com/QATeam/TouchTesting#Triaging|Triaging link below]] and confirm bugs others have found.
Line 57: Line 60:
All bugs filed should use the '''avengers''' tag. Omer has made available a tool to help with the process of filing a bug report. '''Please ensure you add the 'avengers' tag when filing'''. If you think the bug you are reporting is a good candidate for an automated test to prevent future regressions, you may also add '''needs-autopilot-test''' to the bug. All bugs filed should use the '''avengers''' tag. Omer has made available a tool to help with the process of filing a bug report. '''Please ensure you add the 'avengers' tag when filing'''. If you think the bug you are reporting is a good candidate for an automated test to prevent future regressions, you may also add '''touch-needs-autopilot''' to the bug.
Line 61: Line 64:

== List of Reported Bugs ==
[[https://bugs.launchpad.net/bugs/+bugs?field.tag=avengers|All avengers bug reports]]

=== What does each tag mean? ===

|| Tag || Description ||
|| avengers || Used for filing bugs found during exploratory testing of ubuntu touch images ||
|| needs-autopilot-test || Used to tag bugs that are candidates for automation to prevent recurrence once fixed ||
|| qa-touch-automated || Used to tag bugs that are being automated with autopilot ||
Line 152: Line 145:
= Triaging =
Once bugs have been filed they need to be confirmed, and ideally, have a defined scenario for recreating in order for developers to fix them. To help with triaging, simply read a bug report from the list below and try it on the latest build for your device. If you can recreate it, mark the bug status as ''confirmed''.

== Quick links to buglists ==
=== Needs work ===
 * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=avengers&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search|Bugs needing triaged (tagged with 'avengers' and a new or incomplete with response status)]]
 * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&field.status%3Alist=NEW&field.status%3Alist=OPINION&field.status%3Alist=INVALID&field.status%3Alist=WONTFIX&field.status%3Alist=EXPIRED&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=touch-needs-autopilot&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search&orderby=-datecreated&start=0|Bugs that could be automated (tagged with 'touch-needs-autopilot']]
 * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=OPINION&field.status%3Alist=INVALID&field.status%3Alist=WONTFIX&field.status%3Alist=EXPIRED&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=qa-touch-automated&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on|Bugs in progress of being automated (tagged with 'qa-touch-automated' and a non-fix released status)]]

=== Completed work ===
 * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=FIXRELEASED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=avengers&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on|Bugs that have been fixed (tagged with 'avengers' and a fix released status)]]
 * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=FIXRELEASED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=qa-touch-automated&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on|Bugs that have been automated (tagged with 'qa-touch-automated' and a fix released status)]]

=== List of all reported Bugs ===
 * [[https://bugs.launchpad.net/bugs/+bugs?field.tag=avengers|All 'avengers' bug reports]]

== Tags ==
Tags are used to keep handy lists of the work being done. It lets us see the status of bugs and testcase automation.

=== What does each tag mean? ===

|| Tag || Description ||
|| avengers || Used for filing bugs found during exploratory testing of ubuntu touch images ||
|| touch-needs-autopilot || Used to tag bugs that are candidates for automation to prevent recurrence once fixed ||
|| qa-touch-automated || Used to tag bugs that are being automated with autopilot ||

Warning /!\ W.I.P. Warning /!\

Touch Test SWAT

Welcome to the testing page of the Touch Testing SWAT. These pages provide up to date information of current testing activities for Touch until the release of Saucy, check back frequently.

Test coordination

  • Mailing List - The quality mailing list ubuntu-quality@lists.ubuntu.com is used to coordinate testing activities with all participants.

  • IRC - join the Ubuntu QA in #ubuntu-quality and and Ubuntu Touch #ubuntu-touch IRC channels on freenode.

Test process

This describes the daily process of testing. We will flash our devices, look for bugs, then look to write autopilot tests for any fixed bugs to prevent that bug from re-occurring.

Daily Image Testing

Install the current stable image

Flash your phone to the latest stable image (note: we want the "current" image of touch_ro). If you have not yet bootstrapped or unlocked your phone's bootloader, see the install page for more information.

  $ phablet-flash ubuntu-system

In order to wipe clean and start fresh with the latest current image:

  $ phablet-flash ubuntu-system --no-backup

Perform a system update

Alternatively, once you are running the ubuntu image you may perform a system update from the device itself. To do this, select 'system-settings' application, then click update. Press the download button, and once the download is completed, the install button.

Check to see what's changed

A list of changes between stable images can be found here. The list contains the list of changing packages.

Use these changes to drive exploratory testing below.

Test

* Run through the list of new features and changed areas of the phone, paying specific attention to spot regressions and verify the features are working.

* Perform smoke testing on a specific packageset of the phone.

* File bugs against any issues encountered

* Remember this is exploratory testing, so try and break things!

Report Bugs

See the Reporting a Bug section below for details.

Add a testcase for reported bugs

Once testing is completed, review the list of reported bugs with the touch-needs-autopilot tag. Pick a testcase to automate using autopilot that tests the bug to ensure the same bug doesn't reappear. So we can track work happening on these bugs, follow this process while writing the test;

  1. Remove the touch-needs-autopilot tag from the original bug

  2. Open a new bug against the same project, tagged with qa-touch-automated

  3. Assign yourself to the bug and update progress as usual. When you release the test, close the bug with fix released.

Confirm an existing bug

Not finding any bugs yourself? Look at the Triaging link below and confirm bugs others have found.

Reporting a bug

All bugs filed should use the avengers tag. Omer has made available a tool to help with the process of filing a bug report. Please ensure you add the 'avengers' tag when filing. If you think the bug you are reporting is a good candidate for an automated test to prevent future regressions, you may also add touch-needs-autopilot to the bug.

What package should I file against?

File against the ubuntu source package for the failed component. See this page for a full broken out list of components.

Filing the Bug

Using Omer's tool

This is the recommended way to file bugs

Omer has created a tool to allow you to report a bug from your desktop, provided the phone is connected via usb cable. Download and save this file. To run, pass the package name as the only argument. The tool will connect and grab logs from your device, and open a new bug in your desktop browser to file the bug.

There is a crash file in /var/crash

If the bug is a crash and there is a crash file in /var/crash, you can report it with apport.

1. Connect to the device with adb

  $ adb shell

2. Check for any crash file in /var/crash

3. Report a bug with apport:

  $ apport-cli /var/crash/name_of_crash_file.crash

4. Follow the instructions

5. To the question "What would you like to do? Your options are:", answer "S: Send report"

6. Wait for the upload to proceed

7. Upon upload you'll be presented with a prompt like:

*** To continue, you must visit the following URL:

  https://bugs.launchpad.net/ubuntu/+source/upstart-app-launch/+filebug/<UUID>

You can launch a browser now, or copy this URL into a browser on another computer.


Choices:
  1: Launch a browser now
  C: Cancel

8. Copy the URL to a browser running on your desktop/laptop, it is more convenient than running a browser on the phone to report bug and proceed with the bug reporting process in launchpad

9. Then follow the usual tagging plan and triaging process

There is no crash file

1. Connect to the device with adb

  $ adb shell

2. Find the right package (TODO: Document difference between preinstalled dpkg based packages and click package)

  $ ubuntu-bug maliit-framework

3. Wait until data collection is done.

4. At the question:

What would you like to do? Your options are:
  S: Send report (4.3 KB)
  V: View report
[...]
https://bugs.launchpad.net/ubuntu/+source/maliit-framework/+filebug/<UUID>?

Press 'S' or 'K' if you want to copy the report to another machine (for example if networking is not enabled on the device).

3. If you're reporting from the device, on your local machine, click on the link or copy/paste it into your browser.

Triaging

Once bugs have been filed they need to be confirmed, and ideally, have a defined scenario for recreating in order for developers to fix them. To help with triaging, simply read a bug report from the list below and try it on the latest build for your device. If you can recreate it, mark the bug status as confirmed.

Needs work

Completed work

List of all reported Bugs

Tags

Tags are used to keep handy lists of the work being done. It lets us see the status of bugs and testcase automation.

What does each tag mean?

Tag

Description

avengers

Used for filing bugs found during exploratory testing of ubuntu touch images

touch-needs-autopilot

Used to tag bugs that are candidates for automation to prevent recurrence once fixed

qa-touch-automated

Used to tag bugs that are being automated with autopilot

Tips and Tricks

Here are some helpful hints that will come in handy while testing; feel free to add more!

Switch from RO to RW

  • Switch from RO to RW (to install additional packages for example):
    • Remount RW

    $ adb shell
    $ mount /dev/loop0 / -o remount,rw
  • or touch writable_image

    $ adb shell touch /userdata/.writable_image
    $ adb reboot

Take a screenshot

This is helpful for bug reporting. Note this WILL NOT work if you are running Mir.

   $ adb shell /system/bin/screencap /data/screenshot.png
   $ adb pull /data/screenshot.png ~/screenshot.png

You can resize with convert from package imagemagick:

   $ convert screenshot.png -resize 50% screenshot.png

Which build am I running?

  • Find out what build you are running

   $ adb shell cat /etc/media-info

Enable MIR by default

   $ adb shell touch /home/phablet/.display-mir
   $ adb reboot

   to switch back:

   $ adb shell rm /home/phablet/.display-mir
   $ adb reboot
  • MIR is running if no surfaceflinger process is running and the socket /tmp/mir_socket is present

TODO

* Add link to reference documentation (how to flash, specific testing like MIR, how to troubleshoot, ...)

* Add list of who's testing which feature sets / areas

QATeam/TouchTesting (last edited 2015-01-22 17:18:23 by xdsl-83-150-81-40)