TouchTesting
2578
Comment:
|
28008
|
Deletions are marked like this. | Additions are marked like this. |
Line 9: | Line 9: |
* '''Mailing List''' - The quality mailing list [https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality|ubuntu-quality@lists.ubuntu.com] is used to coordinate testing activities with all participants. | * '''Mailing List''' - The quality mailing list [[https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality|ubuntu-quality@lists.ubuntu.com]] is used to coordinate testing activities with all participants. |
Line 13: | Line 13: |
TODO |
This describes the daily process of testing. We will flash our devices, look for bugs, then look to write autopilot tests for any fixed bugs to prevent that bug from re-occurring. == Daily Image Testing == === Install the current stable image === Flash your phone to the latest stable image (note: we want the "current" image of touch_ro). If you have not yet bootstrapped or unlocked your phone's bootloader, [[https://wiki.ubuntu.com/Touch/Install|see the install page for more information]]. {{{ $ phablet-flash ubuntu-system --channel trusty }}} In order to wipe clean and start fresh with the latest current image: {{{ $ phablet-flash ubuntu-system --channel trusty --no-backup }}} To test with the proprosed image: {{{ $ phablet-flash ubuntu-system --no-backup --channel=devel-proposed }}} ==== Perform a system update ==== Alternatively, once you are running the ubuntu image you may perform a system update from the device itself. To do this, select 'system-settings' application, then click update. Press the download button, and once the download is completed, the install button. [[attachment:update.gif|Visual Walkthrough of updating]] [[attachment:noupdate.png|Example of no updates found]] === Check to see what's changed === A list of changes between stable images can be [[http://people.canonical.com/~ogra/touch-image-stats/current/|found here]]. The list contains the list of changing packages. * Changes in current: http://people.canonical.com/~j-lallement/touch/changes/current/ * Changes in pending: http://people.canonical.com/~j-lallement/touch/changes/ Use these changes to drive exploratory testing below. === Test === * Run through the list of new features and changed areas of the phone, paying specific attention to spot regressions and verify the features are working. * Perform smoke testing on a specific packageset of the phone. * File bugs against any issues encountered * Remember this is exploratory testing, so try and break things! === Report Bugs === See the [[https://wiki.ubuntu.com/QATeam/TouchTesting#Reporting_a_bug|Reporting a Bug section below]] for details. === Add a testcase for reported bugs === Once testing is completed, review the list of reported bugs with the [[https://bugs.launchpad.net/bugs/+bugs?field.tag=touch-needs-autopilot| touch-needs-autopilot tag]]. Pick a testcase to [[https://wiki.ubuntu.com/Touch/CoreApps/Testing/ContributeAutopilotTestcase|automate using autopilot]] that tests the bug to ensure the same bug doesn't reappear. So we can track work happening on these bugs, follow this process while writing the test; 1. Remove the '''touch-needs-autopilot''' tag from the original bug 1. Open a new bug against the same project, tagged with '''qa-touch-automated''' 1. Assign yourself to the bug and update progress as usual. When you release the test, close the bug with fix released. === Confirm an existing bug === Not finding any bugs yourself? Look at the [[https://wiki.ubuntu.com/QATeam/TouchTesting#Triaging|Triaging link below]] and confirm bugs others have found. |
Line 17: | Line 71: |
== There is a crash file in /var/crash == If the bug is a crash and there is a crash file in /var/crash, you can report it with apport. 1. Connect to the device with adb {{{ $ adb shell }}} 2. Check for any crash file in /var/crash 3. Report a bug with apport: {{{ $ apport-cli /var/crash/name_of_crash_file.crash }}} 4. Follow the instructions 5. To the question "What would you like to do? Your options are:", answer "S: Send report" 6. Wait for the upload to proceed 7. Upon upload you'll be presented with a prompt like: {{{ |
All bugs filed should use the '''avengers''' tag. Omer has made available a tool to help with the process of filing a bug report. '''Please ensure you add the 'avengers' tag when filing'''. If you think the bug you are reporting is a good candidate for an automated test to prevent future regressions, you may also add '''touch-needs-autopilot''' to the bug. == What package should I file against? == File against the ubuntu source package for the failed component. [[https://wiki.ubuntu.com/QATeam/TouchTesting/BugFiling|See this page]] for a full broken out list of components. == Filing the Bug == === Using Omer's tool === ''' This is the recommended way to file bugs ''' Omer has created a tool to allow you to report a bug from your desktop, provided the phone is connected via usb cable. [[http://goo.gl/Irl3aY|Download and save this file]]. Via the command line, {{{ wget http://goo.gl/Irl3aY -O filephonebug chmod +x filephonebug }}} To run, pass the package name as the only argument. The tool will connect and grab logs from your device, and open a new bug in your desktop browser to file the bug. For example if the phone dialer is causing issues you would do the following: * Connect the phone via usb * Run the tool against the dialer-app package. ''Unsure of which package? [[https://wiki.ubuntu.com/QATeam/TouchTesting/BugFiling|Check this list]]'' {{{ ./filephonebug dialer-app }}} * Apport will run and collect logs and ask you 2 questions. Answer 'S' to send the report, and then '1' to launch a browser to file the report. The output will look like this: {{{ $ ./filephonebug dialer-app *** Collecting problem information The collected information can be sent to the developers to improve the application. This might take a few minutes. ........................................................................................ 182 KB/s (14785 bytes in 0.078s) *** Send problem report to the developers? After the problem report has been sent, please fill out the form in the automatically opened web browser. What would you like to do? Your options are: S: Send report (14.1 KB) V: View report K: Keep report file for sending later or copying to somewhere else I: Cancel and ignore future crashes of this program version C: Cancel Please choose (S/V/K/I/C): s *** Uploading problem information The collected information is being sent to the bug tracking system. This might take a few minutes. 98% |
Line 47: | Line 129: |
https://bugs.launchpad.net/ubuntu/+source/upstart-app-launch/+filebug/<UUID> | https://bugs.launchpad.net/ubuntu/+source/dialer-app/+filebug/59dbaa1c-2162-11e3-a3d4-d485646cd9a4? |
Line 55: | Line 137: |
Please choose (1/C): Choices: 1: Launch a browser now C: Cancel Please choose (1/C): 1 }}} * The browser will launch and display a ready to file bug report. Finish filling in the description of the issue. When filing, make sure you include the steps needed for others to reproduce the issue, and what the result was versus what you expected the result to be. === There is a crash file in /var/crash === If the bug is a crash and there is a crash file in /var/crash, you can report it with apport. 1. Connect to the device with adb {{{ $ adb shell }}} 2. Check for any crash file in /var/crash 3. Report a bug with apport: {{{ $ apport-cli /var/crash/name_of_crash_file.crash }}} 4. Follow the instructions 5. To the question "What would you like to do? Your options are:", answer "S: Send report" 6. Wait for the upload to proceed 7. Upon upload you'll be presented with a prompt like: {{{ *** To continue, you must visit the following URL: https://bugs.launchpad.net/ubuntu/+source/upstart-app-launch/+filebug/<UUID> You can launch a browser now, or copy this URL into a browser on another computer. Choices: 1: Launch a browser now C: Cancel |
|
Line 61: | Line 189: |
== There is no crash file == | === There is no crash file === |
Line 73: | Line 201: |
}}} 3. Wait until data collection is done. 4. At the question: {{{ |
|
Line 76: | Line 210: |
V: View report [...] |
|
Line 80: | Line 215: |
3. On your local machine, click on the link or copy/paste it into your browser. = Test Activities = Describe here any specific test activities. = TODO = * Add link to reference documentation (how to flash, specific testing like MIR, how to troubleshoot, ...) |
Press 'S' or 'K' if you want to copy the report to another machine (for example if networking is not enabled on the device). 3. If you're reporting from the device, on your local machine, click on the link or copy/paste it into your browser. = Triaging = Once bugs have been filed they need to be confirmed, and ideally, have a defined scenario for recreating in order for developers to fix them. To help with triaging, simply read a bug report from the list below and try it on the latest build for your device. If you can recreate it, mark the bug status as ''confirmed''. == Quick links to buglists == === Needs work === * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=avengers&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search|Bugs needing triaged (tagged with 'avengers' and a new or incomplete with response status)]] * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&field.status%3Alist=NEW&field.status%3Alist=OPINION&field.status%3Alist=INVALID&field.status%3Alist=WONTFIX&field.status%3Alist=EXPIRED&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=touch-needs-autopilot&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search&orderby=-datecreated&start=0|Bugs that could be automated (tagged with 'touch-needs-autopilot']] * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=OPINION&field.status%3Alist=INVALID&field.status%3Alist=WONTFIX&field.status%3Alist=EXPIRED&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=qa-touch-automated&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on|Bugs in progress of being automated (tagged with 'qa-touch-automated' and a non-fix released status)]] === Completed work === * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=FIXRELEASED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=avengers&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on|Bugs that have been fixed (tagged with 'avengers' and a fix released status)]] * [[https://bugs.launchpad.net/bugs/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=FIXRELEASED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.tag=qa-touch-automated&field.tags_combinator=ANY&field.status_upstream-empty-marker=1&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on|Bugs that have been automated (tagged with 'qa-touch-automated' and a fix released status)]] === List of all reported Bugs === * [[https://bugs.launchpad.net/bugs/+bugs?field.tag=avengers|All 'avengers' bug reports]] == Tags == Tags are used to keep handy lists of the work being done. It lets us see the status of bugs and testcase automation. === What does each tag mean? === || Tag || Description || || avengers || Used for filing bugs found during exploratory testing of ubuntu touch images || || touch-needs-autopilot || Used to tag bugs that are candidates for automation to prevent recurrence once fixed || || qa-touch-automated || Used to tag bugs that are being automated with autopilot || = Tips and Tricks = Here are some helpful hints that will come in handy while testing; feel free to add more! == Switch from RO to RW == * Switch from RO to RW (to install additional packages for example): * Remount RW {{{ $ adb shell $ mount -o remount,rw / }}} * or touch writable_image {{{ $ adb shell touch /userdata/.writable_image $ adb reboot }}} == Take a screenshot == This is helpful for bug reporting. If you are using a build post 90 you are running with MIR by default. === MIR === [[http://people.canonical.com/~j-lallement/touch/mirfbdump|Use this script to take a screenshot]]. Save the file and run passing a filename (ie screendump.png) as the sole argument. Ensure your device is connected to your pc before running, with the screen on and displaying the screenshot you wish to take. /!\ This does not work with maguro due to a bug in the gpu, capturing the frame buffer kills the device. /!\ Because of how the script works (by dumping the framebuffer directly), you may potentially experience invalid images or otherwise fail to get a screenshot. === SurfaceFlinger === {{{ $ adb shell /system/bin/screencap /data/screenshot.png $ adb pull /data/screenshot.png ~/screenshot.png }}} You can resize with `convert` from package imagemagick: {{{ $ convert screenshot.png -resize 50% screenshot.png }}} == Which build am I running? == * Find out what build you are running {{{ $ adb shell cat /etc/media-info }}} Or for the details of the build (version of android part/version of ubuntu part) {{{ root@ubuntu-phablet:/# system-image-cli -i current build number: 80 device name: mako channel: devel-proposed last update: 2013-10-04 09:59:14 version version: 80 version ubuntu: 20131003.2 version device: 20131003.2 }}} === Enable MIR by default === * Switch to Mir enabled, SurfaceFlinger disabled {{{ $ adb shell touch /home/phablet/.display-mir $ adb reboot to switch back: $ adb shell rm /home/phablet/.display-mir $ adb reboot }}} * MIR is running if no surfaceflinger process is running and the socket /tmp/mir_socket is present * Tag bugs that exist only in MIR with mir-only * MIR is running as default as of build 90. All versions released after 1.0 including 1.0 use MIR by default == Enable gprs if it doesn't come up automatically == * GPRS A simple work around while this is not working is to do: {{{ $ adb shell restart network-manager }}} == Turn back on intro == * Turn back on the initial demo that instructs how to use the phone via swiping {{{ dbus-send --system --print-reply --dest=org.freedesktop.Accounts /org/freedesktop/Accounts/User32011 org.freedesktop.DBus.Properties.Set string:com.canonical.unity.AccountsService string:demo-edges variant:boolean:true }}} == Retracing a crash on phone == In case you don't want to submit a crash to Launchpad to let them be handled by Ubuntu retracers, a way to generate nice stack trace from crash files in /var/crash/ and usable by developers is to use apport retrace directly on the phone. 1. Switch the device to RW mode 2. Install apport-retrace: {{{ apt-get install apport-retrace }}} 3. Enable ddebs repository [[https://wiki.ubuntu.com/DebuggingProgramCrash#Debug_Symbol_Packages|to install debug symbol packages]] 4. Run apport-cli on a crash files in /var/crash, for example: {{{ $ apport-cli /var/crash/_usr_lib_arm-linux-gnueabihf_upstart-app-launch_zg-report-app.32011.crash }}} 4.1. To the question '''Send problem report to the developers?''' answer '''E: Examine locally''' 4.2. To the question '''This will launch apport-retrace in a terminal window to examine the crash.''' answer '''3: Update /var/crash/_usr_lib_arm-linux-gnueabihf_upstart-app-launch_zg-report-app.32011.crash with fully symbolic stack trace'''. Alternatively you can also get into an interactive gdb shell with all symbols loaded for in-depth debugging. 5. Let apport proceed ... Note: Make sure the phone is not blacked because in suspend mode, it will takes an infinite amount of time to do anything. == Retracing a phone crash on desktop == You'll first need some apt sources for apport to download -dbg and -dbgsym packages from. The configurations we use for the production retracers can be found in the Daisy project, so grab those: {{{ bzr branch lp:daisy }}} You'll need the following packages on your desktop: * apport-retrace * gdb-multiarch Next, on the phone, tell apport to collect some additional information for the crash. This adds the packaging information and runs any hooks: {{{ $ apport-cli /var/crash/your_crash_file.crash }}} Hit view '''(V)''', then hit keep '''(K)'''. Finally, on your desktop, pull the crash from your phone and retrace your crash: {{{ $ apport-retrace -S ~/bzr/daisy/retracer/config -o updated_crash_file.crash /var/crash/your_crash_file.crash }}} updated_crash_file.crash should now have full Stacktrace and ThreadStacktrace fields. If you do this more than once it's helpful to use a permanent cache directory with e. g. -C ~/.cache/apport/ or something similar. The next time you retrace a crash it can then re-use the already downloaded debs, ddebs, and indexes. You can throw that at Launchpad by running '''apport-cli updated_crash_file.crash''', then hitting '''S''' to send the report. == Disable Screen Timeout == Run: {{{ $ powerd-cli display on bright }}} == Add APN Settings == ''''Note you'll need to be [[https://wiki.ubuntu.com/QATeam/TouchTesting#Switch_from_RO_to_RW|in r/w mode]] for this to work'''' If you need to edit your APN settings for some reason, currently you have to manually edit the ofono config files. Perform the following: * Shell into your phone or open a terminal * Stop network manager and ofono {{{ stop network-manager stop ofono }}} * Find your proper APN settings from your provider * Use the APN name as the AccessPointName in your gprs config file. The file is found at /var/lib/ofono/''<Identity>''/gprs * ''For example, for an ATT straight talk sim, this is a copy of the file that will work. Notice AccessPointName=tfdata matches the APN value given by straighttalk (att) below.'' {{{ [Settings] Powered=1 RoamingAllowed=1 [context1] Name=STRAIGHT_TALK AccessPointName=tfdata Username= Password= Type=internet Protocol=ip }}} * Reboot your phone * Enjoy working mobile data === MVNO === In general, check your MVNO's website for the settings information. You can use the required settings for any device, such as an android. Ignore the instructions and copy the information. It will look similar to the following: Net10 {{{ NAME net10 APN tfdata MMSC http://mms-tf.net MMS PROXY mms3.tracfone.com MMS PORT 80 }}} You only need the 'APN' field data to replace the AccessPointName= value in the gprs file. === Straight Talk Sim (Walmart) === The straight talk card might be using tmobile or att network. Here's the full APN details from for each. Straight Talk (ATT) {{{ NAME STRAIGHT_TALK APN tfdata MMSC http://mms-tf.net MMS PROXY mms3.tracfone.com MMS PORT 80 MMS_APN tfdata }}} Straight Talk (Tmobile) {{{ NAME STRAIGHT_TALK APN wap.tracfone MMSC http://mms.tracfone.com MMS PROXY MMS PORT MMS_APN wap.tracfone }}} == Debugging == === Running autopilot tests === ==== From a source branch ==== Save this bash script. [[attachment:testclick.sh]] Execute it passing the source branch as the only argument. Ensure you have bzr and phablet-tools installed. {{{ bash testclick.sh lp:calendar-app }}} ==== Via installed click package ==== If the autopilot test suite is already on the device, you can run it like so. Connect the phone to your PC and run: {{{ phablet-test-run -v TEST_NAME }}} The -v option gives more verbose output about failures and what is going on while testing, but isn't required for running. Replace TEST_NAME with the application test suite name you want to test. For example, {{{ phablet-test-run -v calendar_app }}} or even something specific like {{{ phablet-test-run -v calendar_app.tests.test_weekview.TestWeekView }}} === Install a Package from Source === Sometimes we need to test the behavior of the bleeding-edge code that the developers are actively working on. To do this we take advantage of the Debian packaging system by building a package from source on the device and installing it there. Here we'll run through an example of installing {{{lp:ubuntu-system-settings}}} from source on a device. You'll need to [[https://wiki.ubuntu.com/QATeam/TouchTesting#Switch_from_RO_to_RW|set up for a writeable image]]. Do this as the phablet user and make a workspace for your codez: {{{ $ adb shell $ su phablet $ mkdir ~/workspace }}} We'll need a few packages to begin: {{{ $ sudo apt-get install -y bzr devscripts equivs }}} Now get trunk as if you were on your local machine. {{{ $ cd ~/workspace $ bzr branch lp:ubuntu-system-settings }}} This installs all the packages necessary to build/run, should take a while for packages with lots of dependencies. {{{ $ cd ~/workspace/ubuntu-system-settings $ sudo mk-build-deps -i debian/control }}} Modify the changelog to ensure that our built package will take priority over any installed version of the package. {{{ $ dch -R }}} This opens an editor on the debian/changelog in the project directory (note that I had some trouble saving this file under nano, so vi is recommended): {{{ ubuntu-system-settings (0.1+13.10.20131002-0ubuntu2) UNRELEASED; urgency=low * FILL THIS IN -- phablet <phablet@ubuntu-phablet> Wed, 02 Oct 2013 12:52:50 +0000 ubuntu-system-settings (0.1+13.10.20131002-0ubuntu1) saucy; urgency=low [ Sebastien Bacher ] * click: the installed-size info is in kib, not in bytes. }}} Note that our new changelog entry bears an incremented version number. Fill in something like "Test build." for a changelog entry (behind the asterisk), then exit and save the changelog. Now build the package (live the life of a dev for a few mins ;) ). {{{ $ debuild -us -uc }}} The .deb has been deposited in the parent directory. Install it like you'd install any deb :) . Note that we may have built multiple deb files--if you find that you're not able to install a .deb due to a dependency issue, look for that dependency here first. In this case we'll discover that we need to install the 'libsystemsettings1' .deb before installing 'ubuntu-system-settings'. {{{ phablet@ubuntu-phablet:~/workspace/ubuntu-system-settings$ ls .. libsystemsettings1_0.1+13.10.20131002-0ubuntu2_armhf.deb libsystemsettings-dev_0.1+13.10.20131002-0ubuntu2_armhf.deb ubuntu-system-settings ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2_armhf.build ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2_armhf.changes ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2_armhf.deb ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2.dsc ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2.tar.gz phablet:~/workspace/ubuntu-system-settings$ sudo dpkg -i ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2_armhf.deb }}} Lastly verify that we're using the right package--the highlighted version should match the version of the new changelog entry we noted above. {{{ phablet@ubuntu-phablet:~/workspace/ubuntu-system-settings$ apt-cache policy ubuntu-system-settings ubuntu-system-settings: Installed: 0.1+13.10.20131002-0ubuntu2 Candidate: 0.1+13.10.20131002-0ubuntu2 Version table: *** 0.1+13.10.20131002-0ubuntu2 0 100 /var/lib/dpkg/status 0.1+13.10.20131002-0ubuntu1 0 500 http://ports.ubuntu.com/ubuntu-ports/ saucy/universe armhf Packages }}} NOTE: do consult with your developers about additional steps necessary to see the behavior of your built package on a device--if it's a system service, e.g., it may need to be killed and restarted, or you may need a new session to see changes in unity itself. Every package is different! === Get a list of installed click packages === {{{ click list }}} Typing in just click will show you other commands you might be interested in for click packages === Debug an application via command line === To debug an application on the device, start it using the start application command. Application is an upstart user session and you need to be phablet user in order for this to work properly. From your pc, shell in, become phablet user, and run the start application command. Shown below is an example running the filemanager. {{{ adb shell sudo -i -u phablet start application APP_ID=com.ubuntu.filemanager_filemanager_${version} }}} To run other applications, change the APP_ID to the application name. [[https://wiki.ubuntu.com/QATeam/TouchTesting#Get_a_list_of_installed_click_packages|Using click list]] is helpful for finding the proper name. The console output will be logged in ''/home/phablet/.cache/upstart/''. For example, for filemanager as shown above the log exists in ''/home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.1.1.80.log''. ==== Play a video from the command line ==== The general scheme for starting the media player from the command line via upstart is: {{{ start application APP_ID=mediaplayer-app APP_URIS="file_name" }}} This can be used for other apps by changing the APP_ID. |
Contents
|
W.I.P.
Touch Test SWAT
Welcome to the testing page of the Touch Testing SWAT. These pages provide up to date information of current testing activities for Touch until the release of Saucy, check back frequently.
Test coordination
Mailing List - The quality mailing list ubuntu-quality@lists.ubuntu.com is used to coordinate testing activities with all participants.
IRC - join the Ubuntu QA in #ubuntu-quality and and Ubuntu Touch #ubuntu-touch IRC channels on freenode.
Test process
This describes the daily process of testing. We will flash our devices, look for bugs, then look to write autopilot tests for any fixed bugs to prevent that bug from re-occurring.
Daily Image Testing
Install the current stable image
Flash your phone to the latest stable image (note: we want the "current" image of touch_ro). If you have not yet bootstrapped or unlocked your phone's bootloader, see the install page for more information.
$ phablet-flash ubuntu-system --channel trusty
In order to wipe clean and start fresh with the latest current image:
$ phablet-flash ubuntu-system --channel trusty --no-backup
To test with the proprosed image:
$ phablet-flash ubuntu-system --no-backup --channel=devel-proposed
Perform a system update
Alternatively, once you are running the ubuntu image you may perform a system update from the device itself. To do this, select 'system-settings' application, then click update. Press the download button, and once the download is completed, the install button.
Visual Walkthrough of updating
Check to see what's changed
A list of changes between stable images can be found here. The list contains the list of changing packages.
Changes in current: http://people.canonical.com/~j-lallement/touch/changes/current/
Changes in pending: http://people.canonical.com/~j-lallement/touch/changes/
Use these changes to drive exploratory testing below.
Test
* Run through the list of new features and changed areas of the phone, paying specific attention to spot regressions and verify the features are working.
* Perform smoke testing on a specific packageset of the phone.
* File bugs against any issues encountered
* Remember this is exploratory testing, so try and break things!
Report Bugs
See the Reporting a Bug section below for details.
Add a testcase for reported bugs
Once testing is completed, review the list of reported bugs with the touch-needs-autopilot tag. Pick a testcase to automate using autopilot that tests the bug to ensure the same bug doesn't reappear. So we can track work happening on these bugs, follow this process while writing the test;
Remove the touch-needs-autopilot tag from the original bug
Open a new bug against the same project, tagged with qa-touch-automated
- Assign yourself to the bug and update progress as usual. When you release the test, close the bug with fix released.
Confirm an existing bug
Not finding any bugs yourself? Look at the Triaging link below and confirm bugs others have found.
Reporting a bug
All bugs filed should use the avengers tag. Omer has made available a tool to help with the process of filing a bug report. Please ensure you add the 'avengers' tag when filing. If you think the bug you are reporting is a good candidate for an automated test to prevent future regressions, you may also add touch-needs-autopilot to the bug.
What package should I file against?
File against the ubuntu source package for the failed component. See this page for a full broken out list of components.
Filing the Bug
Using Omer's tool
This is the recommended way to file bugs
Omer has created a tool to allow you to report a bug from your desktop, provided the phone is connected via usb cable. Download and save this file. Via the command line,
wget http://goo.gl/Irl3aY -O filephonebug chmod +x filephonebug
To run, pass the package name as the only argument. The tool will connect and grab logs from your device, and open a new bug in your desktop browser to file the bug.
For example if the phone dialer is causing issues you would do the following:
- Connect the phone via usb
Run the tool against the dialer-app package. Unsure of which package? Check this list
./filephonebug dialer-app
- Apport will run and collect logs and ask you 2 questions. Answer 'S' to send the report, and then '1' to launch a browser to file the report. The output will look like this:
$ ./filephonebug dialer-app *** Collecting problem information The collected information can be sent to the developers to improve the application. This might take a few minutes. ........................................................................................ 182 KB/s (14785 bytes in 0.078s) *** Send problem report to the developers? After the problem report has been sent, please fill out the form in the automatically opened web browser. What would you like to do? Your options are: S: Send report (14.1 KB) V: View report K: Keep report file for sending later or copying to somewhere else I: Cancel and ignore future crashes of this program version C: Cancel Please choose (S/V/K/I/C): s *** Uploading problem information The collected information is being sent to the bug tracking system. This might take a few minutes. 98% *** To continue, you must visit the following URL: https://bugs.launchpad.net/ubuntu/+source/dialer-app/+filebug/59dbaa1c-2162-11e3-a3d4-d485646cd9a4? You can launch a browser now, or copy this URL into a browser on another computer. Choices: 1: Launch a browser now C: Cancel Please choose (1/C): Choices: 1: Launch a browser now C: Cancel Please choose (1/C): 1
- The browser will launch and display a ready to file bug report. Finish filling in the description of the issue. When filing, make sure you include the steps needed for others to reproduce the issue, and what the result was versus what you expected the result to be.
There is a crash file in /var/crash
If the bug is a crash and there is a crash file in /var/crash, you can report it with apport.
1. Connect to the device with adb
$ adb shell
2. Check for any crash file in /var/crash
3. Report a bug with apport:
$ apport-cli /var/crash/name_of_crash_file.crash
4. Follow the instructions
5. To the question "What would you like to do? Your options are:", answer "S: Send report"
6. Wait for the upload to proceed
7. Upon upload you'll be presented with a prompt like:
*** To continue, you must visit the following URL: https://bugs.launchpad.net/ubuntu/+source/upstart-app-launch/+filebug/<UUID> You can launch a browser now, or copy this URL into a browser on another computer. Choices: 1: Launch a browser now C: Cancel
8. Copy the URL to a browser running on your desktop/laptop, it is more convenient than running a browser on the phone to report bug and proceed with the bug reporting process in launchpad
9. Then follow the usual tagging plan and triaging process
There is no crash file
1. Connect to the device with adb
$ adb shell
2. Find the right package (TODO: Document difference between preinstalled dpkg based packages and click package)
$ ubuntu-bug maliit-framework
3. Wait until data collection is done.
4. At the question:
What would you like to do? Your options are: S: Send report (4.3 KB) V: View report [...] https://bugs.launchpad.net/ubuntu/+source/maliit-framework/+filebug/<UUID>?
Press 'S' or 'K' if you want to copy the report to another machine (for example if networking is not enabled on the device).
3. If you're reporting from the device, on your local machine, click on the link or copy/paste it into your browser.
Triaging
Once bugs have been filed they need to be confirmed, and ideally, have a defined scenario for recreating in order for developers to fix them. To help with triaging, simply read a bug report from the list below and try it on the latest build for your device. If you can recreate it, mark the bug status as confirmed.
Quick links to buglists
Needs work
Bugs needing triaged (tagged with 'avengers' and a new or incomplete with response status)
Bugs that could be automated (tagged with 'touch-needs-autopilot'
Bugs in progress of being automated (tagged with 'qa-touch-automated' and a non-fix released status)
Completed work
Bugs that have been fixed (tagged with 'avengers' and a fix released status)
Bugs that have been automated (tagged with 'qa-touch-automated' and a fix released status)
List of all reported Bugs
Tags
Tags are used to keep handy lists of the work being done. It lets us see the status of bugs and testcase automation.
What does each tag mean?
Tag |
Description |
avengers |
Used for filing bugs found during exploratory testing of ubuntu touch images |
touch-needs-autopilot |
Used to tag bugs that are candidates for automation to prevent recurrence once fixed |
qa-touch-automated |
Used to tag bugs that are being automated with autopilot |
Tips and Tricks
Here are some helpful hints that will come in handy while testing; feel free to add more!
Switch from RO to RW
- Switch from RO to RW (to install additional packages for example):
- Remount RW
$ adb shell $ mount -o remount,rw /
- or touch writable_image
$ adb shell touch /userdata/.writable_image $ adb reboot
Take a screenshot
This is helpful for bug reporting. If you are using a build post 90 you are running with MIR by default.
MIR
Use this script to take a screenshot. Save the file and run passing a filename (ie screendump.png) as the sole argument. Ensure your device is connected to your pc before running, with the screen on and displaying the screenshot you wish to take.
This does not work with maguro due to a bug in the gpu, capturing the frame buffer kills the device.
Because of how the script works (by dumping the framebuffer directly), you may potentially experience invalid images or otherwise fail to get a screenshot.
SurfaceFlinger
$ adb shell /system/bin/screencap /data/screenshot.png $ adb pull /data/screenshot.png ~/screenshot.png
You can resize with convert from package imagemagick:
$ convert screenshot.png -resize 50% screenshot.png
Which build am I running?
- Find out what build you are running
$ adb shell cat /etc/media-info
Or for the details of the build (version of android part/version of ubuntu part)
root@ubuntu-phablet:/# system-image-cli -i current build number: 80 device name: mako channel: devel-proposed last update: 2013-10-04 09:59:14 version version: 80 version ubuntu: 20131003.2 version device: 20131003.2
Enable MIR by default
Switch to Mir enabled, SurfaceFlinger disabled
$ adb shell touch /home/phablet/.display-mir $ adb reboot to switch back: $ adb shell rm /home/phablet/.display-mir $ adb reboot
- MIR is running if no surfaceflinger process is running and the socket /tmp/mir_socket is present
- Tag bugs that exist only in MIR with mir-only
- MIR is running as default as of build 90. All versions released after 1.0 including 1.0 use MIR by default
Enable gprs if it doesn't come up automatically
- GPRS A simple work around while this is not working is to do:
$ adb shell restart network-manager
Turn back on intro
- Turn back on the initial demo that instructs how to use the phone via swiping
dbus-send --system --print-reply --dest=org.freedesktop.Accounts /org/freedesktop/Accounts/User32011 org.freedesktop.DBus.Properties.Set string:com.canonical.unity.AccountsService string:demo-edges variant:boolean:true
Retracing a crash on phone
In case you don't want to submit a crash to Launchpad to let them be handled by Ubuntu retracers, a way to generate nice stack trace from crash files in /var/crash/ and usable by developers is to use apport retrace directly on the phone.
- Switch the device to RW mode
Install apport-retrace: apt-get install apport-retrace
Enable ddebs repository to install debug symbol packages
- Run apport-cli on a crash files in /var/crash, for example:
$ apport-cli /var/crash/_usr_lib_arm-linux-gnueabihf_upstart-app-launch_zg-report-app.32011.crash
4.1. To the question Send problem report to the developers? answer E: Examine locally
4.2. To the question This will launch apport-retrace in a terminal window to examine the crash. answer 3: Update /var/crash/_usr_lib_arm-linux-gnueabihf_upstart-app-launch_zg-report-app.32011.crash with fully symbolic stack trace. Alternatively you can also get into an interactive gdb shell with all symbols loaded for in-depth debugging.
- Let apport proceed ...
Note: Make sure the phone is not blacked because in suspend mode, it will takes an infinite amount of time to do anything.
Retracing a phone crash on desktop
You'll first need some apt sources for apport to download -dbg and -dbgsym packages from. The configurations we use for the production retracers can be found in the Daisy project, so grab those:
bzr branch lp:daisy
You'll need the following packages on your desktop:
- apport-retrace
- gdb-multiarch
Next, on the phone, tell apport to collect some additional information for the crash. This adds the packaging information and runs any hooks:
$ apport-cli /var/crash/your_crash_file.crash
Hit view (V), then hit keep (K).
Finally, on your desktop, pull the crash from your phone and retrace your crash:
$ apport-retrace -S ~/bzr/daisy/retracer/config -o updated_crash_file.crash /var/crash/your_crash_file.crash
updated_crash_file.crash should now have full Stacktrace and ThreadStacktrace fields.
If you do this more than once it's helpful to use a permanent cache directory with e. g. -C ~/.cache/apport/ or something similar. The next time you retrace a crash it can then re-use the already downloaded debs, ddebs, and indexes.
You can throw that at Launchpad by running apport-cli updated_crash_file.crash, then hitting S to send the report.
Disable Screen Timeout
Run:
$ powerd-cli display on bright
Add APN Settings
'Note you'll need to be in r/w mode for this to work'
If you need to edit your APN settings for some reason, currently you have to manually edit the ofono config files. Perform the following:
- Shell into your phone or open a terminal
- Stop network manager and ofono
stop network-manager stop ofono
- Find your proper APN settings from your provider
Use the APN name as the AccessPointName in your gprs config file. The file is found at /var/lib/ofono/<Identity>/gprs
For example, for an ATT straight talk sim, this is a copy of the file that will work. Notice AccessPointName=tfdata matches the APN value given by straighttalk (att) below.
[Settings] Powered=1 RoamingAllowed=1 [context1] Name=STRAIGHT_TALK AccessPointName=tfdata Username= Password= Type=internet Protocol=ip
- Reboot your phone
- Enjoy working mobile data
MVNO
In general, check your MVNO's website for the settings information. You can use the required settings for any device, such as an android. Ignore the instructions and copy the information. It will look similar to the following:
Net10
NAME net10 APN tfdata MMSC http://mms-tf.net MMS PROXY mms3.tracfone.com MMS PORT 80
You only need the 'APN' field data to replace the AccessPointName= value in the gprs file.
Straight Talk Sim (Walmart)
The straight talk card might be using tmobile or att network. Here's the full APN details from for each.
Straight Talk (ATT)
NAME STRAIGHT_TALK APN tfdata MMSC http://mms-tf.net MMS PROXY mms3.tracfone.com MMS PORT 80 MMS_APN tfdata
Straight Talk (Tmobile)
NAME STRAIGHT_TALK APN wap.tracfone MMSC http://mms.tracfone.com MMS PROXY MMS PORT MMS_APN wap.tracfone
Debugging
Running autopilot tests
From a source branch
Save this bash script.
Execute it passing the source branch as the only argument. Ensure you have bzr and phablet-tools installed.
bash testclick.sh lp:calendar-app
Via installed click package
If the autopilot test suite is already on the device, you can run it like so. Connect the phone to your PC and run:
phablet-test-run -v TEST_NAME
The -v option gives more verbose output about failures and what is going on while testing, but isn't required for running. Replace TEST_NAME with the application test suite name you want to test. For example,
phablet-test-run -v calendar_app
or even something specific like
phablet-test-run -v calendar_app.tests.test_weekview.TestWeekView
Install a Package from Source
Sometimes we need to test the behavior of the bleeding-edge code that the developers are actively working on. To do this we take advantage of the Debian packaging system by building a package from source on the device and installing it there. Here we'll run through an example of installing lp:ubuntu-system-settings from source on a device. You'll need to set up for a writeable image. Do this as the phablet user and make a workspace for your codez:
$ adb shell $ su phablet $ mkdir ~/workspace
We'll need a few packages to begin:
$ sudo apt-get install -y bzr devscripts equivs
Now get trunk as if you were on your local machine.
$ cd ~/workspace $ bzr branch lp:ubuntu-system-settings
This installs all the packages necessary to build/run, should take a while for packages with lots of dependencies.
$ cd ~/workspace/ubuntu-system-settings $ sudo mk-build-deps -i debian/control
Modify the changelog to ensure that our built package will take priority over any installed version of the package.
$ dch -R
This opens an editor on the debian/changelog in the project directory (note that I had some trouble saving this file under nano, so vi is recommended):
ubuntu-system-settings (0.1+13.10.20131002-0ubuntu2) UNRELEASED; urgency=low * FILL THIS IN -- phablet <phablet@ubuntu-phablet> Wed, 02 Oct 2013 12:52:50 +0000 ubuntu-system-settings (0.1+13.10.20131002-0ubuntu1) saucy; urgency=low [ Sebastien Bacher ] * click: the installed-size info is in kib, not in bytes.
Note that our new changelog entry bears an incremented version number. Fill in something like "Test build." for a changelog entry (behind the asterisk), then exit and save the changelog.
Now build the package (live the life of a dev for a few mins ).
$ debuild -us -uc
The .deb has been deposited in the parent directory. Install it like you'd install any deb . Note that we may have built multiple deb files--if you find that you're not able to install a .deb due to a dependency issue, look for that dependency here first. In this case we'll discover that we need to install the 'libsystemsettings1' .deb before installing 'ubuntu-system-settings'.
phablet@ubuntu-phablet:~/workspace/ubuntu-system-settings$ ls .. libsystemsettings1_0.1+13.10.20131002-0ubuntu2_armhf.deb libsystemsettings-dev_0.1+13.10.20131002-0ubuntu2_armhf.deb ubuntu-system-settings ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2_armhf.build ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2_armhf.changes ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2_armhf.deb ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2.dsc ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2.tar.gz phablet:~/workspace/ubuntu-system-settings$ sudo dpkg -i ubuntu-system-settings_0.1+13.10.20131002-0ubuntu2_armhf.deb
Lastly verify that we're using the right package--the highlighted version should match the version of the new changelog entry we noted above.
phablet@ubuntu-phablet:~/workspace/ubuntu-system-settings$ apt-cache policy ubuntu-system-settings ubuntu-system-settings: Installed: 0.1+13.10.20131002-0ubuntu2 Candidate: 0.1+13.10.20131002-0ubuntu2 Version table: *** 0.1+13.10.20131002-0ubuntu2 0 100 /var/lib/dpkg/status 0.1+13.10.20131002-0ubuntu1 0 500 http://ports.ubuntu.com/ubuntu-ports/ saucy/universe armhf Packages
NOTE: do consult with your developers about additional steps necessary to see the behavior of your built package on a device--if it's a system service, e.g., it may need to be killed and restarted, or you may need a new session to see changes in unity itself. Every package is different!
Get a list of installed click packages
click list
Typing in just click will show you other commands you might be interested in for click packages
Debug an application via command line
To debug an application on the device, start it using the start application command. Application is an upstart user session and you need to be phablet user in order for this to work properly. From your pc, shell in, become phablet user, and run the start application command. Shown below is an example running the filemanager.
adb shell sudo -i -u phablet start application APP_ID=com.ubuntu.filemanager_filemanager_${version}
To run other applications, change the APP_ID to the application name. Using click list is helpful for finding the proper name.
The console output will be logged in /home/phablet/.cache/upstart/. For example, for filemanager as shown above the log exists in /home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.1.1.80.log.
Play a video from the command line
The general scheme for starting the media player from the command line via upstart is:
start application APP_ID=mediaplayer-app APP_URIS="file_name"
This can be used for other apps by changing the APP_ID.
QATeam/TouchTesting (last edited 2015-01-22 17:18:23 by xdsl-83-150-81-40)