• Test plan for: Ubuntu UI Toolkit

  • Component checklist:https://wiki.ubuntu.com/Process/Merges/Checklists/ui-toolkit

  • Trunk URL: lp:ubuntu-ui-toolkit

  • Ubuntu Package URL (LP): https://launchpad.net/ubuntu/+source/ubuntu-ui-toolkit

Test Plan

Install latest (utopic-proposed) image on phone. Install freshly build MPs that are needed for landing Ensure that all AP tests pass on the device what are marked OK on the CI dash.

• - p reminders-app-autopilot reminders
• -p dialer-app-autopilot dialer_app
• -p messaging-app-autopilot messaging_app
• -p ubuntu-system-settings-autopilot ubuntu_system_settings
• -p ubuntu-html5-ui-toolkit-autopilot ubuntu_html5_ui_toolkit
• -p unity-webapps-qml-autopilot unity_webapps_qml
• filemanager
• dropping_letters_app
• sudoku_app
• -p ubuntu-ui-toolkit-autopilot ubuntuuitoolkit
• -p webbrowser-app-autopilot webbrowser_app
• -n unity8
• shorts_app
• calendar_app
• music_app
• -p mediaplayer-app-autopilot mediaplayer_app
• ubuntu_terminal_app
• ubuntu_clock_app
• ubuntu_calculator_app
• ubuntu_weather_app
• -p address-book-app-autopilot address_book_app
• online_accounts_ui
• gallery_app
• -p camera-app-autopilot camera_app

Local Validation

$ bzr branch lp:ubuntu-ui-toolkit/staging $ cd ubuntu-ui-toolkit/tests

The uitk_test_plan.sh is the script to execute the test plan

Usage

uitk_test_plan.sh -s [serial number] -m -c

• -r : Reset after each tests.
• -s: Use the device with the given serial number. The default is to use the first available adb device
• -c : Comission the device with the PPA enabled
• -n : Do not run the test set. Useful just to comission the device.
• -o : Output directory.
• -p : Source PPA for the UITK. Use -p archive to test stock image.
• -f : Filter for the test suite.

Example

$./uitk_test_plan.sh -c -p ppa:ci-train-ppa-service/landing-010 -f music

• This will flash the image from the ubuntu-touch/utopic-propose, set up the click test nd install the UITK from the Silo10 PPA
• The script dumps the test results in the local directory and tries max 3 times if an app test is not OK.

Typical use cases

By default the uitk_test_plan.sh flashes the latest RTM image on the device, installs the click application tests, configures the ppa:ubuntu-sdk-team/staging PPA, installs the UITK from the PPA and executes the test plan.

Validate the staging branch of the UITK against the RTM image

$ ./uitk_test_plan.sh -c

Validate the UITK from a CI silo on an RTM image

$ ./uitk_test_plan.sh -c -p 001

Validate the UITK from a specific CI silo on an Ubuntu Utopic image

$ ./uitk_test_plan.sh -c -p 001 -u

Provision the device for manual testing with the latest RTM image

$ ./uitk_test_plan.sh -c -p archive -n

Provision the device for manual testing with the latest Ubuntu Utopic image

$ ./uitk_test_plan.sh -c -p archive -u -n

Run the test plan on an already provisioned device

$ ./uitk_test_plan.sh

Run only the UITK tests on an already provisioned device

$ ./uitk_test_plan.sh -f ubuntuuitoolkit

Suggested process by QA

• First, flash the latest image.
• Boot into the image, go through the wizard, get rid of the welcome tutorial, and disable automatic updates.
• adb shell mount -o remount,rw /
• Set up click tests: phablet-click-test-setup (or, for RTM: phablet-click-test-setup --distribution=ubuntu-rtm --series=14.09 )
• apt-get install python3-lxml
• Remove the UITK tests which were just installed: adb shell rm -rf /home/phablet/autopilot/ubuntuuitoolkit
• Install the silo. (add the PPA, disable all other package sources, apt-get update, apt-get dist-upgrade, re-enable other package sources)
• adb shell reboot
• Grab the UITK source: bzr branch lp:ubuntu-ui-toolkit
• cd ubuntu-ui-toolkit/tests/
• mkdir results.silo results.base
• Modify uitk_test_plan.sh as needed. Might need to change the variables at the top, perhaps more depending on your specific needs.
• time ./uitk_test_plan.sh -r -o results.silo
• Save the overall output into results.silo/summary.txt (or similar).

If anything failed (and it probably did), reflash without the silo then re-test the components which failed. Requires editing uitk_test_plan.sh again, to remove components which don't need re-testing. Then compare the results against the first run. If the silo has the same or fewer failures as the base image, it passes.

Note that several AP tests like to produce a different number of failures on each test run. This can greatly complicate the above process.