Contents

  1. Silo Testing Guidelines
    1. Component Test Plan
    2. QA Test Plan
    3. Exploratory Testing
    4. Language checks
    5. Trello Board
    6. FAQ
      1. Which Silo should I be testing?
      2. Where can I find details of the Silo?
      3. Install silos with overlay PPA enabled
      4. What do I do if something fails?
      5. How to pause an upgrade to insert a fix (OTA)
      6. Backup/Restore

Silo Testing Guidelines

This document provides a set of guidelines for testing silos prior to approving them for landing. Several categories of tests will be run, plus exploratory tests.

Component Test Plan

QA will identify the appropriate component test plans to run and execute it independently of the development team. An index of the test plans is at: http://wiki.ubuntu.com/Merges/TestPlans

QA Test Plan

Where the change is significant enough to warrant it, a whole run through the manual test plan for QA will be performed. This will only be done for core components: unity8, ubuntu-ui-toolkit, mir, qt-ubuntu

Exploratory Testing

The silo tester will perform some additional exploratory testing around the component that is landing. Any bugs found should be reported back to the developer for fixing and a bug should be created and tagged with ‘testcase’ so that a test case can be created from the exploratory test and added to the appropriate test plan.

Language checks

Look after changes in the UI to spot untranslated strings. Switch the phone to a non-English language and check that the component strings are correctly translated. Use citrain-diff tool to review the modifications. Special attention to QML files: text and label properties must be marked with i18n.tr(), dtr() or ctr(). If new strings are landing, check whether the corresponding .pot file in the source code is being updated too.

Trello Board

We are using a Trello board for tracking silo testing requests. Please follow a few bits of etiquette when using it, for the benefit of other silo testers and the landers:

* If there is an issue with the silo such as insufficient information or a regression that you think is caused by the silo then make sure the contact the lander immediately. If you can't then add the 'Blocked' tag to the card by selecting 'Edit Labels' and ticking the Red 'Blocked' tag.

FAQ

Which Silo should I be testing?

Look on the Trello board in the ‘Needs QA Sign-off’ column. Follow a few rules of thumb to pick one if there are several:

Pick one that has the lander present, so you can ask them questions (and they’ll appreciate it landing during their day) Prefer smaller silos to big ones after that.

Where can I find details of the Silo?

Details of the silo can be found on the Landing Spreadsheet. It will give details on which test plans to run and any other details needed for testing. If you look at the silo on the dashboard you will see it has a description including ‘spreadsheet row #’, which will tell you where it is. Most relevant things are the ‘Testing pass’ column which gives details of who tested the silo from the development team, which device they used and which image number, as well as the Test plans What do I need to do to set up the device for testing?

Run 

adb reboot bootloader
ubuntu-device-flash touch --bootstrap --channel=<channel> --recovery-image=recovery.img

recovery.img can be obtained from the following places:
Arale http://people.canonical.com/~alextu/tangxi/recovery/recovery.img
Krillin http://people.canonical.com/~jhm/barajas/recovery.img

<channel> should be a place where you are testing
ubuntu-touch/stable=Current released image (e.g RTM 14.10)
ubuntu-touch/rc-proposed=Current wip release (e.g. Vivid 15.04)
ubuntu-touch/devel-proposed=Next wip release (e.g. Wily 15.10)

After that you need to add the silo. On ubuntu this is easy - use citrain device-upgrade <silo #> <pin/password on device>.
Warning /!\ This will fail if you do not unlock the device after the initial reboot, it will also fail if you don't keep the screen active after that till the second reboot.

sudo apt install phablet-tools-citrain If you don't have the package installed.

After installing the silo, from system settings, reset the device so that the user data area is clean.

Install silos with overlay PPA enabled

When testing silos with the overlay PPA is enabled, the priority of silo packages needs to be set in /etc/apt/preferences.d/filename.pref, similar to this: 

Package: *
Pin: release o=LP-PPA-ci-train-ppa-service-landing-027
Pin-Priority: 1002

What do I do if something fails?

Make sure to confirm that it’s a regression caused by the silo. Reflash the image and retest for the failure. If the regression is caused by the silo mark it as failed. File a bug against the source package you think the bug is in (there may be several in the silo).

How to pause an upgrade to insert a fix (OTA)

If there is an ota fix that you need to enable then the way to do that is as follows: 

adb reboot bootloader
ubuntu-device-flash --revision 16 touch --bootstrap --channel ubuntu-touch/ubuntu-rtm/14.09 --recovery-image recovery.img

~ # export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
~ # mount /system
~ # chroot /system /bin/bash

Backup/Restore

Backup

Restore

LandingTeam/SiloTestingGuidelines (last edited 2015-08-19 08:26:22 by timo-jyrinki)