Silo Testing Guidelines

This document provides a set of guidelines for testing silos prior to approving them for landing. Several categories of tests will be run, plus exploratory tests.

Component Test Plan

QA will identify the appropriate component test plans to run and execute it independently of the development team. An index of the test plans is at: http://wiki.ubuntu.com/Merges/TestPlans

QA Test Plan

Where the change is significant enough to warrant it, a whole run through the manual test plan for QA will be performed. This will only be done for core components: unity8, ubuntu-ui-toolkit, mir, qt-ubuntu

Exploratory Testing

The silo tester will perform some additional exploratory testing around the component that is landing. Any bugs found should be reported back to the developer for fixing and a bug should be created and tagged with ‘testcase’ so that a test case can be created from the exploratory test and added to the appropriate test plan.

Language checks

Look after changes in the UI to spot untranslated strings. Switch the phone to a non-English language and check that the component strings are correctly translated. Use citrain-diff tool to review the modifications. Special attention to QML files: text and label properties must be marked with i18n.tr(), dtr() or ctr(). If new strings are landing, check whether the corresponding .pot file in the source code is being updated too.

Trello Board

We are using a Trello board for tracking silo testing requests. Please follow a few bits of etiquette when using it, for the benefit of other silo testers and the landers:

• Only put a card in the ‘Under Testing’ column if you are definitely going to work on it - not if you are planning to at some point in the future - the Trello board must reflect the actual status of testing.
• Assign yourself to the card using the ‘Members’ option and set a ‘Due Date’ to an exact time when you think you will be done - it can be modified if you run into problems.
• For any major change in the card status, add a comment (column changes, especially when going from ‘Under Testing’ back to ‘Needs QA Sign-off’, due date changes). Also feel free to put any comments you feel would be useful.
• If you can’t finish a silo before you end your day - make sure you leave all of the information that might be necessary for someone else who picks it up to avoid repeating too much of what you did - e.g. if there are any test plans you fully completed, note them.

* If there is an issue with the silo such as insufficient information or a regression that you think is caused by the silo then make sure the contact the lander immediately. If you can't then add the 'Blocked' tag to the card by selecting 'Edit Labels' and ticking the Red 'Blocked' tag.

FAQ

Which Silo should I be testing?

Look on the Trello board in the ‘Needs QA Sign-off’ column. Follow a few rules of thumb to pick one if there are several:

Pick one that has the lander present, so you can ask them questions (and they’ll appreciate it landing during their day) Prefer smaller silos to big ones after that.

Where can I find details of the Silo?

Details of the silo can be found on the Landing Spreadsheet. It will give details on which test plans to run and any other details needed for testing. If you look at the silo on the dashboard you will see it has a description including ‘spreadsheet row #’, which will tell you where it is. Most relevant things are the ‘Testing pass’ column which gives details of who tested the silo from the development team, which device they used and which image number, as well as the Test plans What do I need to do to set up the device for testing?

Run ubuntu-device-flash --channel=<channel> --wipe --device=krillin

<channel> should be devel-proposed for ubuntu silos (during TRAINCON-0) and ubuntu-touch/ubuntu-rtm/14.09-proposed for ubuntu-rtm silos.

After that you need to add the silo. On ubuntu this is easy - use citrain device-upgrade <silo #>. Currently with ubuntu-rtm you need to manually edit /etc/apt/sources.list and add a line for the Silo PPA. This can be found in the ‘Technical details for this PPA’ section of the Silos PPA page (e.g. https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu-rtm/landing-001)

After installing the silo, from system settings, reset the device so that the user data area is clean.

What do I do if something fails?

Make sure to confirm that it’s a regression caused by the silo. Reflash the image and retest for the failure. If the regression is caused by the silo mark it as failed. File a bug against the source package you think the bug is in (there may be several in the silo).

How to pause an upgrade to insert a fix (OTA)

If there is an ota fix that you need to enable then the way to do that is as follows:

• Install image 16 from rtm

adb reboot bootloader
ubuntu-device-flash --revision 16 touch --bootstrap --channel ubuntu-touch/ubuntu-rtm/14.09 --recovery-image recovery.img

• Add the data you need to test
• Tap on updates
• Wait for the BIG spinning ubuntu logo to finish
• Press volume up and hold, before the system boots

• (optional 3 steps if the recovery lacks adb) Select fastboot mode

• fastboot flash recovery recovery.img
• fastboot reboot
• Press and hold the volume up key again
• Go into recovery
• adb shell
• enable the env

~ # export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
~ # mount /system
~ # chroot /system /bin/bash

• mkdir /system/silo<number>

• In an other terminal download the debs from the ppa and then adb them to /system/silo<number>

• in the first terminal do cd /system/silo<number>

• dpkg -i *.deb
• exit
• exit
• adb reboot

Backup/Restore

Backup

• Flash the base image (224 is the equivalent of 16 the factory image)
• During the welcome wizard skip wifi setup and don;t unlock sims so there is no network
• Once the phone is up and running open system settings and updates
• Change wifi bottom right to never
• Now enable wifi and unlock sim
• Make all the modifications you need to make to the system
• Enable developer mode
• adb reboot bootloader
• fastboot flash recovery recovery.img
• fastboot reboot
• adb reboot recovery
• adb shell
• export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

• find /data -type d \( -path /data/android-data/nvram \) -prune -o -print | cpio -ovH newc > /tmp/backup<version number>.cpio

• In another terminal do adb pull /tmp/backup<version number>.cpio .

• exit
• adb reboot

Restore

• Flash the base image (224 is the equivalent of 16 the factory image)
• Enable developer mode
• adb reboot bootloader
• fastboot flash recovery recovery.img
• fastboot reboot
• adb reboot recovery
• adb shell
• export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

• In another terminal do adb push backup<version number>.cpio /tmp/

• now in the original system do cpio -ivdu < /tmp/backup<version number>.cpio

• exit
• adb reboot