• Test plan for component: ubuntu-system-image

• Component Checklist: https://wiki.ubuntu.com/Process/Merges/Checklists/ubuntu-system-image

• Trunk URL: lp:ubuntu-system-image/client

• Ubuntu Package URL (LP): http://launchpad.net/ubuntu/+source/system-image

• Project wiki page: https://wiki.ubuntu.com/ImageBasedUpgrades

Dependents/Clients

Dependents:

• ubuntu-download-manager

Clients:

• ubuntu-system-settings

General Recommendations

Note: You'll want to begin each test in a "fresh" state. Specifically, make sure that before you start a test, the system-image-dbus process is not running. I've noticed that this process may already be running after a reboot, and this could affect the success of your test. You can check this with:

adb shell ps -waux | grep system-image-dbus

If you see something like the following, then the process is running and you'll need to kill it (e.g. via SIGTERM) or rebooting.

You may also want to clear the log file of any previous results:

truncate -s 0 /var/log/system-image/client.log

If you're testing packages in a landing silo, after flashing your device as instructed in the test plans below, you will want to install the new packages from the silo. You can do that this way:

• Install freshly built MPs that are needed for landing, e.g.
  • Make device writable

    • touch /userdata/.writable_image

    • reboot

  • add-apt-repository the relevant PPA.

  • apt-get update

  • apt-get install system-image-cli system-image-common system-image-dbus

• Do not install the system-image-dev package as this is normally not installed on devices and its absence/presence has an effect on the operation of system-image.

• Do not do an apt-get upgrade

• Install any new versions of ubuntu-download-manager and ubuntu-system-settings from the same silo.

• Possibly install any new versions of the Qt networking stack that may also be part of the silo.

Alternatively, just download the .debs from the silo PPA and dpkg -i *.deb manually.

If any of the tests fail, file a bug attaching the following information. It is requested that you file the bug on the upstream package (using that link), although it's okay -- but not necessary -- to also file a bug on the Ubuntu package. In either case, please tag the bug with the client tag.

• If the test fails, capture the log files:

  • adb pull /var/log/system-image/client.log

  • Optionally capture the u-d-m log files (TBD)
• Attach these logs to the bug.

After you flash your device with --bootstrap you will probably lose your network settings. Restore them with:

• adb shell nmcli dev wifi connect <network> password <password>

There are two sets of test plans depending on whether you're QAing system-image 2.5 or system-image 3.

Test Plan for system-image 3

This test plan is not supposed to be complete; use it to guide your manual testing so they don't miss big functional areas that are part in the component; also this should be used as guideline to inspire the exploratory testing which should be adapted smartly based on the real content of a merge proposal.

Test A -- Good Path Automatic Download

This tests standard upgrading from an older version to the current version. It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.

• cd /etc/system-image

• system-image-cli --info | grep "current build number"

• Subtract 1 from the build number

• printf "[service]\nbuild_number: NNN\n" > config.d/02_testing.ini where NNN is the current build number minus 1

• system-image-cli --info | grep "current build number" to verify the new, lower build number

• tail -f /var/log/system-image/client.log

• Open System Settings panel, click on Updates
• Wait for it to say there is an update available.
• Wait for updates to finish downloading.

• Click on Restart & install

• After the reboot, you MUST first rm -f /etc/system-image/config.d/02_testing.ini

• Open System Settings panel, then Updates. It should report that your Software is up to date

Test B -- Good Path Manual Download

This tests standard upgrading from an older version to the current version. It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.

• Enable manual downloads: system-image-cli --set auto_download=0

• cd /etc/system-image

• system-image-cli --info | grep "current build number"

• Subtract 1 from the build number

• printf "[service]\nbuild_number: NNN\n" > config.d/02_testing.ini where NNN is the current build number minus 1

• Restart system-image-dbus like so: kill -TERM `ps -C system-image-dbus -o pid=` - ignore this if it prints an error; that just means that the process isn't running.

• system-image-cli --info | grep "current build number" to verify the new, lower build number

• tail -f /var/log/system-image/client.log

• Open System Settings panel

• Click on Updates

• Click on Download

• Wait for the Download button to say Install...

• Click on Install...

• Click on Restart & install

• After the reboot, you MUST first rm -f /etc/system-image/config.d/02_testing.ini and restart system-image-dbus as above

• Open System Settings panel, then Updates. It should report that your Software is up to date

• system-image-cli --set auto_download=1

Test Plan for system-image 2.5

This test plan is not supposed to be complete; use it to guide your manual testing so they don't miss big functional areas that are part in the component; also this should be used as guideline to inspire the exploratory testing which should be adapted smartly based on the real content of a merge proposal.

Test A -- Good Path Automatic Download

This tests standard upgrading from an older version to the current version. It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.

• Flash device to an older version than the current one, e.g. ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --revision -5 --bootstrap

• Make sure system-image-dbus isn't running -- see above

• Open System Settings panel.
• Wait for it to say there is an update available.

• Click on Updates available

• Wait for updates to finish downloading.

• Click on Install & restart

• After the reboot, open System Settings panel, then Updates. It should report that your Software is up to date

Test B -- Good Path Automatic Download Current Packages

This tests standard upgrading like above, but it uses the current set of packages and fakes the system upgrading to a newer version. It actually upgrades from the newest version to the newest version because we're going to lie about what version the device is actually on. This is a good test to run to ensure that the current latest build is working as expected.

• Flash device to the current version, e.g. ubuntu-device-flash --channel ubuntu-touch/devel-proposed --bootstrap

• Falsify which version your device thinks it's on:

  • adb shell

  • cd /etc/system-image

  • cat channel.ini

  • Look at the build_number line to get the current build number. Subtract 1. (Next line assumes you're at build 246.)

  • sed -e "s/246/245/" -i.bak channel.ini

  • (If this gets messed up, just restore channel.ini from channel.ini.bak)

• Make sure system-image-dbus isn't running -- see above

• Open System Settings panel.
• Wait for it to say there is an update available.

• Click on Updates available

• Wait for updates to finish downloading.

• Click on Install & restart

• After the reboot, open System Settings panel, then Updates. It should report that your Software is up to date

Test C -- Good Path Manual Download

This tests standard upgrading from an older version to the current version. It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.

• Flash device to an older version than the current one, e.g. ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --revision -1 --bootstrap

• Enable manual downloads:

  • adb shell system-image-cli --set auto_download=0

• Open System Settings panel
• Wait for it to say there is an update available.

• Click on Updates available

• Click on Install 1 update

• Click on Install & restart

• After the reboot, open System Settings panel, then Updates. It should report that your Software is up to date

• Restore automatic downloads!

  • adb shell system-image-cli --set auto_download=1

Test D -- Good Path Manual Download Current Packages

This tests standard upgrading like above, but it uses the current set of packages and fakes the system upgrading to a newer version. It actually upgrades from the newest version to the newest version because we're going to lie about what version the device is actually on. This is a good test to run to ensure that the current latest build is working as expected.

• Flash device to the current version, e.g. ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --bootstrap

• Enable manual downloads:

  • adb shell system-image-cli --set auto_download=0

• Falsify which version your device thinks it's on:

  • adb shell

  • cd /etc/system-image

  • cat channel.ini

  • Look at the build_number line to get the current build number. Subtract 1. (Next line assumes you're at build 246.)

  • sed -e "s/246/245/" -i.bak channel.ini

  • (If this gets messed up, just restore channel.ini from channel.ini.bak)

• Open System Settings panel.
• Wait for it to say there is an update available.

• Click on Updates available

• Click on Install 1 update

• Wait for updates to finish downloading.

• Click on Install & restart

• After the reboot, open System Settings panel, then Updates. It should report that your Software is up to date

• Restore automatic downloads!

  • adb shell system-image-cli --set auto_download=1

Test E -- Auto Download Multiple Checks

• Flash device to an older version than the current one, e.g. ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --revision -1 --bootstrap

• Open System Settings panel
• After one second, open Updates panel
• Watch for progress
• If update completes and you are prompted to reboot, then the test succeeded
• Optionally reboot
• Install the update, let the system reboot, and then try again. This time you should be up-to-date.

Test F

• Set the device to download manually.
• Flash to an older revision

• Open System Settings and wait for it to say Updates available

• Click on About this phone

• Click on Check for Update and wait for it to say Install 1 update

• Click on Install 1 update and while the files are downloading, click Back

• Click on Check for Update again

• Wait for the Update System overlay to come up, and then install the update, and reboot

Test G -- Phased Updates

This test verifies the operation of phased updates. Phased updates means that a percentage is assigned to an end-point target image, and only that percentage of devices should be updated to that image. Devices with a percentage outside that range will fall back to the previous image. The server guarantees that only end-point images will have a phased update other than 100% (and the absence of a phased-percentage key in the JSON is equivalent to 100%). The client ignores phased percentages on any mid-point image in an upgrade path. An image with a phased percentage of 0% is never selected.

As an example, let's say that channel 'devel' has an image 10 and a new image 11 is released with a phased percentage of 25%. Each device on channel 'devel' calculates its own percentage (its guinea pig value) from the following tuple of information:

• The unique device UUID
• The update channel (e.g. 'devel' in this example)
• The highest target image number of all candidate upgrade paths (e.g. 11 in this example)

Let's say the device calculates its percentage as 17%; this device would elect to apply the image 11 update. Let's say another device calculates its percentage as 33%; this device would elect to ignore the image 11 update.

When there are multiple upgrade candidate paths available, the client scores each candidate independent of the phased percentage, and sorts them in winning order from lowest score to highest. If the lowest scored path has a phased percentage lower than the device's percentage, this candidate is discarded and the next lowest scoring candidate is considered. This continues until a candidate upgrade path with a phased percentage greater than or equal to the device's percentage; this candidate is then selected as the winning upgrade path. If no candidate matches this criteria, the device is considered up-to-date.

A missing phase percentage means every device should update; it's equivalent to an image percentage of 100%. An image percentage of 0% means that the image has been effectively pull from release, and no devices will update to it.

To test this scenario, we need a staging server with a known end-point update on a particular channel with a non-100% phase percentage. We currently have available:

• host https://phablet.stgraber.org

• channel ubuntu-touch/test

• device mako

You also need a way to override the device's default phase percentage calculation, which can be done with system-image-cli -p.

Performing the test

• cd /etc/system-image

• Grab the staging server's archive master keys:

  • mv archive-master.tar.xz{,.aside}

  • mv archive-master.tar.xz.asc{,.aside}

  • wget https://phablet.stgraber.org/gpg/archive-master.tar.xz

  • wget https://phablet.stgraber.org/gpg/archive-master.tar.xz.asc`

• Edit /etc/system-image/channel.ini (not client.ini)

  • Change [service]base to phablet.stgraber.org

• Do a dry-run update with your device's natural phased percentage.

  • system-image-cli -c "ubuntu-touch/test" -d mako -b 0 --dry-run -v

  • You will either see an upgrade path to image 1 (with the device's phased percentage included in the output), or the device will claim to be up-to-date
• Do a dry-run update with a forced phase percentage that is within image 1's phase:

  • system-image-cli -c "ubuntu-touch/test" -d mako -b 0 --dry-run -v -p 33

  • You should see upgrade path to image 1 and a target phase of 33%
• Do a dry-run update with a forced phase percentage outside the image phase:

  • system-image-cli -c "ubuntu-touch/test" -d mako -b 0 --dry-run -v -p 66

  • The device will claim to be up-to-date.

• Try some other dry-run updates with various -p options. The image's phase is 40%

Be sure to restore your archive-master files and channel.ini file if you want to do other tests.

Additional tests candidates for automation

The following test cases are being automated. Work items are described in this blueprint.

Testing the image update path

• Test that updates from image n-1 work
• Test that updates from image n-2 work

Testing the image updater software

• Test that the updater software in image n can do an update to a single newer version in a synthetic channel
• Test that the updater software in image n can do a multi-step update (applying 5? updates in sequence) in a synthetic channel
• Test that the updater software in image n correctly applies a full-image update when it’s a smaller download than a series of stacked partial upgrades