ubuntu-system-image
Test plan for component: ubuntu-system-image
Component Checklist: https://wiki.ubuntu.com/Process/Merges/Checklists/ubuntu-system-image
Trunk URL: lp:ubuntu-system-image/client
Ubuntu Package URL (LP): http://launchpad.net/ubuntu/+source/system-image
Project wiki page: https://wiki.ubuntu.com/ImageBasedUpgrades
Contents
Dependents/Clients
Dependents:
- ubuntu-download-manager
Clients:
- ubuntu-system-settings
General Recommendations
Note: You'll want to begin each test in a "fresh" state. Specifically, make sure that before you start a test, the system-image-dbus process is not running. I've noticed that this process may already be running after a reboot, and this could affect the success of your test. You can check this with:
adb shell ps -waux | grep system-image-dbus
If you see something like the following, then the process is running and you'll need to kill it (e.g. via SIGTERM) or rebooting.
You may also want to clear the log file of any previous results:
truncate -s 0 /var/log/system-image/client.log
If you're testing packages in a landing silo, after flashing your device as instructed in the test plans below, you will want to install the new packages from the silo. You can do that this way:
- Install freshly built MPs that are needed for landing, e.g.
- Make device writable
touch /userdata/.writable_image
- reboot
add-apt-repository the relevant PPA.
apt-get update
apt-get install system-image-cli system-image-common system-image-dbus
- Make device writable
Do not install the system-image-dev package as this is normally not installed on devices and its absence/presence has an effect on the operation of system-image.
Do not do an apt-get upgrade
Install any new versions of ubuntu-download-manager and ubuntu-system-settings from the same silo.
- Possibly install any new versions of the Qt networking stack that may also be part of the silo.
Alternatively, just download the .debs from the silo PPA and dpkg -i *.deb manually.
If any of the tests fail, file a bug attaching the following information. It is requested that you file the bug on the upstream package (using that link), although it's okay -- but not necessary -- to also file a bug on the Ubuntu package. In either case, please tag the bug with the client tag.
- If the test fails, capture the log files:
adb pull /var/log/system-image/client.log
- Optionally capture the u-d-m log files (TBD)
- Attach these logs to the bug.
After you flash your device with --bootstrap you will probably lose your network settings. Restore them with:
adb shell nmcli dev wifi connect <network> password <password>
There are two sets of test plans depending on whether you're QAing system-image 2.5 or system-image 3.
Test Plan for system-image 3
This test plan is not supposed to be complete; use it to guide your manual testing so they don't miss big functional areas that are part in the component; also this should be used as guideline to inspire the exploratory testing which should be adapted smartly based on the real content of a merge proposal.
Test A -- Good Path Automatic Download
This tests standard upgrading from an older version to the current version. It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.
cd /etc/system-image
system-image-cli --info | grep "current build number"
- Subtract 1 from the build number
printf "[service]\nbuild_number: NNN\n" > config.d/02_testing.ini where NNN is the current build number minus 1
system-image-cli --info | grep "current build number" to verify the new, lower build number
tail -f /var/log/system-image/client.log
- Open System Settings panel, click on Updates
- Wait for it to say there is an update available.
- Wait for updates to finish downloading.
Click on Restart & install
After the reboot, you MUST first rm -f /etc/system-image/config.d/02_testing.ini
Open System Settings panel, then Updates. It should report that your Software is up to date
Test B -- Good Path Manual Download
This tests standard upgrading from an older version to the current version. It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.
Enable manual downloads: system-image-cli --set auto_download=0
cd /etc/system-image
system-image-cli --info | grep "current build number"
- Subtract 1 from the build number
printf "[service]\nbuild_number: NNN\n" > config.d/02_testing.ini where NNN is the current build number minus 1
Restart system-image-dbus like so: kill -TERM `ps -C system-image-dbus -o pid=` - ignore this if it prints an error; that just means that the process isn't running.
system-image-cli --info | grep "current build number" to verify the new, lower build number
tail -f /var/log/system-image/client.log
- Open System Settings panel
Click on Updates
Click on Download
Wait for the Download button to say Install...
Click on Install...
Click on Restart & install
After the reboot, you MUST first rm -f /etc/system-image/config.d/02_testing.ini and restart system-image-dbus as above
Open System Settings panel, then Updates. It should report that your Software is up to date
system-image-cli --set auto_download=1
Test Plan for system-image 2.5
This test plan is not supposed to be complete; use it to guide your manual testing so they don't miss big functional areas that are part in the component; also this should be used as guideline to inspire the exploratory testing which should be adapted smartly based on the real content of a merge proposal.
Test A -- Good Path Automatic Download
This tests standard upgrading from an older version to the current version. It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.
Flash device to an older version than the current one, e.g. ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --revision -5 --bootstrap
Make sure system-image-dbus isn't running -- see above
- Open System Settings panel.
- Wait for it to say there is an update available.
Click on Updates available
- Wait for updates to finish downloading.
Click on Install & restart
After the reboot, open System Settings panel, then Updates. It should report that your Software is up to date
Test B -- Good Path Automatic Download Current Packages
This tests standard upgrading like above, but it uses the current set of packages and fakes the system upgrading to a newer version. It actually upgrades from the newest version to the newest version because we're going to lie about what version the device is actually on. This is a good test to run to ensure that the current latest build is working as expected.
Flash device to the current version, e.g. ubuntu-device-flash --channel ubuntu-touch/devel-proposed --bootstrap
- Falsify which version your device thinks it's on:
adb shell
cd /etc/system-image
cat channel.ini
Look at the build_number line to get the current build number. Subtract 1. (Next line assumes you're at build 246.)
sed -e "s/246/245/" -i.bak channel.ini
(If this gets messed up, just restore channel.ini from channel.ini.bak)
Make sure system-image-dbus isn't running -- see above
- Open System Settings panel.
- Wait for it to say there is an update available.
Click on Updates available
- Wait for updates to finish downloading.
Click on Install & restart
After the reboot, open System Settings panel, then Updates. It should report that your Software is up to date
Test C -- Good Path Manual Download
This tests standard upgrading from an older version to the current version. It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.
Flash device to an older version than the current one, e.g. ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --revision -1 --bootstrap
- Enable manual downloads:
adb shell system-image-cli --set auto_download=0
- Open System Settings panel
- Wait for it to say there is an update available.
Click on Updates available
Click on Install 1 update
Click on Install & restart
After the reboot, open System Settings panel, then Updates. It should report that your Software is up to date
Restore automatic downloads!
adb shell system-image-cli --set auto_download=1
Test D -- Good Path Manual Download Current Packages
This tests standard upgrading like above, but it uses the current set of packages and fakes the system upgrading to a newer version. It actually upgrades from the newest version to the newest version because we're going to lie about what version the device is actually on. This is a good test to run to ensure that the current latest build is working as expected.
Flash device to the current version, e.g. ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --bootstrap
- Enable manual downloads:
adb shell system-image-cli --set auto_download=0
- Falsify which version your device thinks it's on:
adb shell
cd /etc/system-image
cat channel.ini
Look at the build_number line to get the current build number. Subtract 1. (Next line assumes you're at build 246.)
sed -e "s/246/245/" -i.bak channel.ini
(If this gets messed up, just restore channel.ini from channel.ini.bak)
- Open System Settings panel.
- Wait for it to say there is an update available.
Click on Updates available
Click on Install 1 update
- Wait for updates to finish downloading.
Click on Install & restart
After the reboot, open System Settings panel, then Updates. It should report that your Software is up to date
Restore automatic downloads!
adb shell system-image-cli --set auto_download=1
Test E -- Auto Download Multiple Checks
Flash device to an older version than the current one, e.g. ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --revision -1 --bootstrap
- Open System Settings panel
- After one second, open Updates panel
- Watch for progress
- If update completes and you are prompted to reboot, then the test succeeded
- Optionally reboot
- Install the update, let the system reboot, and then try again. This time you should be up-to-date.
Test F
- Set the device to download manually.
- Flash to an older revision
Open System Settings and wait for it to say Updates available
Click on About this phone
Click on Check for Update and wait for it to say Install 1 update
Click on Install 1 update and while the files are downloading, click Back
Click on Check for Update again
Wait for the Update System overlay to come up, and then install the update, and reboot
Test G -- Phased Updates
This test verifies the operation of phased updates. Phased updates means that a percentage is assigned to an end-point target image, and only that percentage of devices should be updated to that image. Devices with a percentage outside that range will fall back to the previous image. The server guarantees that only end-point images will have a phased update other than 100% (and the absence of a phased-percentage key in the JSON is equivalent to 100%). The client ignores phased percentages on any mid-point image in an upgrade path. An image with a phased percentage of 0% is never selected.
As an example, let's say that channel 'devel' has an image 10 and a new image 11 is released with a phased percentage of 25%. Each device on channel 'devel' calculates its own percentage (its guinea pig value) from the following tuple of information:
- The unique device UUID
- The update channel (e.g. 'devel' in this example)
- The highest target image number of all candidate upgrade paths (e.g. 11 in this example)
Let's say the device calculates its percentage as 17%; this device would elect to apply the image 11 update. Let's say another device calculates its percentage as 33%; this device would elect to ignore the image 11 update.
When there are multiple upgrade candidate paths available, the client scores each candidate independent of the phased percentage, and sorts them in winning order from lowest score to highest. If the lowest scored path has a phased percentage lower than the device's percentage, this candidate is discarded and the next lowest scoring candidate is considered. This continues until a candidate upgrade path with a phased percentage greater than or equal to the device's percentage; this candidate is then selected as the winning upgrade path. If no candidate matches this criteria, the device is considered up-to-date.
A missing phase percentage means every device should update; it's equivalent to an image percentage of 100%. An image percentage of 0% means that the image has been effectively pull from release, and no devices will update to it.
To test this scenario, we need a staging server with a known end-point update on a particular channel with a non-100% phase percentage. We currently have available:
host https://phablet.stgraber.org
channel ubuntu-touch/test
device mako
You also need a way to override the device's default phase percentage calculation, which can be done with system-image-cli -p.
Performing the test
cd /etc/system-image
- Grab the staging server's archive master keys:
mv archive-master.tar.xz{,.aside}
mv archive-master.tar.xz.asc{,.aside}
wget https://phablet.stgraber.org/gpg/archive-master.tar.xz
wget https://phablet.stgraber.org/gpg/archive-master.tar.xz.asc`
Edit /etc/system-image/channel.ini (not client.ini)
Change [service]base to phablet.stgraber.org
Do a dry-run update with your device's natural phased percentage.
system-image-cli -c "ubuntu-touch/test" -d mako -b 0 --dry-run -v
- You will either see an upgrade path to image 1 (with the device's phased percentage included in the output), or the device will claim to be up-to-date
- Do a dry-run update with a forced phase percentage that is within image 1's phase:
system-image-cli -c "ubuntu-touch/test" -d mako -b 0 --dry-run -v -p 33
- You should see upgrade path to image 1 and a target phase of 33%
- Do a dry-run update with a forced phase percentage outside the image phase:
system-image-cli -c "ubuntu-touch/test" -d mako -b 0 --dry-run -v -p 66
- The device will claim to be up-to-date.
Try some other dry-run updates with various -p options. The image's phase is 40%
Be sure to restore your archive-master files and channel.ini file if you want to do other tests.
Additional tests candidates for automation
The following test cases are being automated. Work items are described in this blueprint.
Testing the image update path
- Test that updates from image n-1 work
- Test that updates from image n-2 work
Testing the image updater software
- Test that the updater software in image n can do an update to a single newer version in a synthetic channel
- Test that the updater software in image n can do a multi-step update (applying 5? updates in sequence) in a synthetic channel
- Test that the updater software in image n correctly applies a full-image update when it’s a smaller download than a series of stacked partial upgrades
Process/Merges/TestPlan/ubuntu-system-image (last edited 2016-03-04 20:58:34 by localhost)