Client
Differences between revisions 46 and 51 (spanning 5 versions)
|
Size: 19455
Comment:
|
← Revision 51 as of 2016-10-31 14:09:03 ⇥
Size: 5930
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 2: | Line 2: |
| Line 3: | Line 4: |
| The client tool is used to calculate the update, download the files, validate them and then set the needed flags for the upgrader to pick them up and apply. | The [[http://packages.ubuntu.com/search?keywords=system-image&searchon=names&suite=all§ion=all|system-image client tool]] is used to calculate the update, download the files, validate them, and then perform the device- and distro-specific operation to apply the update. On ubuntu-touch, this means rebooting into recovery. On snappy, this means applying the update to the "B" partition. |
| Line 5: | Line 6: |
| == Requirements == | Also, on ubuntu-touch '''all''' downloads are performed by the `ubuntu-download-manager` service. On snappy, (and optionally on touch) downloads are performed by a built-in cURL based downloader. == Overview == |
| Line 7: | Line 11: |
| * Support for everything described in the [[../GPG|GPG]] spec (revocation list, multiple keyrings, ...) | * Support for everything described in the [[../GPG|GPG]] spec |
| Line 10: | Line 14: |
| * Flexible implementation to allow different upgrader setups with minimal changes required | * Hooks to provide customized "apply-the-upgrade" operations. |
| Line 13: | Line 17: |
| == Nice to have == * Bandwidth limiting is a nice to have and might be tricky to implement. |
== Implementation == |
| Line 16: | Line 19: |
| == Implementation == The current implementation is a command line tool, however it's expected to be turned into a DBus service that the Touch UI can drive. |
There source package is divided into three (+1 for a "development" package not normally deployed) binary packages: `system-image-common` for the base system and Python libraries; `system-image-cli` for the command line tool; `system-image-dbus` for the D-Bus service. Note that on snappy, only `system-image-cli` is used since there are no D-Bus clients (e.g. `system-settings` on touch). The cli tool also provides a much more robust functionality, allowing you to do things on the cli that you can't do over D-Bus. There is a default image server available at http://system-image.ubuntu.com but it is relatively easy to deploy your own servers. |
| Line 20: | Line 24: |
| Whenever called the client does the following: | The client does the following: |
| Line 23: | Line 29: |
| 1. Read the current version number of the device (ubuntu-build file) 1. Look for the most recent version available 1. Resolve an upgrade path to it, minimizing download size and number of reboots |
1. Read the current version number of the device 1. Look for the most recent version available for the specified channel on the current device 1. Calculate the candidate upgrade paths, minimizing download size, number of reboots, and other factors (including Phased Updates) 1. Score the candidate paths (lowest score wins), yielding the images to effect the upgrade |
| Line 29: | Line 36: |
| 1. Write the list of updates for the upgrader to use 1. Reboot into the upgrader |
1. Write the commands for upgrading the device to a file 1. Apply the update (in-place "B" partition unpacking for snappy, reboot into recovery for touch) |
| Line 38: | Line 45: |
| The client will export a DBus API on the system bus which will allow for a [[SoftwareUpdates#Phone|u/i in the System Settings]] to query, begin, cancel, and apply a system update. This service starts via DBus activation and exits automatically after a configurable amount of time (i.e. it does not run forever). It maintains state such that the client can exit and get restarted to continue the update, even across reboots. The client uses the [[../Downloader|download service]] to manage all file downloads. Here is the DBus API specification: === Signals === '''!UpdateAvailableStatus(is_available, downloading, available_version, update_size, last_update_date, descriptions, error_reason)'''<<BR>> Sent in response to '''!CheckForUpdate()''' async calls, this signal provides information about the state of the update. * ''is_available'' - boolean which indicates whether an update is available or not. This will be {{{false}}} if the device's build number is equal to or greater than any candidate build on the server (IOW, there is no candidate available). This flag will be {{{true}}} when there is an update available. * ''downloading'' - boolean indicating whether a download is in progress. This doesn't include any preliminary downloads needed to determine whether a candidate is available or not (e.g. keyrings, blacklists, {{{channels.json}}}, and {{{index.json}}} files). This flag will be {{{false}}} if a download is paused. * ''available_version'' - string specifying the update target candidate version. * ''update_size'' - integer providing total size in bytes for an available upgrade. This does not include any preliminary files needed to determine whether an update is available or not. * ''last_update_date'' - the ISO 8601 format UTC date (to the second) that the last update was applied to this device. This will be the empty string if no update has been previously applied. * ''descriptions'' - array of mappings containing the descriptions for all updates that will be downloaded. Each element of the array is a mapping of language-specific description keys to the UTF-8 description for that language. See [[../Server|the server documentation]] for details. * ''error_reason'' - A UTF-8 English human readable string indicating why the download did not start. Only useful if the second argument ('''downloading''') is false, otherwise ignore this value. Depending on the state of the system, as described below, some of the arguments of this signal may be ignored. '''!UpdateProgress(percentage, eta)'''<<BR>> Sent periodically, while a download is in progress. This signal is not sent when an upgrade is paused. * ''percentage'' - integer between 0 and 100 indicating how much of the download (not including preliminary files) have been currently downloaded. This may be 0 if we do not yet know what percentage has been downloaded. * ''eta'' - estimated time remaining to complete the download, in float seconds. This may be 0 if we don't have a reasonable estimate. '''!UpdatePaused(percentage)'''<<BR>> Sent whenever a download is paused as detected via the download service. * ''percentage'' - integer between 0 and 100 indicating how much of the download (not including preliminary files) have been currently downloaded. May be 0 if this information cannot be obtained. '''!UpdateDownloaded()'''<<BR>> Sent when the currently in progress update has been completely and successfully downloaded. When this signal is received, it means that the device is ready to have the update applied via '''!ApplyUpdate()'''. '''!UpdateFailed(consecutive_failure_count, last_reason)'''<<BR>> Sent when the update failed for any reason (including cancellation, but only if a download is in progress). The client will remain in the failure state until the next '''!CheckForUpdate()''' call. * ''consecutive_failure_count'' - an integer specifying the number of times in a row that a '''!CheckForUpdate()''' has resulted in an update failure. This increments until an update completes successfully (i.e. until the next '''!UpdateDownloaded()''' signal is issued). * ''last_reason'' - a UTF-8 string containing the English reason for why this updated failed. '''Rebooting(status)'''<<BR>> Sent just before the device reboots. * ''status'' - a boolean indicating whether the application of the update is successful or not. Generally, when ''status'' is {{{true}}} you won't ever receive the signal because the device will reboot. When ''status'' is {{{false}}} it means the application of the update or reboot failed for some reason. '''!SettingChanged(key, value)'''<<BR>> Sent when a setting is changed. This signal is ''not'' sent if the new value is the same as the old value. * ''key'' - the key of the value that was changed. * ''value'' - the new value for the key. === Methods === '''!CheckForUpdate()'''<<BR>> Asynchronous call instructing the client to check for an available update. If a check is already in progress, it continues. If the client is in ''auto-download'' mode (see below), then this call will both check the server for an available update, and automatically initiate the download of an available update. If the client is in ''manual-download'' mode, then the downloading of any available update will ''not'' occur automatically (use '''!DownloadUpdate()''' explicitly to start the download). It is also possible for the download to only occur automatically if other criteria are met, e.g. we are not on a wifi, or the device has a low battery. Note that '''!CheckForUpdate()''' will never resume a download. In all cases, an '''!UpdateAvailableStatus''' signal will be issued once an update candidate is determined to be available; the arguments can have the following values: * '''!UpdateAvailableStatus(true, true, build_number, size, "YYYY-MM-DDTHH:MM:SS", descriptions, "")''' - This means that an update is available and is currently downloading. The build number of the candidate update is given, as is its total size in bytes, and the descriptions of the updates in all available languages. * '''!UpdateAvailableStatus(true, false, build_number, size, "YYYY-MM-DDTHH:MM:SS", descriptions, "paused")''' - This means that an update is available, but it is not yet downloading, possibly because the client is in manual-update mode, or because the download is currently paused. The reason is given in the last argument, and the build number, size, and descriptions are given as above. * '''!UpdateAvailableStatus(false, ?, ?, ?, "YYYY-MM-DDTHH:MM:SS", ?, ?)''' - There is no update available. The ISO 8601 date of the last applied update is given, but all other arguments should be ignored. If we are in auto-download mode, an '''!UpdateProgress()''' signal is sent as soon as the download is started, usually with a {{{percentage}}} of 0 and an {{{eta}}} of 0. '''!DownloadUpdate()'''<<BR>> Asynchronous method to download an available update. No-ops if there is no update to download, an automatic download is already in progress, '''!CheckForUpdate()''' was not called first, or the update status is in an error condition. If a previous download was paused, this resumes the download. As with auto-download mode, an '''!UpdateProgress()''' signal is sent as soon as the download is started. '''!ApplyUpdate()'''<<BR>> Asynchronous method to apply the update and initiate the reboot. No-ops if no new update has been downloaded. Because the success path leads almost immediately to a reboot, this method does not return anything, but a '''Rebooting''' signal is sent. '''!CancelUpdate()'''<<BR>> Synchronous method to cancel any update check or download (even finished or paused) in progress. The empty string is returned unless an error occurred, in which case the error message is returned as a UTF-8 English string. '''!PauseDownload()'''<<BR>> Synchronous method to pause the current download. The empty string is returned unless an error occurred, in which case the error message is returned as a UTF-8 English string. '''Info()'''<<BR>> Get information about the system images, similar to what is available via {{{system-image-cli --info}}}. Returns the following pieces of information. * '''current build number''' - integer * '''device name''' - string * '''channel name''' - string * '''last update date''' - ISO 8601 UTC format string, with second resolution, e.g. '''"YYYY-MM-DDTHH:MM:SS"''' * '''version details''' - dictionary of strings-to-strings where the keys are component names and the values are version strings. '''Info2()'''<<BR>> Like '''Info()''', but returns extended information that includes the last time a '''!CheckForUpdate()''' was performed. Returns the following pieces of information. * '''current build number''' - integer * '''device name''' - string * '''channel name''' - string * '''last update date''' - ISO 8601 UTC format string, with second resolution, e.g. '''"YYYY-MM-DDTHH:MM:SS"''' * '''version details''' - dictionary of strings-to-strings where the keys are component names and the values are version strings. * '''last check date''' - ISO 8601 UTC format string, with second resolution, e.g. '''"YYYY-MM-DDTHH:MM:SS"''' '''!FactoryReset()'''<<BR>> Wipe the data partition and issue a reboot to recovery. Takes no arguments, but possibly sends a {{{Rebooting()}}} signal (depending on the race against the system reboot procedure). ''New in system-image 2.3.'' '''!SetSetting(key, value)'''<<BR>> Synchronous method to write or update a setting. {{{key}}} and {{{value}}} are UTF-8 strings. While any key/value pair may be set, only the following currently have useful semantics: * '''min_battery''' - The minimum battery strength which will allow downloads to proceed. The value is the string representation of a number between 0 and 100 percent. If the battery level is less than this, an available download will not be automatically initiated, although an explicit '''!DownloadUpdate()''' call can force this. * '''auto_download''' - A tri-state (currently) value indicating whether downloads should normally proceed automatically if an update is available or not. The value is the string representation of the following integer values. * 0 - Never download automatically (i.e. an explicit '''!DownloadUpdate()''' call is required to start the download) * 1 - Only auto-download if the device is connected via wifi (the default) * 2 - Always download the update automatically * '''failures_before_warning''' - Unused by the client, but stored here for use by the user interface. If the new value is different than the old value, or if this key was not previously set, a '''!SettingChanged(key, value)''' signal is sent. For the values with semantics, any invalid value is ignored (i.e. ''not'' set or stored). '''!GetSetting(key)'''<<BR>> Synchronous method to read a setting. If {{{key}}} is unknown, the empty string is returned. '''Exit()'''<<BR>> Synchronous method causing the client process to exit immediately. There is no return value. If this is not called, the client will still exit normally after some configurable amount of time. DBus activation will restart it. |
The `system-image-dbus` process exposes a D-Bus API on the System Bus which will is used by touch processes such as the [[SoftwareUpdates#Phone|u/i in the System Settings]] to query, begin, cancel, and apply a system update. This service starts via D-Bus activation and exits automatically after a configurable amount of time (i.e. it does not run forever). It maintains state such that the client can exit and get restarted to continue the update, even across reboots. The client uses the [[../Downloader|download service]] to manage all file downloads. See the [[http://manpages.ubuntu.com/manpages/xenial/man8/system-image-dbus.8.html|manpage]] for the up-to-date description of the D-Bus interface. |
| Line 144: | Line 48: |
| * https://code.launchpad.net/~ubuntu-system-image/ubuntu-system-image/client | |
| Line 146: | Line 49: |
| == Mock scenarios == | The code is now managed under Git on Launchpad: https://code.launchpad.net/ubuntu-system-image |
| Line 148: | Line 51: |
| === Normal update in auto mode === | The old Bazaar repository is preserved for read-only access. |
| Line 150: | Line 53: |
| option: --testing=update-auto-success 1. !CheckForUpdate() is received 2. wait for 3 seconds 3. !UpdateAvailableStatus(is_available: true, downloading: true, available_version: 42, update_size: 1337*1024*1024, last_update_date: 1983-09-13T12:13:14, descriptions: [{'description': "Ubuntu Edge support", 'description-en_GB': 'change the background colour', 'description-fr': "Support d'Ubuntu Edge"}, {'description': "Flipped container with 200% boot speed improvement"}], error_reason: "") is sent by the daemon 4. immediately the daemon sends !UpdateProgress(0, 50) 5. then every half a second, send !UpdateProgress(+1, -0.5) 6. daemon sends !UpdateDownloaded() 7. client sends !ApplyUpdate(), daemon gives back an empty string (success) |
== Phased Updates == |
| Line 159: | Line 55: |
| Note that while download is in progress, calling: * !PauseDownload will pause the emission of !UpdateProgress(). It's a noop otherwise (even if called when not download). * !DownloadUpdate will resume the emission of this signal. It's a noop otherwise (even if called when not download). * !CancelUpdate will cancel the update. It's a noop otherwise (even if called when not download). Next !CheckForUpdate will restart from 1. * !CheckForUpdate will immediately resends !UpdateSignal and !UpdateProgress (if a download is in progress). Then, it will eventually send !UpdateDownload if the update was already done, as per spec. |
It is possible on the server to ''phase'' an update, such that only a percentage of devices will receive the update. This can be used to smoke test an update in the field, even after extensive pre-publishing testing. The way this works is: |
| Line 165: | Line 57: |
| General: * !GetSetting("auto_download") returns "1" * !SetSetting("auto_download", ''int'') will set the value in memory and then !GetSetting() will return it * !GetSetting("foo") returns "" |
* Every terminal upgrade image (i.e. highest available on a device/channel) has a percentage assigned to it, as an integer in the set [0..100] * 0% means an image has been pulled, and no device should upgrade to it * 100% means the image is available to all devices. This is the default if the image has no explicit phase percentage assigned to it * Each device calculates its own "phase", also as an integer in the set [0..100). Inputs into this random number include the channel, target image number, the device's unique ''machine id'', and a random number * If the device's phase percentage is less than or equal to the target image's percentage, then the device is "phased in" and selects the upgrade leading to this image number * If the device's phase percentage is greater than the target image's percentage, then the device is "phased out" and will not upgrade to the target image number |
| Line 170: | Line 64: |
| === Normal update in manual mode === | When a device is phased out, the candidate upgrade path scoring algorithm simply discards the winning upgrade candidate and falls back to the next lowest scoring candidate path, until a phased-in upgrade is found. If no phased-in upgrade path is found, the device will not upgrade. |
| Line 172: | Line 66: |
| --testing=update-manual-success 1. !CheckForUpdate() is received 2. wait for 3 seconds 3. !UpdateAvailableStatus(is_available: true, downloading: false, available_version: 42, update_size: 1337*1024*1024, last_update_date: 83-09-13, descriptions: [{'description': "Ubuntu Edge support", 'description-en_GB': 'change the background colour', 'description-fr': "Support d'Ubuntu Edge"}, {'description': "Flipped container with 200% boot speed improvement"}], error_reason: "") is sent by the daemon 4. clients sends !DownloadUpdate 5. immediately the daemon sends !UpdateProgress(0, 50) 6. then every half a second, send !UpdateProgress(+1, -0.5) 7. daemon sends !UpdateDownloaded() 8. client sends !ApplyUpdate, daemon gives back an empty string (success) |
'''Note:''' the algorithm described above is for `system-image` 3.0 or better. The `system-image` 2.5 phase algorithm is slightly different: a few different inputs are used to calculate the device's phase, and if a winning upgrade path is phased-out, no fallback is performed; the device simply won't upgrade. |
| Line 182: | Line 68: |
| === Failing update === --testing=update-failed The daemon is already in failure mode 1. the client will send !CheckForUpdate 2. the daemon answers with !UpdateAvailableStatus(is_available: true, downloading: false, available_version: 42, update_size: 1337*1024*1024, last_update_date: 83-09-13, descriptions: [{'description': "Ubuntu Edge support", 'description-en_GB': 'change the background colour', 'description-fr': "Support d'Ubuntu Edge"}, {'description': "Flipped container with 200% boot speed improvement"}], error_reason: "You need some network for downloading.") 3. the daemon then send !UpdateFailed(9, "You need some network for downloading.") 4. the client will then send !CancelUpdate 5. the client will send !CheckForUpdate 6. the daemon answers with !UpdateAvailableStatus(is_available: true, downloading: false, available_version: 42, update_size: 1337*1024*1024, last_update_date: 83-09-13, descriptions: [{'description': "Ubuntu Edge support", 'description-en_GB': 'change the background colour', 'description-fr': "Support d'Ubuntu Edge"}, {'description': "Flipped container with 200% boot speed improvement"}], error_reason: "") === Failing apply === option: --testing=fail-apply the daemon is in !UpdateDownloaded mode 1. the client will send !CheckForUpdate 2. the daemon answers with !UpdateAvailableStatus(is_available: true, downloading: false, available_version: 42, update_size: 1337*1024*1024, last_update_date: 83-09-13, descriptions: [{'description': "Ubuntu Edge support", 'description-en_GB': 'change the background colour', 'description-fr': "Support d'Ubuntu Edge"}, {'description': "Flipped container with 200% boot speed improvement"}], error_reason: "") 3. then the daemon sends the !UpdateDownloaded signal 4. client sends !ApplyUpdate and receive "Not enough battery, you need to plug your phone." string === Failing Resume / Manual download === --testing=fail-resume the daemon is paused, 42% of download is done 1. the client will send !CheckForUpdate 2. the daemon answers with !UpdateAvailableStatus(is_available: true, downloading: false, available_version: 42, update_size: 1337*1024*1024, last_update_date: 83-09-13, descriptions: [{'description': "Ubuntu Edge support", 'description-en_GB': 'change the background colour', 'description-fr': "Support d'Ubuntu Edge"}, {'description': "Flipped container with 200% boot speed improvement"}], error_reason: "") 3. the daemons sends the !UpdatePaused(42) signal 4. client tries to call !DownloadUpdate() and receive an !UpdateFailed(9, "You need some network for downloading.") === Failing pause === --testing=fail-pause the daemon is downloading, initially, 10% is downloaded, there is no known ETA. 1. the client will send !CheckForUpdate 2. the daemon answers with !UpdateAvailableStatus(is_available: true, downloading: true, available_version: 42, update_size: 1337*1024*1024, last_update_date: '1983-09-13T12:13:14', descriptions: [{'description': "Ubuntu Edge support", 'description-en_GB': 'change the background colour', 'description-fr': "Support d'Ubuntu Edge"}, {'description': "Flipped container with 200% boot speed improvement"}], error_reason: "") 3. immediately the daemon sends !UpdateProgress(10, 0) (note! 0 for no ETA) 4. the clients then sends !PauseDownload() and will receive "no no, not now" as a string from the daemon. === No Update === --testing=no-update Nothing is downloading and no check is in progress. 1. client sends !CheckForUpdate 2. daemon answers with !UpdateAvailableStatus(is_available: false, downlading: false, ? ?, last_update_date: '1983-09-13T12:13:14', ", ") |
Also note that in all cases, `system-image-cli` allows the user to override the device's phase percentage, thus forcing the winning path to be explicitly phased in or out. Dry-run operation of `system-image-cli` can be used to see what the scores and phase percentages are without effecting an upgrade. |
Client side
Introduction
The system-image client tool is used to calculate the update, download the files, validate them, and then perform the device- and distro-specific operation to apply the update. On ubuntu-touch, this means rebooting into recovery. On snappy, this means applying the update to the "B" partition.
Also, on ubuntu-touch all downloads are performed by the ubuntu-download-manager service. On snappy, (and optionally on touch) downloads are performed by a built-in cURL based downloader.
Overview
- Secure download of the indexes (HTTPS + GPG)
Support for everything described in the GPG spec
- Resolution of the best upgrade path based on different policies (total download size, least number of reboots, ...)
- Download and validation of the files
- Hooks to provide customized "apply-the-upgrade" operations.
- Support for suspend/resume of downloads
Implementation
There source package is divided into three (+1 for a "development" package not normally deployed) binary packages: system-image-common for the base system and Python libraries; system-image-cli for the command line tool; system-image-dbus for the D-Bus service. Note that on snappy, only system-image-cli is used since there are no D-Bus clients (e.g. system-settings on touch). The cli tool also provides a much more robust functionality, allowing you to do things on the cli that you can't do over D-Bus.
There is a default image server available at http://system-image.ubuntu.com but it is relatively easy to deploy your own servers.
Step-by-step example for an update
The client does the following:
Grab https://server/channels.json and lookup the index for the current channel. If present, also grab the device GPG keyring.
Grab https://server/<channel>/<model>/index.json
- Read the current version number of the device
- Look for the most recent version available for the specified channel on the current device
- Calculate the candidate upgrade paths, minimizing download size, number of reboots, and other factors (including Phased Updates)
- Score the candidate paths (lowest score wins), yielding the images to effect the upgrade
- Download any file needed up until the next reboot
- Validate all the files
- Write them to the cache partition
- Write the commands for upgrading the device to a file
- Apply the update (in-place "B" partition unpacking for snappy, reboot into recovery for touch)
Those steps don't include all of the specific GPG validation bits required to ensure the authenticity of all files. Those are detailed in the separate GPG wiki page.
Security (e.g. what to download over https/http) is outlined in the server security section.
DBus API
The system-image-dbus process exposes a D-Bus API on the System Bus which will is used by touch processes such as the u/i in the System Settings to query, begin, cancel, and apply a system update. This service starts via D-Bus activation and exits automatically after a configurable amount of time (i.e. it does not run forever). It maintains state such that the client can exit and get restarted to continue the update, even across reboots. The client uses the download service to manage all file downloads. See the manpage for the up-to-date description of the D-Bus interface.
Code
The code is now managed under Git on Launchpad: https://code.launchpad.net/ubuntu-system-image
The old Bazaar repository is preserved for read-only access.
Phased Updates
It is possible on the server to phase an update, such that only a percentage of devices will receive the update. This can be used to smoke test an update in the field, even after extensive pre-publishing testing. The way this works is:
- Every terminal upgrade image (i.e. highest available on a device/channel) has a percentage assigned to it, as an integer in the set [0..100]
- 0% means an image has been pulled, and no device should upgrade to it
- 100% means the image is available to all devices. This is the default if the image has no explicit phase percentage assigned to it
Each device calculates its own "phase", also as an integer in the set [0..100). Inputs into this random number include the channel, target image number, the device's unique machine id, and a random number
- If the device's phase percentage is less than or equal to the target image's percentage, then the device is "phased in" and selects the upgrade leading to this image number
- If the device's phase percentage is greater than the target image's percentage, then the device is "phased out" and will not upgrade to the target image number
When a device is phased out, the candidate upgrade path scoring algorithm simply discards the winning upgrade candidate and falls back to the next lowest scoring candidate path, until a phased-in upgrade is found. If no phased-in upgrade path is found, the device will not upgrade.
Note: the algorithm described above is for system-image 3.0 or better. The system-image 2.5 phase algorithm is slightly different: a few different inputs are used to calculate the device's phase, and if a winning upgrade path is phased-out, no fallback is performed; the device simply won't upgrade.
Also note that in all cases, system-image-cli allows the user to override the device's phase percentage, thus forcing the winning path to be explicitly phased in or out. Dry-run operation of system-image-cli can be used to see what the scores and phase percentages are without effecting an upgrade.
ImageBasedUpgrades/Client (last edited 2016-10-31 14:09:03 by localhost)