OneConf

Launchpad Entry: desktop-maverick-oneconf
Created: 2010-05-18
Contributors: DidierRoche, MatthewPaulThomas
Packages affected: software-center, ubiquity

OneConf is a mechanism for recording software information in Ubuntu One, and synchronizing with other computers as needed. In Maverick, the list of installed software is stored. This may eventually expand to include some application settings and application state. Other tools like Stipple can provide more advanced settings/control.

Contents

WhiteBoard from UDS
1. Design sketches

Rationale

This will enable to solve several scenarios to allow a user to synchronize the list and the settings with other Ubuntu systems as needed:

Scenario A.

A user reinstalls Ubuntu and would like to easily reinstall some/all his/her applications from his old install. The user also wants to easily restore some/all of his settings from his old installation, from ubiquity, as a migration path.

Scenario B.

A user has two computers, a personal computer and a work computer. Because the types of programs he uses at home are different from the programs used at work, the user wants to save an independent list of applications and settings for each computer.

Scenario C.

A user has two computers. After having played for a month with one, he wants to get exactly the same list of applications from machine A (current machine) from machine B (host with additional/removed apps). This user doesn't want to bother with the details of which apps are installed on one and not on the other.

Scenario D.

A user has a desktop computer and a netbook. The user wants to install and remove some of his applications, looking at the difference between the two computers, but doesn't want to sync the complete list. Indeed, the two computers have different use cases, and he doesn't want to install his painting application (e.g., The Gimp) on his netbook as it won't be useful for him. A diff list will enable him to make choices about what to install. To avoid unneeded noise (excessive information about the default choices between his ubuntu-netbook and his ubuntu-desktop) (only if one default is removed is relevant to him) such as with the choices between maverick, firefox vs chromium, rhythmbox vs banshee.

(Out of scope for maverick: Scenario E.

A user is actively browsing the web and carrying on conversations on Empathy/Pidgin. The user wants to switch to using his netbook without interrupting his work flow. He would like to be able to simply book his netbook and continue browsing with the same recent history, open tabs. The user also wants to continue all previously-opened Empathy/Pidgin conversations. The user also wishes to keep some/all of his application settings in sync between both computers.)

Comparing software across computers

An updated design for the Ubuntu Software Center 5.0 layout is now in the USC specification.

In Ubuntu Software Center’s “File” menu should be a section with “Save Inventory…” and “Inventory on Ubuntu One…” items.

“Save Inventory…” should open a “Save Inventory” save dialog. A locally-saved inventory should include the package name, repository (only if non-official?), and package version (only if forced?) of every package installed on this computer.

The “Inventory on Ubuntu One…” item should open an “Inventory on Ubuntu One” ordinary window.

The window should have primary text: “With multiple Ubuntu computers, you can use Ubuntu One to compare the software installed on each.” And secondary text: “No-one else can see what you have installed.”

Next should be a well with your Ubuntu One status. Whenever you are not signed in, at its leading end should be the text “You are not signed in.”, and at its trailing end should be “Sign In…” and “Join Ubuntu One…” buttons, which launch the standard Ubuntu One interfaces for those tasks. Whenever you are signed in, at the well’s leading end should be text of the form “Signed in as foo@example.com.”, and at its trailing end an “Ubuntu One Settings…” button that launches the Ubuntu One settings window.

A checkbox for “Share this computer’s inventory” should be unchecked by default, while a checkbox for “Show inventories from other computers” should be checked by default. Both checkboxes should be insensitive whenever you are not signed in to Ubuntu One.

Status text should be displayed at the trailing end of the window, opposite the two checkboxes. The status text should use the secondary font size, and its baseline should line up with the baseline of the checkbox label (regardless of whether a spinner is present).

For “Share this computer’s inventory”, the status text should be:

If it is insensitive or unchecked, nothing.
Otherwise, if syncing is underway, a spinner and the text “Syncing…”.
Otherwise, if no sync has happened, “Waiting to sync”.
Otherwise, text of the form “Last synced 8:30 AM” or “Last synced yesterday 11:55 PM”.

For “Show inventories from other computers”, the status text should be:

If it is insensitive or no sync has yet completed, nothing.
Otherwise, if it is unchecked, text of the form “None registered” or “2 registered”.
Otherwise, if it is currently syncing, a spinner and the text “Syncing…”.
Otherwise, if it has not yet completed a sync, “Waiting to sync”.
Otherwise, text of the form “Last synced 8:30 AM” or “Last synced yesterday 11:55 PM”, unless this is already shown for the previous checkbox.

In the navigation pane, the Installed Software group should begin with one item for the computer you’re using. If “Show inventories from other computers” is checked, the group should also have one item for each of the other computers that are sharing their inventory on your Ubuntu One account. For each computer, the icon should represent that computer as much as possible, including a miniature of the background picture currently set on it. The label for the current computer should be “Installed Software”, while the label for the other computers should be their hostnames. Is there a better label we could use?

When another computer is selected in the navigation pane:

The status bar should have text of the form “318 items installed”.
The pathbutton should consist of one element, with the same name as that computer.
The search field should be available, to search the items in the inventory.
The rest of the main pane should show a software list view of the items installed on that computer, categorized into two branches: a branch with a label of the form “3 items that aren’t on this computer” that is expanded by default each USC session, and a branch of the form “218 other items” that is collapsed by default each USC session.
Inside the “{A} items that aren’t on this computer” branch, as usual in a “Get Software” list view, a selected item should have “More Info” and “Install” buttons when selected. The “Install” button should be insensitive if the item is not available in any of the software sources for the current computer. In addition, the branch as a whole should have an “Install All” button next to it.

Inside the “{B} other items” branch, there should be “More Info” buttons but not “Install” buttons.
Non-application items should be hidden by default with a disclosure bar.

Implementation: Being worked on by Didier Roche.

Future work for this feature:

Ask you if you want to add the source for an item where the source isn’t set up on this computer. Batch this for multiple sources with “Install All”.
Show items that are on this computer but not the other one.
- As said on email, this can be easily done as backend fully support that and we will already have multiple treeview on the same pane. So no technical difficulty raised, just need correct wording.
Queue items to be installed on another computer?
Publish lists of recommended items.

NEWS

Some testing available at: https://wiki.ubuntu.com/OneConf/Testing

Release Note

TODO

Storage

Storage is made into couchdb, which gives for free the ubuntu one synchronization.

We store there:

list of ppa, associated with the computer name
list of all packages, associated with the computer name, with a marker for:
- list installed applications and codecs not in computer default installation (metapackage)
- list metapackages

dbus service

A dbus service enables to ask for the previous lists, and to update them in desktopcouch. Another call will give the list of installed applications/codecs only.

List of installed applications

Based on apt-daemon to get that list.

We will store all package names in desktopcouch with a marker for automatic installed packages and one for application not in default.

Indeed, when showing the list to the user, we only want to get the list of applications the user actually SELECTED to install. We don't want to show packages automatically provided by the default Ubuntu install or dependencies of the packages that the user selects for installation. This way the user can restore his applications on a newer version of Ubuntu without getting dependencies that are possibly no longer needed. Furthermore, the ubuntu-desktop and ubuntu-netbook case on two computers (scenario B.) is addressed.

The algorithm in "Annexe: algorithm for detecting list of installed package" tries to tackle this issue by a stupid heuristic.

We also can add a marker to associate with the PPA list from which the app comes from.

Software center integration

The design will be created by mpt.

Some random ideas was: http://people.canonical.com/~didrocks/oneconf/sc.png

Adding some kind of historical view with "last added/last removed" will be great

Ubiquity

A user should be able to restore his list of applications using the Ubuntu Installer. This would allow the user to take care of getting all his critical applications reinstalled up front without requiring the usual post-installation trip to Synaptic a lot of are accustomed to :). A user should also be able to restore his list of applications using Software Center as well.

This should be optional and only if network connection is available.

The user should be free to install some or all of the applications from his list of saved applications. When appropriate, we can simply present the user with a list and he can check the packages he wants to install. This allows the user to opt out of reinstalling applications he are no longer interested in.

NOTE: This should be optional. Default is « install all »

Design will be provided by Michael Forrest

Missing packages from the heuristic (to add to whitelist)

Nothing known right now.

Appendix: An algorithm for detecting a list of installed packages

This script detect CLI and graphical apps, for apps not installed by default. It adds some whitelist as well for popular codec and things people usually install which aren't applications.

import apt
import re

cache = apt.Cache()

default_packages = set()
# these are false default as you have some alternatives deps that are taken
# into account like file-roller depends zip | p7zip-full
# -> p7zip-full won't be listed
false_defaults = set(['p7zip-full', 'vim-gnome', 'vim'])

# default whitelist, completed then by ubuntu-restricted-extras deps
whitelist = set(['flashplugin-nonfree', 'gnash',
'gstreamer0.10-fluendo-mpegdemux', 'swfdec-gnome',
'swfdec-mozilla', 'ubuntu-restricted-extras'])
blacklist_regexp = re.compile('.*-dev')

desktop_pkg_file_pattern = re.compile('/usr/share/applications/.*\.desktop')
bin_file_pattern = re.compile('/bin/.*')
sbin_file_pattern = re.compile('/sbin/.*')
usr_bin_file_pattern = re.compile('/usr/bin/.*')
usr_sbin_file_pattern = re.compile('/usr/sbin/.*')

def get_dep_rec_list(root_package, default_packages, recursive=True):
    '''Get list of dep of package_root'''

    for relations in (root_package.candidate.dependencies,
                      root_package.candidate.recommends):
        for dep in relations:
            for or_dep in dep.or_dependencies:
                if or_dep.name not in default_packages:
                    default_packages.add(or_dep.name)
                    try:
                        if recursive:
                            get_dep_rec_list(cache[or_dep.name],
default_packages)
                    except KeyError:
                        pass
get_dep_rec_list(cache['ubuntu-desktop'], default_packages)
get_dep_rec_list(cache['ubuntu-minimal'], default_packages)
get_dep_rec_list(cache['ubuntu-standard'], default_packages)
default_packages -= false_defaults

#add to the whitelist ubuntu-restricted-extras direct dep
get_dep_rec_list(cache['ubuntu-restricted-extras'], whitelist, recursive=False)

# get list of all apps installed
candidate_pkg = set()
for pkg in cache:
    if (pkg.is_installed and not pkg.is_auto_installed and not
        pkg.priority in ('required', 'important')):
        if pkg.name in whitelist:
            candidate_pkg.add(pkg.name)
        elif blacklist_regexp.match(pkg.name):
            continue
        else:
            for pkg_file in pkg.installed_files:
                if (desktop_pkg_file_pattern.match(pkg_file) or
                    bin_file_pattern.match(pkg_file) or
                    sbin_file_pattern.match(pkg_file) or
                    usr_bin_file_pattern.match(pkg_file) or
                    usr_sbin_file_pattern.match(pkg_file)):
                    candidate_pkg.add(pkg.name)
                    break

# archive those not in default set
pkg_to_archive = set()
pkg_to_archive = candidate_pkg - default_packages

#debug
pkg_name_list = [pkg_name for pkg_name in pkg_to_archive]
#for pkg_name in default_packages:
#    if pkg_name == 'p7zip-full':
#        print "grrrr"
for name in pkg_name_list:
    print name

For maverick +1/2

2. Application Settings

Which settings would be good to save/restore?
- Messaging (empathy)
- Gwibber (already done with ubuntu one?)
- E-mail (evolution) (contact already done, email setting?)
- chromium/firefox favorites
Others suggestions:
- Compiz
- Gedit
- Panel setup?
- Desktop background?
- Themes?
There are already programs that know how to save/restore stuff for specific programs... can we piggyback?
Must be careful not to save/restore settings that will get user into trouble, we shouldn't save password IMHO.
How to restore settings ? Silently ? Present a window?

3. Application State

It can be interested to set the state of some applications dynamically so that switching from an app to the other is possible.

Following a conversation in empathy can be complicated, but we can for tabs for instance store/follow open tabs in chromium/firefox.

Others ideas welcome, but state storing will be complicated to achieve. We should at least for maverick make a short list.

4. Initial Notes

Two scenarios:
- you reinstall your computer
- You have two computers (desktop and a netbook for instance)
You have to reinstall all additional apps: that's tedious. You can use dpkg --get-selections/--set-selections but end with hardware package selections that maybe doesn't fit on another computer. S-C should enable to only select "apps" which are addition to default layout.
Having a desktop and a netbook: how to switch from one to the other easily (browser history, opened tab, empathy conversation…). I want to have a smooth workflow even when migrating from my machine A to B.
I don't want as well having to configure things twice like microblogging account, mail client: importing from ubuntu one at first launch?
- store everything but passwords?
- select apps that will be handled by that
Both app selection and synchronization should be handle into ubuntu one as an additional step of ubiquity installer (if the Internet detected at install) of in an ubuntu one UI.

WhiteBoard from UDS

1. List of installed applications.
script :  https://wiki.ubuntu.com/OneConf
-> debhorphan ?
 - codecs / fonts…
 - all of those not in defaults!
proposed script:
 http://people.canonical.com/~didrocks/oneconf/getapplist.py
similar script for creating bundles (interesting because of the dependency calculation):
https://code.edge.launchpad.net/~mvo/+junk/apt_download_bundle

I think we have two use-cases here:
1. install new machine that "clones" another machine 
2. show apps recently installed apps on machine A on a machine B so that they can be installed there as well

* Issues with pushing/pulling automatically the list of installed applications:
1) Machine A: installed, synchronized with U1
2) Machine B: installed, get machine A list
3) if Machine A is installing a new application -> what to do on machine B (once synced)? Get a prompt ? if the user isn't sudoer?
3 bis): Machine A install gimp, machine B install gimp, and remove it, what to do? some kind of versioned installation ?
3 ter) Not sure that you want the same apps in your netbook and in your home computer

Solution? -> profiles
-> difficult to maintain, better to base on machine name (we have it in U1)
-> People pull from configuration, in software-center

* Integration in software-center
Wireframe: http://people.canonical.com/~didrocks/oneconf/sc.png
* Integration in ubiquity


How to backup the list of packages ?
-> integration to s-c? (can't be run as root as desktopcouch wants to store it)
-> it shouldn't depend on syncdaemon (locking issues, depends on file synchronization ?)
-> ubuntuone service (another desktop daemon)



2. Some application settings

- No gsettings <-> desktopcouch backend (hard to maintain, API breakage)
- pick some settings that we want to sync, make a poll on forum?
- Use hooks in dconf service for syncing with desktopcouch

Settings conflicts:
- resolved by the application (so, ubuntuone desktop daemon there):

Which settings would be good to save/restore?
 * chromium/firefox favorites(addons/extensions?) explore weave for fx
 * Messaging (empathy):
   - account configuration (stored in mission control)
   - if the password isn't present, currently no way to get the account information
     will be fix in next months
   - rye has the code for telepathy logging to couchdb. Basic one but works.
 * Gwibber (already done with ubuntu one?):
   - account (all in gnome-keyring, how to push it in ubuntu one, has to figure out with ken)
 * E-mail (evolution) (contacts already done, email settings?):
   - using backup evolution, but in incremental way:
   - but a lot of drawbacks, first step is to synchronize the settings
Others suggestions:
 * Compiz
 * Gedit
 * Panel setup?
 * Desktop background?
 * Themes?
There are already programs that know how to save/restore stuff for specific programs... can we piggyback?
Must be careful not to save/restore settings that will get the user into trouble. We shouldn't save the password IMHO.
How to restore settings ? Silently ? Present a window?

3. Some application/user state
Again, being picky about what we want to save.
It would be interesting to set the state of some applications dynamically so that switching from one app to another is possible.
Following a conversation in empathy can be complicated, but we can for tabs for instance store/follow open tabs in chromium/firefox.

Design sketches

OneConf (last edited 2012-06-19 22:33:15 by mailgate)

Ubuntu Wiki