MultipleComputersSynchronization

Differences between revisions 12 and 24 (spanning 12 versions)
Revision 12 as of 2007-03-15 08:39:48
Size: 7661
Editor: 102
Comment:
Revision 24 as of 2008-08-06 16:26:19
Size: 12251
Editor: localhost
Comment: converted to 1.6 markup
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
 * '''Packages affected''': unison, unison-gtk  * '''Packages affected''': unison, unison-gtk, gshare
Line 10: Line 10:
This spec discuss the inclusion and integration of a remote synchronization/storage/sharing solution in Edgy. The proposed solution is currently based on unison. This spec discuss the inclusion and integration of a remote synchronization/storage/sharing solution in Edgy. The proposed solution is currently based on unison,
''although suitability of other tools (may be one of the version control systems, i.e. [[http://www.selenic.com/mercurial/wiki/|Mercurial]]) should be considered.''
Line 15: Line 16:
Keeping tracks of a given file or synchronisation is often a real pain.
Also, there's no easy way to share a file across the internet if you don't have webserver and the knowledge of ftp/sftp.
Keeping tracks of a given file or synchronization is often a real pain.
Also, there's no easy way to share a file across the Internet if you don't have webserver and the knowledge of ftp/sftp.
Line 19: Line 20:

1. Emma will soon fly to a conference in America with her whole new laptop. She wants to be sure that she can access all the necessarg y files that are on her home computer.
 1. Joe wants to share a 6 MB file with Jack. Unfortunatly, his SMTP service will only allow mail messages that are less than 5 MB.
 1. Emma will soon fly to a conference in America with her whole new laptop. She wants to be sure that she can access all the necessary files that are on her home computer.
 1. Joe wants to share a 6 MB file with Jack. Unfortunately, his SMTP service will only allow mail messages that are less than 5 MB.
Line 23: Line 23:
 1. Sally has three computers, and wants a common home directory, including her firefox extensions, and mail setup.  1. Sally has three computers, and wants a common home directory, including her Firefox extensions, and mail setup.
Line 26: Line 26:
Line 34: Line 33:
 * It's preferable that the solution could be used with other distribution or desktop/window managers (like Xubuntu and Xfce).
Line 36: Line 36:
=== General ===
Line 40: Line 40:
=== File sharing ===
Line 41: Line 42:
A configuration dialog would allow a default set of sync targets would enable a user to easily configure home dir syncing of things like firefox bookmarks/extensions (whole profiles, extensions, without caches!!!). === File synchronization ===
From the Preferences Dialog the user coul
d allow a default set of sync targets. This will enable a user to easily configure home dir syncing of things like firefox bookmarks/extensions (whole profiles, extensions, without caches!!!).
Line 43: Line 45:
A configuration dialog could allow check box selection of what to sync between home folders (see above), and choose whether to sync video, music, or other large files. This could be bandwidth dependent, something which can be tested. From the Prefereces Dialog the user could check what to sync between home folders (see above), and choose whether to sync video, music, or other large files. This could be bandwidth dependent, something which can be tested.
Line 45: Line 47:
A wizard will allow a fast service setup enabling a default public share folder and default application synchronization (gaim, firefox, evolution) From the Preferences Dialog the user could check if he/she want all folder synchronized at log on or select some of them.

From the Preferences Dialog the user could manualy add new synchronization shares. This will allow Ubuntu Synchronization to work with other system.

A remote folder will be synchronized with the local folder having the same path.


=== User Friendly Unison ===
 * In the case of inconsistencies warning the shares should be brought to a consisent state without warning the user. Just follow the indication from unison warning.
 * If there are no conflicts, the default action should be performed and a progress bar displayed to the user.
 * Integration with gnome-keyring
 * In the case remote host does not have unison installed, a static version of unison should be transfered to the host into the .unison dir.
 * In the case the remote host has other version of unison and there is no unison database in the user folder, transfer our version to the system. Else warn the user about the conflict
Line 49: Line 63:
 * ["iFolder"] , an opensource solution from Novell would fit quite nicely.
 * [http://www.cis.upenn.edu/~bcpierce/unison/ Unison] a command line program that can keep multiple directories in sync. (even if there are changes on both) Works great, but would need a good nautilus extension (with a real gtk/gnome frontend) to really make it amazing.
 * [[iFolder]] , an open source solution from Novell would fit quite nicely.
 * [[http://www.cis.upenn.edu/~bcpierce/unison/|Unison]] a command line program that can keep multiple directories in sync. (even if there are changes on both) Works great, but would need a good nautilus extension (with a real gtk/gnome front end) to really make it amazing.
Line 57: Line 71:
 * based on openSLP [http://www.openslp.org/doc/html/IntroductionToSLP/index.html info]
 * the "fshare" Service Agent will broadcast it's shares
 * the "fshare" User Agent will gather broadcasts from other "fshares".
 * the "fshare" application will make the information gathered by User Agent available at a "fshare://" location in the file browser.
 * Location "fshare://blinky" will display all shares provided by node "blinky". Depending of the share type opening a share will redirect the file browser to a scp:// or ftp:// location.
 * based on avahi [[http://avahi.org/]]
 * the "fshare" will advertise it's shares and share types
 * the shares will be available in nautilus or other file browser at "network:///" location
Line 66: Line 78:
=== Code === === DNS-SD entries ===
* public share - anonymous ftp
 * type = _ftp._tcp
 * subtype = _anon._sub
 * txt keys: path=<path>

* private share - ssh and sftp-subsystem
 * type = _sftp-ssh._tcp
 * txt keys: u=<username>,path=<path>, sync=no

* sync share - ssh and sftp-subsystem
 * type = _sftp-ssh._tcp
 * txt keys: u=<username>,path=<path>, sync=yes,

=== Emblems used===

* public share - anonymous ftp
 * artwork needed
* private share - ssh and sftp-subsystem
 * artwork needed
* sync share - ssh and sftp-subsystem
 * artwork needed

=== Screenshots ===
Preferences Dialog

{{http://adi.roiban.ro/misc/Screenshot-FShare%20Preferences.png}}

http://adi.roiban.ro/misc/fshare.glade
Line 75: Line 115:
  * I use this program to keep my usb key in sync with a directory on my hard drive. This allows me to work on either and still sync the changes. The only minor issue is that if you change a non-text file on both directories you will have to decide which one to keep. It will not merge the differences. This program can keep complete backups if necessary and can merge more then two directories, in any order. Amazingly powerful it just needs a GUI and to be integrated into nautilus. -- PeterVanderKlippe [[DateTime(2006-04-24T13:23:53Z)]]
 * iFolder looks like a good product, but I think the enterprise mode server would be a bit much for Ubuntu's main purpose (though great for a server).  The workgroup mode seems to be more appropriate, but it is not yet complete, according to Novell.  It is also unclear how workgroup mode will work; to some extent it sounds like it will operate as extensions to chat clients and such.  Also, a solution that can operate behind a firewall (and NAT) would be ideal, because, in my experience, many people are do not know how to modify the settings on their firewall/router (or can not because of their ISP). -- MatthewIsaacs [[DateTime(2006-04-27T01:16:14Z)]]
 * Unison seems like a good product.  It currently works without the need for a central server, though it lacks a gui.  A linux/Ubuntu gui could be modeled after gui for the windows version.  The Unison FAQ points out some deficiencies that may arise in more complicated situations, such as sync loops and differences in timestamp granularity between different filesytems, and others.  I'm not sure if iFolder has some of these problems or not.  If Unison were used as a basis for the Ubuntu sync tool, some mechanism to handle this would probably have to be developed (though Unison does fail gracefully).  -- MatthewIsaacs [[DateTime(2006-04-27T01:21:29Z)]]
  * I use this program to keep my usb key in sync with a directory on my hard drive. This allows me to work on either and still sync the changes. The only minor issue is that if you change a non-text file on both directories you will have to decide which one to keep. It will not merge the differences. This program can keep complete backups if necessary and can merge more then two directories, in any order. Amazingly powerful it just needs a GUI and to be integrated into nautilus. -- PeterVanderKlippe <<DateTime(2006-04-24T13:23:53Z)>>

 * iFolder looks like a good product, but I think the enterprise mode server would be a bit much for Ubuntu's main purpose (though great for a server). The workgroup mode seems to be more appropriate, but it is not yet complete, according to Novell. It is also unclear how workgroup mode will work; to some extent it sounds like it will operate as extensions to chat clients and such. Also, a solution that can operate behind a firewall (and NAT) would be ideal, because, in my experience, many people are do not know how to modify the settings on their firewall/router (or can not because of their ISP). -- MatthewIsaacs <<DateTime(2006-04-27T01:16:14Z)>>
Update:
The 3.x open source versions of iFolder can operate in p2p mode using Avahi and has the advantage of being cross platform.

* Unison seems like a good product. It currently works without the need for a central server, though it lacks a gui. A linux/Ubuntu gui could be modeled after gui for the windows version. The Unison FAQ points out some deficiencies that may arise in more complicated situations, such as sync loops and differences in timestamp granularity between different filesytems, and others. I'm not sure if iFolder has some of these problems or not. If Unison were used as a basis for the Ubuntu sync tool, some mechanism to handle this would probably have to be developed (though Unison does fail gracefully). -- MatthewIsaacs <<DateTime(2006-04-27T01:21:29Z)>>
Line 81: Line 125:
 * Unison requires ssh login, and is relatively time consuming. iFolder syncs itself in the background with a saved login. Unison would need an unpriviledged account with only access to the sync folder on the remote machine. iFolder watches the filesystem for changes and can sync more cleverly. A Unison daemon could run which watches inotify.  * Unison requires ssh login, and is relatively time consuming. iFolder syncs itself in the background with a saved login. Unison would need an unprivileged account with only access to the sync folder on the remote machine. iFolder watches the file system for changes and can sync more cleverly. A Unison daemon could run which watches inotify.
Line 86: Line 130:

* Take a look at Conduit (http://www.conduit-project.org/). It does synchronization. Gshare handles file sharing already, http://yimports.com/~cpinto/projects/gnome/gshare

  * I found conduit to complex. It is a great project but i don't think it is suitable to be a default application in ubuntu.
   * We need to decide how many feature will be included in the synchronization tool. For now I think it is enough to have a simple documents and some application setting synchronization between computer. No gmail, pda or ipod synchronization.
   * Maybe we need another specification for advance device and web application synchronization

  * Gshare is great! It is very close to the specification and might be a good start.
   * Allows only one share folder
   * Only FTP is supported. Good for password-less public data, not enough for synchronization of private data.
   * Is very good that it uses avahi
   * instead of a builtin FTP server I prefer an anonymous only version of VSFTPD.
   * I don't know if we need mono for this application. I prefer plain C.

* Realistically, any app which will easily share files across a trusted local network is going to have security issues or is going to require a knowledge base above that of the average PC user. There are also plenty of 3rd party solutions to internet information sharing (IM transfers, email, rapidshare, dropbox (soon) etc etc). Why don't we concentrate on something robust for a local network - this is increasingly important with laptots like the Asus EEE pc, which bring multi-pc syncing issues to a new userbase.

* Having all shares at location "network:///" might be a problem in the case of many share available in the LAN. It will be difficult to locate them. Instead the usage of a location "fshare://blinky" will display all shares provided by node "blinky". Depending of the share type opening a share will redirect the file browser to a scp:// or ftp:// location.

* Emblems are a good way of taging files. Unfortunatly Nautilus has to many emblems shiped by default and no application make use of them. It would be great to manage all share/sync options using emblems. Like having special emblems for public share, private share, sync folder, sync at startup, master sync (don't delete files)

* Using Source Control Managemnt Tools could be useful for backup, but I dont' think we need to do synchronization with it. These tools are designed for text files. There is a reference of using such tools for simulating "time machine" on the Ubuntu GSoC2007 Page.
Line 88: Line 153:

= Ideas =

Just to add some ideas, stuff, here is [[http://deverne.free.fr/wik/doku.php/projet/personaldomains|a project]], I'm thinking about. It talks about the concept of personal domain, which may give some ideas about multiple computer synchronization.
----
CategoryUdu

Summary

This spec discuss the inclusion and integration of a remote synchronization/storage/sharing solution in Edgy. The proposed solution is currently based on unison, although suitability of other tools (may be one of the version control systems, i.e. Mercurial) should be considered.

Rationale

Having a multiple computers configuration is more and more common everyday : one desktop at work, one desktop at home, one laptop. Keeping tracks of a given file or synchronization is often a real pain. Also, there's no easy way to share a file across the Internet if you don't have webserver and the knowledge of ftp/sftp.

Use cases

  1. Emma will soon fly to a conference in America with her whole new laptop. She wants to be sure that she can access all the necessary files that are on her home computer.
  2. Joe wants to share a 6 MB file with Jack. Unfortunately, his SMTP service will only allow mail messages that are less than 5 MB.
  3. William can never remember if a given file is on his laptop or desktop. Sometimes the file is on the desktop AND on the laptop, but he only wants the newest version.
  4. Sally has three computers, and wants a common home directory, including her Firefox extensions, and mail setup.

Scope

  • Decentralized model
  • 2 types of share public(no password)/private
    • A strong cryptography and security model - like SSH - for confidential data
    • A weak security model - read only - like anonymous ftp - for use case 2
  • Integration with the system file browser
  • On-demand file sharing and file synchronization
  • Display a list of available shares, their type and available actions
  • It's preferable that the solution could be used with other distribution or desktop/window managers (like Xubuntu and Xfce).

Design

General

A special location in the file browser will show all nodes from LAN having files for sharing. Selecting a node will show all it's shares and right clicking a share will display a context menu with available actions (open, sync to, sync from)

Extended attributes or nautilus emblems should be used to mark folders for syncing/not syncing, public or private share, instead of the clumsy and incomprehensible Unison configuration files.

File sharing

File synchronization

From the Preferences Dialog the user could allow a default set of sync targets. This will enable a user to easily configure home dir syncing of things like firefox bookmarks/extensions (whole profiles, extensions, without caches!!!).

From the Prefereces Dialog the user could check what to sync between home folders (see above), and choose whether to sync video, music, or other large files. This could be bandwidth dependent, something which can be tested.

From the Preferences Dialog the user could check if he/she want all folder synchronized at log on or select some of them.

From the Preferences Dialog the user could manualy add new synchronization shares. This will allow Ubuntu Synchronization to work with other system.

A remote folder will be synchronized with the local folder having the same path.

User Friendly Unison

  • In the case of inconsistencies warning the shares should be brought to a consisent state without warning the user. Just follow the indication from unison warning.
  • If there are no conflicts, the default action should be performed and a progress bar displayed to the user.
  • Integration with gnome-keyring
  • In the case remote host does not have unison installed, a static version of unison should be transfered to the host into the .unison dir.
  • In the case the remote host has other version of unison and there is no unison database in the user folder, transfer our version to the system. Else warn the user about the conflict

Implementation

Existing solutions

  • iFolder , an open source solution from Novell would fit quite nicely.

  • Unison a command line program that can keep multiple directories in sync. (even if there are changes on both) Works great, but would need a good nautilus extension (with a real gtk/gnome front end) to really make it amazing.

iFolder has disadvantages compared to Unison: it is less conservative in wiping data instead of adding new files. However, it is supported albeit semi-commercial, whereas Unison appears to be stagnant.

iFolder had a great disadvantages compared to Unison: you need the server part! In many use cases Ubuntu Linux is used just on the client side and the server side is some other GNU/Linux based distribution. Having a static build of Unison (for Linux, FreeBSD, Solaris) we will only need a valid ssh account on another computer.

Architecture

  • based on avahi http://avahi.org/

  • the "fshare" will advertise it's shares and share types
  • the shares will be available in nautilus or other file browser at "network:///" location
  • public file sharing is using ftp
  • private file sharing is using scp
  • File synchronization works on top of private file using unison

DNS-SD entries

* public share - anonymous ftp

  • type = _ftp._tcp
  • subtype = _anon._sub
  • txt keys: path=<path>

* private share - ssh and sftp-subsystem

  • type = _sftp-ssh._tcp
  • txt keys: u=<username>,path=<path>, sync=no

* sync share - ssh and sftp-subsystem

  • type = _sftp-ssh._tcp
  • txt keys: u=<username>,path=<path>, sync=yes,

=== Emblems used===

* public share - anonymous ftp

  • artwork needed

* private share - ssh and sftp-subsystem

  • artwork needed

* sync share - ssh and sftp-subsystem

  • artwork needed

Screenshots

Preferences Dialog

http://adi.roiban.ro/misc/Screenshot-FShare Preferences.png

http://adi.roiban.ro/misc/fshare.glade

Data preservation and migration

Outstanding issues

BoF agenda and discussion

  • Added unison as existing solution
    • I use this program to keep my usb key in sync with a directory on my hard drive. This allows me to work on either and still sync the changes. The only minor issue is that if you change a non-text file on both directories you will have to decide which one to keep. It will not merge the differences. This program can keep complete backups if necessary and can merge more then two directories, in any order. Amazingly powerful it just needs a GUI and to be integrated into nautilus. -- PeterVanderKlippe 2006-04-24 13:23:53

  • iFolder looks like a good product, but I think the enterprise mode server would be a bit much for Ubuntu's main purpose (though great for a server). The workgroup mode seems to be more appropriate, but it is not yet complete, according to Novell. It is also unclear how workgroup mode will work; to some extent it sounds like it will operate as extensions to chat clients and such. Also, a solution that can operate behind a firewall (and NAT) would be ideal, because, in my experience, many people are do not know how to modify the settings on their firewall/router (or can not because of their ISP). -- MatthewIsaacs 2006-04-27 01:16:14

Update: The 3.x open source versions of iFolder can operate in p2p mode using Avahi and has the advantage of being cross platform.

  • Unison seems like a good product. It currently works without the need for a central server, though it lacks a gui. A linux/Ubuntu gui could be modeled after gui for the windows version. The Unison FAQ points out some deficiencies that may arise in more complicated situations, such as sync loops and differences in timestamp granularity between different filesytems, and others. I'm not sure if iFolder has some of these problems or not. If Unison were used as a basis for the Ubuntu sync tool, some mechanism to handle this would probably have to be developed (though Unison does fail gracefully). -- MatthewIsaacs 2006-04-27 01:21:29

  • Unison has a GUI, but not a great one. Unison is more ambivalent about filesystems than iFolder (which just takes charge in sometimes destructive ways).
  • Unison requires ssh login, and is relatively time consuming. iFolder syncs itself in the background with a saved login. Unison would need an unprivileged account with only access to the sync folder on the remote machine. iFolder watches the file system for changes and can sync more cleverly. A Unison daemon could run which watches inotify.
  • A lightweight http server could be use for file sharing instead of ftp allowing the use of webdav enabled applications http://www.webdav.org

  • A further use case requiring something similar to iFolder enterprise mode would be: At work Anna has two computer (one laptop, one desktop) and a ssh account on the company's central server running a Unix-Like OS.The company sysadmin will not install any other software on the server. She wants to have it's application settings and documents synchronized (the newest version on each computer).

* Take a look at Conduit (http://www.conduit-project.org/). It does synchronization. Gshare handles file sharing already, http://yimports.com/~cpinto/projects/gnome/gshare

  • I found conduit to complex. It is a great project but i don't think it is suitable to be a default application in ubuntu.
    • We need to decide how many feature will be included in the synchronization tool. For now I think it is enough to have a simple documents and some application setting synchronization between computer. No gmail, pda or ipod synchronization.
    • Maybe we need another specification for advance device and web application synchronization
  • Gshare is great! It is very close to the specification and might be a good start.
    • Allows only one share folder
    • Only FTP is supported. Good for password-less public data, not enough for synchronization of private data.
    • Is very good that it uses avahi
    • instead of a builtin FTP server I prefer an anonymous only version of VSFTPD.
    • I don't know if we need mono for this application. I prefer plain C.

* Realistically, any app which will easily share files across a trusted local network is going to have security issues or is going to require a knowledge base above that of the average PC user. There are also plenty of 3rd party solutions to internet information sharing (IM transfers, email, rapidshare, dropbox (soon) etc etc). Why don't we concentrate on something robust for a local network - this is increasingly important with laptots like the Asus EEE pc, which bring multi-pc syncing issues to a new userbase.

* Having all shares at location "network:///" might be a problem in the case of many share available in the LAN. It will be difficult to locate them. Instead the usage of a location "fshare://blinky" will display all shares provided by node "blinky". Depending of the share type opening a share will redirect the file browser to a scp:// or ftp:// location.

* Emblems are a good way of taging files. Unfortunatly Nautilus has to many emblems shiped by default and no application make use of them. It would be great to manage all share/sync options using emblems. Like having special emblems for public share, private share, sync folder, sync at startup, master sync (don't delete files)

* Using Source Control Managemnt Tools could be useful for backup, but I dont' think we need to do synchronization with it. These tools are designed for text files. There is a reference of using such tools for simulating "time machine" on the Ubuntu GSoC2007 Page.


CategorySpec

Ideas

Just to add some ideas, stuff, here is a project, I'm thinking about. It talks about the concept of personal domain, which may give some ideas about multiple computer synchronization.


CategoryUdu

MultipleComputersSynchronization (last edited 2008-08-06 16:26:19 by localhost)