TechnicalDetails

Technical Details

Storage

This provides remotely accessible disk space stored on Amazon S3. Rather than using an existing protocol (eg. FTP, SFTP, NFS, Webdav) access to this remote disk space is via an invented-here communication protocol called "ubuntuone-storageprotocol" based on Google Protocol Buffers.

The system is intended to be for mirroring of content between computers, opportunistically synchronizing those mirrors when the user is online. There is a delay because a file must be completely uploaded before the changes can start to be shared. The heavy duty backend storage is ultimately out-sourced to Amazon S3 (Amazon Simple Storage Service).

At a high-level, Storage has been compared to Dropbox. In June 2009, Mandriva have produced something similar called mdvbackup, (branded as "Mandriva Click'n'Backup").

During early development, the codename for this facility was "Hammertime".

General concepts

  • There is a backing database which stores file objects, directory objects, and the BLOB objects (files) themselves. Drawing a comparison to a filesystem, the inodes are stored in a PostgreSQL database and the extents are stored in S3.

  • On the user's computer, a Share is a directory that is actively being mirrored.

  • When a file is modified, the complete new version is uploaded and saved in a BLOB. The file object is updated to point at the new BLOB (directories are not themselves changed for a simple overwrite operation).

  • A Subtree is a particular directory object, representing itself and all of its descendants.

  • A Subtree can be shared with other users. This is called a Share.

  • The View given to another user can be either read-only, or read-write.

  • On connection, or reconnection, the u1sync-agent is responsible for spidering the full directory tree and identifying any changes on its own.

  • While connected, the ubuntuone-syncdaemon will receive unsolicited notifications of changed objects. Notifications are an optimisation. They allow for lower latency in synchronisation and they avoid unnecessary polling.

  • u1sync is an example of a simple one-shot client that ignores unsolicited notifications.

Authentication

An OAuth token is used for authentication between the desktop and server side infrastructures. This token is stored in the platform's keyring (for Ubuntu it is gnome-keyring). Actual login is performed via Ubuntu SSO client which requests the OAuth tokens from https://login.ubuntu.com. For third-party web sites (e.g. https://u1.to) the login is done entirely within a web-browser, so that any password/user ID is kept within the website and not exported. There are no username/passwords stored by ubuntuone-syncdaemon.

ubuntuone-syncdaemon calls com.ubuntuone.CredentialsManagement DBus methods which in turn calls the relevant ubuntu-sso-client methods to retrieve the credentials. In case the credentials don't exist an Ubuntu SSO login screen appears prompting the user to provide the credentials to Ubuntu SSO or register a new Ubuntu SSO (and Ubuntu One) user. For third-party web sites an OpenID web login is used.

updown

This is a backend librarian service. It listens for connections via HTTP and accepts the upload, or returns the download requested. It does not have a user-visible interface. This process also serves the public files.

It does not use the storageprotocol for access, talking instead to the backend database and is therefore effectively a raw HTTP<->S3 gateway. Additionally it implements part of the REST API that is specific to content upload and download.

At the moment updown instances run in Amazon EC2 and are accessible via files.one.ubuntu.com.

web

Also known as "web ui". This is a pretty HTTP/web-browser interface to view directory listings and files held in the user's storage service (and other Shares which they have access to). It is universal and cross-platform and allows viewing of files, uploading, downloading and publshing. The web interface also provides the access to user's contacts stored in Ubuntu One.

The user's own files in ~/Ubuntu One directory are displayed at https://one.ubuntu.com/files. The user-designated folders and incoming Shares are under "My synced folders" header.

ubuntuone-storageprotocol

This protocol is built on top of Google Protocol Buffers (a sort of fast binary XMLish schema language). This abstraction is to allow future extension of the ubuntuone-storageprotocol without breaking compatibility.

The intention of the protocol is to enabling synchronisation by mirroring of content. The protocol itself uses UUIDs as identifiers for directory objects and their contents (files). Requests are always initiated from the client end.

The .proto descriptions (AGPLv3+assignment) can be used as starting points for other language implementations.

API slaves

This is the term for a farm of twisted daemons running in EC2. They accepts incoming SSL/TLS wrapped ubuntuone-storageprotocol connections on port 443. Note that although port 443 (HTTPS) is used to allow passage through firewalls, the protocol has nothing to do with HTTP.

API slaves send a client connected via the ubuntuone-storageprotocol unsolicited updates of any reference that have been updated, so that the other end can re-download the changed objects.

You can connect to the instance with:

$ telnet-ssl -z ssl fs-1.one.ubuntu.com 443
...
3 ubuntu one storage server revision 5383.

Please note that while these are called API slaves, they have nothing to do with REST API introduced later.

ubuntuone-syncdaemon

This is a per-user daemon run on each desktop computer and has no user interface. It runs in the background and exposes a DBus API interface. The agent does the actual work of deciding what to synchronise in which direction and handles doing so.

By default, one folder called ~/Ubuntu One/ is automatically created and configured for mirroring. Changes to this folder (and any others added) are watched using inotify. Synchronisation meta-data about which directories are being mirrored is stored in ~/.cache/ubuntuone. When remote content has changed, the agent acts on the incoming unsolicited notification sent by api slave and starts downloading.

Authentication is via OAuth.

The implementation, written in Python (GPLv3), may also be referred to as chicharra. It is started by /usr/share/dbus-1/services/com.ubuntuone.SyncDaemon.service and exposes its DBus interface under com.ubuntuone.SyncDaemon. Logs files are kept in ~/.cache/ubuntuone/log/syncdaemon.log.

u1sdtool

Command line utility to control the behaviour of a running ubuntuone-syncdaemon instance. The tool does not perform any actions directly, but sends messages to the running agent.

libnautilus-ubuntuone

An extension (plugin) for the default GNOME file-browser called Nautilus to adds a status icon/button. This can send connection and disconnection requests to a running ubuntuone-syncdaemon instance. Additional directories (Shares) can be nominated for sharing using a new right-click context menu.

At the moment there is one implementation, written in C and packaged in ubuntuone-client-gnome.

u1sync

This is a command line and test utility tool more akin to bzr branch/push or git clone/push (version control systems). Synchronisation metadata is stored in a hidden directory at the root folder of each share named $dir/.ubuntuone-sync. This utility does not appear in Natty.

Note: It is incompatible with the rest of the ubuntuone-syncdaemon infrastructure.

Name mappings

This document

Suite

Canonical name

Bugs

Source

updown

1

updown

ubuntuone-servers

Python, n/a

web

1

web ui

ubuntuone-servers

Python, n/a

api slave

2,3

storage

ubuntuone-servers

Python, n/a

ubuntuone-storage-protocol

2,3

ubuntuone-storage-protocol

ubuntone-storage-protocol

Python, AGPLv3 u-s-p/trunk

ubuntuone-syncdaemon

2

ubuntuone-syncdaemon

ubuntone-client

Python, GPLv3 u-c/trunk

u1sdtool

2

u1sdtool

ubuntone-client

Python, GPLv3 u-c/trunk

libnautilus-ubuntuone

2

ubuntuone-client-gnome

ubuntone-client

C, GPLv3 u-c/trunk

u1sync

3

u1sync

ubuntone-client

Python, GPLv3 u-c/trunk

  • Components denoted '(N)' and having the same number are each part of the same suite.

UbuntuOne/TechnicalDetails (last edited 2012-04-30 05:48:40 by rye)