Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

Summary

We would like Ubuntu to be able to ask for mirror recommendations and make efficient decisions as to which mirror to use - and to be able to do this after installation, as the mirror network expands and changes over time. Launchpad will hold the server-side data needed to inform any given desktop / server about its recommended mirrors, and systems will in turn be able to tell Launchpad what they experience when interacting with those mirrors. The goal is to make the whole mirror infrastructure more dynamic.

Rationale

In order to keep an Ubuntu system up to date it is necessary to periodically download updates from the Ubuntu archive. Currently, users are only presented with the option to use a limited number of the mirrors of the Ubuntu archive that are available on the Internet. Typically, the longest part of an update is downloading new packages from the archive. The mirrors that are currently presented to a user may not be the fastest mirror available. Providing users with a tool to find mirrors and determine their speed based on a user's network access might allow for faster package downloads.

The current system that is based on the country code is too inflexible and does not offer enough granularity. Some countries like the US are just too big to have only a single mirror, some others (like South Afrika) have a network topology that is so difficult that a single mirror for the country may mean very good results for some users and very bad results for others.

The other rationale is that we want to get user feedback in case of network problems automatically (preferably with additional information like a traceroute dump).

Use cases

Scope

The problems we face are:

This specification requires adding additional functionality to Software Properties and also ensuring that format of the mirror information in Launchpad is standardized. In the future we will extend this to provide feedback for us to report back problems with mirrors so that launchpad can dynamically update the mirror list.

Design

The user will need an application to retrieve the Ubuntu mirror list, test the speed of different mirrors, and update sources.list with the mirror that the user chooses.

Launchpad will have to ensure that mirror data is presented to clients in a standard way. Currently Launchpad offers RSS feeds of mirror information (such as https://launchpad.net/distros/ubuntu/+cdmirrors-rss). It is likely that we will need special RSS for this feature, though.

For automatic mirror selection we offer a option at install time. The user will be asked if he preferes automatic mirror selection (that involves some feedback reporting back to us) or manual selection which means to select the mirror manually based on the launchpad rss feed. The feedback will consist of reporting failures and the relative speed of the tested mirrors. It will also submit the IP adress so that we know what autonomous zone the request came from. The system will send a list of mirror sorted by what it things is the fastest based on the submited AS information.

To do meaningful speed measuring we need to download 64-128kb data per mirror to get around issues like slow start with tcp. We want to test three mirrors, the first two based on what we think is the fastest and the third one random. This is a certain hit for dialup users (64kb download takes ~16sec for a 56kb modem, multiplied by three is a long time just for a test). Testdata download has the disadvantage of wasting bandwidth so we should look into optimzing this as far as possible. A alternative approach would be to not use testdata and just download from the mirror that LP provided. Then record the download speed and calculate it relative to the fastest speed and report back to LP. This should give people a good average over time. The disadvantage is that sometimes really bad mirrors might be choosen (e.g. when insufficient data for a AS is available).

Implementation

An implementation of the spec has already been started by a member of the community.

For manual selection, we need to do:

For fully automatic selection we need:

For a version that can report feedback back to launchpad we need:

Given the scope of the automatic mirror selection it should probably be implemented in steps. Firstly the manual selection based on the current launchpad mirror list, then automatic selection, then failure reporting, then speed reporting.

Code

An implementation for software-properties is currently in development in the mirrors branch of Update Manager by BenjaminMontgomery that is able to download the launchpad list and offer mirror selection based on this list.

The scope for apt for feisty is:

The current code in http://people.ubuntu.com/~mvo/bzr/apt/apt--mirror/ implements:

If downloading the mirror information fails, the last downloaded mirror information is used.

Future work

The speed reporting is out of the scope for feisty.

Data migration

Once we have a working and well tested mirror method we can add code to the software-properties utility to make switching to automatic mirror selection easy. To do this we add a new entry to the "Download from:" combobox: "Automatically select mirror".

Unresolved issues

For feedback reporting the server side needs to be specified. This needs input from the server team.

The automatic mirror selection with the new mirror method makes consistency across mirrors more important than it was before because the checksums needs to match. As long as the selected mirror gives no problems that is not a issue. But when there is a 404 or a checksum-mismatch this may become a problem if e.g. the next selected mirror is not in sync. Than it may not have the requested package or the indexfile has no matching checksum. This is not a regression to the current behavior were we just error out.

One problem we have is that test-data may get cached (proxy/transparent proxy) and that makes our testdata bad. It needs to be investigated if we can detect this somehow.

Comments

CategorySpec

DynamicMirrorDecisions (last edited 2008-08-06 16:30:32 by localhost)