PushPackageMirroring

Summary

The current mirror system requires mirrors (even the official country mirrors) to do an rsync with the main server at various intervals. Ideally the mirrors (especially the country mirrors) should be informed of changes to the main server immediately and told which packages to update.

Rationale

  • Country mirrors are always behind the main mirror, sometimes by a day or more.
  • If the mirror happens to rsync when the main mirror has an error (or dependency problems) then that error is present on the mirror until the next update, regardless of how quickly it is fixed on the main mirror (this happened a fair bit in the dapper release cycle).
  • People that aren't prepared to be behind (e.g. testers) may manually change to the main mirror.
  • Countries with a slow international connection cannot increase the frequency over a certain point because they risk one large update running into the next (this is a problem that the maintainer of NZ's mirror explained to me).
  • Security updates populate to the mirrors so slowly that a central mirror is usually used. Ideally the security mirror would be a backstop but the country mirror would be up-to-date enough to be the usual provider of security updates.

Use cases

  • Mary operates a country mirror. She wants to ensure that her mirror is absolutely up-to-date so that there is never a need for people in her country to use the main mirror;
  • Susan is a tester. She wants to make sure that she has all the most recent packages so that she can manage her bugs properly. She also wants to use her local mirror;
  • Tana is in a country where the mirror only updates once a day. He tries to reload his packages and gets an error in the package index. He tries again all day but the problem persists;
  • John operates a country mirror. He has immense bandwidth internally (at peering exchanges) but a slow international connection. If he sets his mirror to rsync any more frequently than every 5 hours, an update over 2GB will cause one rsync to overlap the next.

Scope

Design

Implementation

Code

Data preservation and migration

Outstanding issues

Someone else that is interested will need to pick up this idea and run it as I do not have the ability to design an effective solution.

BoF agenda and discussion

CategorySpec

PushPackageMirroring (last edited 2008-08-06 16:27:57 by localhost)