server-lucid-apt-mirror-ec2
Launchpad Entry: server-lucid-apt-mirror-ec2
Created: NickBarcet
Contributors: JamesTroup
Packages affected: -
Summary
Improve the way our mirrors works on EC2. This is mainly an IS job.
Release Note
Package mirrors on EC2 have been improve to provide more robustness and better service.
Rationale
The current mirrors have the following problems:
- We assume that the mirror are not going to crash
- Long time to recover from a crash
- We need proper HA mirrors
- From AWS SLA, you can loose a complete availability zone without a problem but mirrors withtin a round robin need to be completely in sync
==> we currently cannot guarantee this
User stories
Assumptions
Design
Identical to what is offered today, just more robust
Implementation
3 possible implementations to chose from:
- sync proxy type setup
- a proxy within each zone, mirror syncing from it
==> has to be tested need 6 boxes total instead of 4
- a proxy within each zone, mirror syncing from it
- LVM snapshots to make them sync transactionally
- Using squid (similar to 1) Peer squid in diffrent regions, upstream being archive.u.c
UI Changes
No UI
Code Changes
Migration
No user migration required
Test/Demo Plan
Unresolved issues
BoF agenda and discussion
How does it work?
- Can it work in S3 directly?
- Problem with symlinks
- never been looked at by IS
- unappropriate packages for a ppa
- would have been elegant to have a repo be directly in S3
- would need to improve many things
- need to add a methd to apt
- need tools to update the mirror
plans for supporting fuse S3? -> scott moser [I tried creating an apt mirror in S3 using an rsync to an s3fs file system (fuse based). It was taking days to do the initial copy so I abandoned it, but would be willing to provide steps to whoever is investigating this -- Eric Hammond]
=== Problems ====
- WE assume that the mirror are not going to crash
- Long time to recover from a crash
- We need proper HA mirrors
- From AWS SLA, you can loose a complete region without a problem but mirrors withtin a round robin need to be completely in sync
==> we cannot guarantee this
- [Correction: The EC2 SLA assumes that an entire *availability zones* in a region can fail, but not an entire region --Eric Hammond]
Improvements
- sync proxy type setup
- a proxy within each zone, mirror syncing from it
==> has to be tested need 6 boxes total instead of 4
- a proxy within each zone, mirror syncing from it
- LVM snapshots to make them sync transactionally
- Using squid (similar to 1) Peer squid in diffrent regions, upstream being archive.u.c
- Should we talk with amazon ? Simon/Nick to try establish communitcation
- Documentation ?? Not right now, but once it is in good shape, yes, it should
Would be usefull to have a recommandation for uec, but not basedon this work (not the same constraints -> not te same solution)
- Good candidate for a puppet reciepe
blueprints/server-lucid-apt-mirror-ec2 (last edited 2009-11-23 17:26:21 by mx)