BazaarImportsSpinoffs

BazaarImportsSpinoffs

Status

Introduction

The BazaarImports braindump touched many points that deserve their own spec. The BazaarImportSpinoffs is a meta-spec for those points.

Rationale

In order to make progress on the BazaarImports spec without losing the information that was captured during the braindump phase, potentioal specification topics were moved to this page.

This allows to tighten the focus of the BazaarImports spec to the immediately relevant issues.

Scope and Use Cases

Implementation Plan

Huge Buildbot Logs

Buildbot has a performance bug in handling big text logs. Combined with the high volume of logging produced by CSCVS it makes it impossible to import big repositories like OpenOffice or XOrg.

This performance problem can be fixed in two ways:

  • Do not accumulate logs in memory, but use file storage instead.
  • Truncate logs to keep only the last few thousand lines.

The latter solution would be simple to implement.

Although it discards information, that should not be a problem, since failures are generally diagnosed by examining the tail of the conversion log. More difficult conversion problems will be diagnosed by driving imports with a command line tool instead of Buildbot.

TODO: Can that be done in a way which would be acceptable to buildbot upstream?

Buildbot Load Balancing

Build does not provide load balancing functionality. Instead, each job is assigned statically to a slave. In the short term, one system will be used mainly for syncs so the import load will not interfere with syncs.

ImportD Job Migration

Updating master from mirror when migrating a job across slaves.

Certifying Imports by Annotation

Error Reporting by Email

A small percentage of syncs fail, because

  • upstream modified the repository in an destructive manner
  • CSCVS bugs

TODO: retrieve log data from the ImportDBuilder

TODO: send log data by email to a multiple recipients email address.

CVS module aliases

Manual handling of CVS module aliases.

Should at least do email notification when a CVS import is actually a module alias.

If possible, automate handling.

Transient import errors

Telling server outages from conversion failures.

Sync Scheduling

Sync scheduling, broken by slave bouncing.

Testing environment

Testing environment is messy.

Fire and Forget imports

Fire and Forget imports.

ImportD Exec Barrier

Hitting a wall (e.g. out of memory) in an import breaks the slave in a way that needs a manual restart.

Late Mirror Creation

Do not create archives on arch.ubuntu.com at import time, but only at initial sync time.

Nice Autotesting

Should schedule autotesting on spare cycles of importd slaves.

Data Preservation and Migration

Packages Affected

User Interface Requirements

Outstanding Issues

UDU BOF Agenda

UDU Pre-Work

CategoryUdu CategorySpec

UbuntuDownUnder/BOFs/BazaarImportsSpinoffs (last edited 2008-08-06 16:22:20 by localhost)