ReplaceArchiveAdminShellAccess

Differences between revisions 27 and 28
Revision 27 as of 2012-05-16 13:24:06
Size: 10464
Editor: 82-69-40-219
Comment:
Revision 28 as of 2012-05-16 14:16:36
Size: 10470
Editor: 82-69-40-219
Comment: backport-source decommissioned
Deletions are marked like this. Additions are marked like this.
Line 55: Line 55:
 * Flushing uploads created by `backport-source` (use `backportpackage` instead?)  * --(Flushing uploads created by `backport-source` (use `backportpackage` instead?))--

Summary

The Ubuntu archive administration team has always required direct privileged shell access to the ftpmaster system in order to perform many of its routine tasks. This is a security problem, it prevents us from opening some tasks up to those who are not Canonical employees, and it makes it hard for us to improve our own tools. Improve the Launchpad API to handle all our requirements and write suitable API clients.

We will know we have succeeded when archive admins no longer require shell access to do their jobs.

Release Note

This is not a user-facing feature.

Rationale

Fixing this has been a long-term goal for many years. Specific problems caused by the current mechanism include:

  • The archive administration account has a horrendous amount of privilege. While we have not spent much time coming up with specific attack vectors, an Ubuntu archive administrator could almost certainly inject malicious code into Ubuntu bypassing any of the usual auditing mechanisms, could probably forge uploads by other developers, and so on; and they have direct access to the archive signing key (though not the master key used for rollovers). While all our archive admins are highly trustworthy individuals, this means that we are constrained in how we can expand the team because their integrity has to be absolutely unquestionable, and it means that a compromise of an archive admin's local machine could have disastrous consequences if the attacker realised what they'd stumbled on. This is much more privilege than we in fact need.
  • Some archive administration tasks cannot be delegated beyond Canonical staff. There are community members who would happily take some of this on if we'd let them, and who could be trusted to do so.
  • It is difficult for the archive team to improve their own tools, because they are part of the Launchpad source code. While the basic database operations clearly must be in Launchpad, we could be more efficient if we could more easily build our own user interfaces. The ftpmaster tools have particularly weak UI by comparison with the rest of Launchpad because the Launchpad developers rarely if ever use them outside the test suite.

The direction for improvement is in large part obvious: the Launchpad API should be extended so that all operations currently performed by archive admins using shell access to ftpmaster have been replaced by API clients. At that point our shell access can be removed.

In fact, a substantial part of this work has been done over the years, and particularly recently as part of the derived distributions project, along with some rsync modules that allowed us to move several reporting jobs to another machine. However, there is still a good deal to do, and it seems likely that it will only ever get done if we make a specific coordinated effort to do it. This will require substantial although shallow work on Launchpad, and continued work by the Ubuntu archive admins to migrate our scripts elsewhere.

Implementation

This is broken down by user. Completing everything in either of the first two categories would allow incremental reductions in archive admin privilege before the entire project is complete.

lp_publish user

This user owns the published archive itself. Removing our access to this user and its associated group would remove our direct access to the archive signing key and much of our ability to untraceably modify the archive.

  • Copying custom uploads to -updates

  • more-extra.override.* symlinks when creating new series

  • [jdstrand] taking publisher offline can be important at certain times, and is currently done by updating the crontab for the lp_publish user. Is this something that we should continue to support or just rely on a losa (or similar)?
    • [cjwatson] Specific examples would be useful. The ones I'm aware of are:
      • Copying custom uploads to -updates (sometimes; should be fixed properly per the above)
      • Opening new series (should no longer be necessary, and if it is we should fix it)
      • Avoiding having to wait an hour when in a rush at release times (should be much less of an issue now that the publisher runs every half an hour; otherwise, ask webops to do it)
      • Security regressions (IME turning off the publisher is no longer particularly useful there; also see the previous item)
      • Arcane situations that require us to do a careful publisher run (these should be fixed in other ways, e.g. adding a way for archive admins to mark a pocket dirty in the API)
    • [cjwatson] I'm inclined to say that in general once we've fixed a few bits of low-hanging fruit it will be tolerable to rely on webops for this, and a worthwhile tradeoff for the gain in datacentre security.
  • The archive key is owned by lp_archive, and group lp_publish

lp_queue user

This user is responsible for processing incoming uploads. Removing our access to this user would remove our ability to forge uploads by other developers.

  • Flushing uploads created by sync-source (mostly replaced by Archive.copyPackage; needs ~katie user unsuspending so that auto-sync client can use it, needs auto-sync client finishing, and possibly other odds and ends of API syncs)

  • Flushing uploads created by backport-source (use backportpackage instead?)

  • Write up webops procedures for upload queue inspection/reprocessing

lp_archive user

Everything else.

  • change-override

    • [jdstrand] must work with all pockets as the security team relies on this to fix up various mis-copies prior to USN publication
  • copy-package (API won't copy to PPAs yet unless the feature flag soyuz.copypackageppa.enabled is set; limitations on copying from PPAs)

    • [jdstrand] must support copying from an arbitrary PPA to an arbitrary pocket. Eg, the security often copies from ubuntu-mozilla-security or ubuntu-security-proposed to -proposed (this allows us to build packages without -updates enabled in the build, so they can later be copied to -security after going through verification)
    • [cjwatson] Archive.syncSource times out on large packages and Archive.copyPackage seems to have some mysterious trouble with copying from stable-proposed to stable-updates (this was confusion due to them landing in the queue; sru-release now prints a message about this as a hint)

  • lp-remove-package
  • queue
    • [jdstrand] while I have moved away from doing NEW reviews on cocoplum itself, I still use queue for the following:
      • to see the list of packages to process (while I prefer a cli view for this, it wouldn't be the worst if I had to use the web frontend)
        • [cjwatson] I intend to add a queue API sufficient to implement a similar CLI client-side, at the cost of a bit of extra download time. I know of no reason it shouldn't have a feature set equivalent to our current CLI tool.
      • to download packages which I then scp locally. While I could also use the web frontend here, I have scripting around this and it would be nice if the new scripts supported downloading locally as well. (The API may already support this; I went with the easiest thing I could find at the moment I wrote my scripts for this)
        • [cjwatson] Ditto.
      • changing overrides in one of the queues *before* deNEWing them. Eg, if I see a NEW package is in binary NEW because it simply added a new binary package, after reviewing the package I will use queue to adjust the overrides in the NEW queue before accepting it. This saves times for everyone involved.
        • [cjwatson] Ditto.
  • unembargo-package
    • [jdstrand] the security team uses the API currently for unembargoing packages (ie, don't have to be a member of ubuntu-archive to publish to the security pocket) via lp:ubuntu-qa-tools/security-tools/unembargo. However, sometimes large packages timeout and an archive admin must use unembargo-package. IMO this should continue to be supported for emergency situations.

      • [cjwatson] Emergency situations could be handled via webops, especially if we improve the timeout situation. This unembargo tool is using Archive.syncSource, which is a synchronous API and is a standard source of timeouts; if we can fix things so that it can use Archive.copyPackage instead (which may require some work on the Launchpad side, but should be quite feasible), then this should become much less of a problem.
      • [cjwatson] It might be a little more complicated than this, since unembargoing also requires re-uploading files to the unrestricted librarian.
  • point-release-snapshot (could be done on a mirror, especially if we only snapshotted dists/)
  • sync blacklist (should move to blacklist entries in LP proper)
  • new-source (fairly easy to do on top of mass-sync client once implemented)

  • override helpers: new-binary-debian-universe, new-remove-duplicates, kernel-overrides, firefox-overrides, security-kernel-overrides
  • copy-report (easy to do on top of copy-package API client, though should be done in a cron job on lillypilly using a bot LP account in the ubuntu-archive team; see also 616704)

  • publish-queue (easy to do on top of queue API client)

  • populate-archive
  • Fix NewReleaseCycleProcess documentation to say that new-series sanity checking should be done on a mirror

  • Stale files pile up in /srv/launchpad.net/ubuntu-archive/ubuntu-germinate/ from time to time, even for the current series, when seeds or flavours or architectures are removed. At the moment we notice this because the archive reports stop updating. These should be cleared out automatically.

BoF agenda and discussion

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.


CategorySpec

FoundationsTeam/ReplaceArchiveAdminShellAccess (last edited 2013-10-02 11:45:46 by 82-69-40-219)