ReplaceArchiveAdminShellAccess

Differences between revisions 2 and 54 (spanning 52 versions)
Revision 2 as of 2011-12-31 17:43:18
Size: 5648
Editor: 82-69-40-219
Comment:
Revision 54 as of 2013-10-02 11:45:46
Size: 11422
Editor: 82-69-40-219
Comment: lillypilly -> snakefruit
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
 * '''Packages affected''': `ubuntu-archive-tools`  * '''Packages affected''': Launchpad, `ubuntu-archive-tools`
Line 22: Line 22:
 * The archive administration account has a horrendous amount of privilege. While we have not spent much time coming up with specific attack vectors, an Ubuntu archive administrator could almost certainly inject malicious code into Ubuntu bypassing any of the usual auditing mechanisms, could probably forge uploads by other developers, and so on; and they have direct access to the archive signing key (though not the master key used for rollovers). While all our archive admins are highly trustworthy individuals, this means that we are constrained in how we can expand the team because their integrity has to be absolutely unquestionable, and it means that a compromise of an archive admin's local machine could have disastrous consequences if the attacker realised what they'd stumbled on. This is much more privilege than we in fact need.  * The archive administration account has a horrendous amount of privilege. While we have not spent much time coming up with specific attack vectors, an Ubuntu archive administrator could almost certainly inject malicious code into Ubuntu bypassing any of the usual auditing mechanisms, --(could probably forge uploads by other developers)--, and so on; and they have direct access to the archive signing key (though not the master key used for rollovers). While all our archive admins are highly trustworthy individuals, this means that we are constrained in how we can expand the team because their integrity has to be absolutely unquestionable, and it means that a compromise of an archive admin's local machine could have disastrous consequences if the attacker realised what they'd stumbled on. This is much more privilege than we in fact need.
Line 38: Line 38:
 * Copying custom uploads to -updates
 * `more-extra.*` symlinks when creating new series
 * More NewReleaseCycleProcess stuff? (Probably mostly webops-able these days)
 * --([[Bug:231371|Copying custom uploads to -updates]])--
 * --([[Bug:827941|Copying DDTP tarballs to -updates]] (slightly different from previous bug, although dependent on it))--
 * --([[Bug:966092|more-extra.override.* symlinks when creating new series]])--
 * [jdstrand] taking publisher offline can be important at certain times, and is currently done by updating the crontab for the lp_publish user. Is this something that we should continue to support or just rely on a losa (or similar)?
  * [cjwatson] Specific examples would be useful. The ones I'm aware of are:
   * --(Copying custom uploads to -updates)--
   * --(Opening new series)--
   * Avoiding having to wait an hour when in a rush at release times (should be much less of an issue now that the publisher runs every half an hour; otherwise, ask webops to do it)
   * Security regressions (IME turning off the publisher is no longer particularly useful there; also see the previous item)
   * Arcane situations that require us to do a careful publisher run (these should be fixed in other ways, e.g. adding a way for archive admins to mark a pocket dirty in the API; however, the simple workaround for this is to do some trivial change-override operation in the relevant pocket, such as changing the section of a source publication)
  * [cjwatson] I'm inclined to say that in general once we've fixed a few bits of low-hanging fruit it will be tolerable to rely on webops for this, and a worthwhile tradeoff for the gain in datacentre security.
 * The archive key is owned by lp_archive, and group lp_publish
Line 44: Line 53:
'''Done!'''
Line 46: Line 57:
 * Flushing uploads created by `sync-source` (mostly replaced by `Archive.copyPackage`; needs [[https://answers.launchpad.net/launchpad/+question/183109|~katie user unsuspending]] so that `mass-sync` client can use it, needs `mass-sync` client finishing, and possibly [[https://bugs.launchpad.net/launchpad/+bugs?field.tag=derivation|other odds and ends of API syncs]])
 * Flushing uploads created by `backport-source` (use `backportpackage` instead?)
 * Write up webops procedures for upload queue inspection/reprocessing
 * --(Flushing uploads created by `sync-source` (mostly replaced by `Archive.copyPackage`; needs [[https://answers.launchpad.net/launchpad/+question/183109|~katie user unsuspending]] so that `auto-sync` client can use it, needs `auto-sync` client finishing, and possibly [[https://bugs.launchpad.net/launchpad/+bugs?field.tag=derivation|other odds and ends of API syncs]]))--
 * --(Flushing uploads created by `backport-source` (use `backportpackage` instead?))--
 * --([[ArchiveAdministration#Reprocessing_failed_source_uploads|Write up webops procedures for upload queue inspection/reprocessing]])--
Line 54: Line 65:
 * [[Bug:853831|change-override]]
 * copy-package
 * lp-remove-package
 * queue
 * unembargo-package?
 * point-release-snapshot (could be done on a mirror, especially if we only snapshotted dists/)
 * sync blacklist (should move to blacklist entries in LP proper)
 * new-source (fairly easy to do on top of mass-sync client once implemented)
 * override helpers: new-binary-debian-universe, new-remove-duplicates, kernel-overrides, firefox-overrides, security-kernel-overrides
 * copy-report (easy to do on top of copy-package API client, though should be done in a cron job on lillypilly using a bot LP account in the ubuntu-archive team)
 * publish-queue (easy to do on top of queue API client)
 * --([[Bug:853831|change-override]])--
  * --([jdstrand] must work with all pockets as the security team relies on this to fix up various mis-copies prior to USN publication)--
 * --(copy-package (API won't copy to PPAs yet unless the feature flag soyuz.copypackageppa.enabled is set; [[Bug:334858|limitations on copying from PPAs]]))--
  * --([jdstrand] should support copying from an arbitrary PPA to an arbitrary pocket. Eg, the security often copies from ubuntu-mozilla-security or ubuntu-security-proposed to -proposed (this allows us to build packages without -updates enabled in the build, so they can later be copied to -security after going through verification))--
   * --([jdstrand] workaround: use `unembargo --copy` from lp:ubuntu-qa-tools which does work with arbitrary PPAs (only needed for private-to-public copies))--
  * --([cjwatson] Archive.syncSource times out on large packages and Archive.copyPackage seems to have some mysterious trouble with copying from stable-proposed to stable-updates (this was confusion due to them landing in the queue; sru-release now prints a message about this as a hint))--
 * --(lp-remove-package)--
 * --([[Bug:1006173|queue]])--
  * --([jdstrand] while I have moved away from doing NEW reviews on cocoplum itself, I still use queue for the following:)--
   * --(to see the list of packages to process (while I prefer a cli view for this, it wouldn't be the worst if I had to use the web frontend))--
    * --([cjwatson] I intend to add a queue API sufficient to implement a similar CLI client-side, at the cost of a bit of extra download time. I know of no reason it shouldn't have a feature set equivalent to our current CLI tool.)--
   * --(to download packages which I then scp locally. While I could also use the web frontend here, I have scripting around this and it would be nice if the new scripts supported downloading locally as well. (The API may already support this; I went with the easiest thing I could find at the moment I wrote my scripts for this))--
    * --([cjwatson] Ditto.)--
   * --(changing overrides in one of the queues *before* deNEWing them. Eg, if I see a NEW package is in binary NEW because it simply added a new binary package, after reviewing the package I will use queue to adjust the overrides in the NEW queue before accepting it. This saves times for everyone involved.)--
    * --([cjwatson] Ditto.)--
 * --(unembargo-package)--
  * --([jdstrand] the security team uses the API currently for unembargoing packages (ie, don't have to be a member of ubuntu-archive to publish to the security pocket) via [[http://bazaar.launchpad.net/~ubuntu-bugcontrol/ubuntu-qa-tools/master/view/head:/security-tools/unembargo|lp:ubuntu-qa-tools/security-tools/unembargo]]. However, sometimes large packages timeout and an archive admin must use unembargo-package. IMO this should continue to be supported for emergency situations.)--
   * --([cjwatson] Scripts aren't the only answer to timeouts. Jobs are fine for this too, and Archive.copyPackage creates an asynchronously-executed job. Once unembargo can use Archive.copyPackage, unembargo-package will no longer be necessary.)--
   * --([cjwatson] It might be a little more complicated than this, since unembargoing also requires re-uploading files to the unrestricted librarian.)--
 * --(point-release-snapshot (could be done on a mirror, especially if we only snapshotted dists/))--
 * --(sync blacklist (should move to blacklist entries in LP proper))--
  * --([cjwatson] Moving everything in sync-blacklist to blacklist entries in LP turns out to be difficult because not everything in the blacklist has a Distro``Series``Difference, for example because packages are sometimes pre-emptively blacklisted before they exist in Debian, or because the version of the package currently in Debian has already been removed from Ubuntu so it isn't currently considered worth a DSD but may be in future. However, we can meet the goals of this specification by simply moving the blacklist to a branch in Launchpad, and so I've moved it to [[https://code.launchpad.net/~ubuntu-archive/+junk/sync-blacklist|lp:~ubuntu-archive/+junk/sync-blacklist]].)--
 * --(new-source (fairly easy to do on top of mass-sync client once implemented))--
 * override helpers: --(new-binary-debian-universe, new-remove-duplicates, kernel-overrides, firefox-overrides)--, security-kernel-overrides
 * --(copy-report (easy to do on top of copy-package API client, though should be done in a cron job on snakefruit using a bot LP account in the ubuntu-archive team; see also [[Bug:616704]]; impeded by [[Bug:1006871]] and [[Bug:1006917]]))--
 * --(publish-queue (easy to do on top of queue API client))--
Line 66: Line 92:

== BoF agenda and discussion ==

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.
 * --(Fix NewReleaseCycleProcess documentation to say that new-series sanity checking should be done on a mirror)--
 * --(Stale files pile up in `/srv/launchpad.net/ubuntu-archive/ubuntu-germinate/` from time to time, even for the current series, when seeds or flavours or architectures are removed. At the moment we notice this because the archive reports stop updating. These should be cleared out automatically.)--

Summary

The Ubuntu archive administration team has always required direct privileged shell access to the ftpmaster system in order to perform many of its routine tasks. This is a security problem, it prevents us from opening some tasks up to those who are not Canonical employees, and it makes it hard for us to improve our own tools. Improve the Launchpad API to handle all our requirements and write suitable API clients.

We will know we have succeeded when archive admins no longer require shell access to do their jobs.

Release Note

This is not a user-facing feature.

Rationale

Fixing this has been a long-term goal for many years. Specific problems caused by the current mechanism include:

  • The archive administration account has a horrendous amount of privilege. While we have not spent much time coming up with specific attack vectors, an Ubuntu archive administrator could almost certainly inject malicious code into Ubuntu bypassing any of the usual auditing mechanisms, could probably forge uploads by other developers, and so on; and they have direct access to the archive signing key (though not the master key used for rollovers). While all our archive admins are highly trustworthy individuals, this means that we are constrained in how we can expand the team because their integrity has to be absolutely unquestionable, and it means that a compromise of an archive admin's local machine could have disastrous consequences if the attacker realised what they'd stumbled on. This is much more privilege than we in fact need.

  • Some archive administration tasks cannot be delegated beyond Canonical staff. There are community members who would happily take some of this on if we'd let them, and who could be trusted to do so.
  • It is difficult for the archive team to improve their own tools, because they are part of the Launchpad source code. While the basic database operations clearly must be in Launchpad, we could be more efficient if we could more easily build our own user interfaces. The ftpmaster tools have particularly weak UI by comparison with the rest of Launchpad because the Launchpad developers rarely if ever use them outside the test suite.

The direction for improvement is in large part obvious: the Launchpad API should be extended so that all operations currently performed by archive admins using shell access to ftpmaster have been replaced by API clients. At that point our shell access can be removed.

In fact, a substantial part of this work has been done over the years, and particularly recently as part of the derived distributions project, along with some rsync modules that allowed us to move several reporting jobs to another machine. However, there is still a good deal to do, and it seems likely that it will only ever get done if we make a specific coordinated effort to do it. This will require substantial although shallow work on Launchpad, and continued work by the Ubuntu archive admins to migrate our scripts elsewhere.

Implementation

This is broken down by user. Completing everything in either of the first two categories would allow incremental reductions in archive admin privilege before the entire project is complete.

lp_publish user

This user owns the published archive itself. Removing our access to this user and its associated group would remove our direct access to the archive signing key and much of our ability to untraceably modify the archive.

  • Copying custom uploads to -updates

  • Copying DDTP tarballs to -updates (slightly different from previous bug, although dependent on it)

  • more-extra.override.* symlinks when creating new series

  • [jdstrand] taking publisher offline can be important at certain times, and is currently done by updating the crontab for the lp_publish user. Is this something that we should continue to support or just rely on a losa (or similar)?
    • [cjwatson] Specific examples would be useful. The ones I'm aware of are:
      • Copying custom uploads to -updates

      • Opening new series

      • Avoiding having to wait an hour when in a rush at release times (should be much less of an issue now that the publisher runs every half an hour; otherwise, ask webops to do it)
      • Security regressions (IME turning off the publisher is no longer particularly useful there; also see the previous item)
      • Arcane situations that require us to do a careful publisher run (these should be fixed in other ways, e.g. adding a way for archive admins to mark a pocket dirty in the API; however, the simple workaround for this is to do some trivial change-override operation in the relevant pocket, such as changing the section of a source publication)
    • [cjwatson] I'm inclined to say that in general once we've fixed a few bits of low-hanging fruit it will be tolerable to rely on webops for this, and a worthwhile tradeoff for the gain in datacentre security.
  • The archive key is owned by lp_archive, and group lp_publish

lp_queue user

Done!

This user is responsible for processing incoming uploads. Removing our access to this user would remove our ability to forge uploads by other developers.

lp_archive user

Everything else.

  • change-override

    • [jdstrand] must work with all pockets as the security team relies on this to fix up various mis-copies prior to USN publication

  • copy-package (API won't copy to PPAs yet unless the feature flag soyuz.copypackageppa.enabled is set; limitations on copying from PPAs)

    • [jdstrand] should support copying from an arbitrary PPA to an arbitrary pocket. Eg, the security often copies from ubuntu-mozilla-security or ubuntu-security-proposed to -proposed (this allows us to build packages without -updates enabled in the build, so they can later be copied to -security after going through verification)

      • [jdstrand] workaround: use unembargo --copy from lp:ubuntu-qa-tools which does work with arbitrary PPAs (only needed for private-to-public copies)

    • [cjwatson] Archive.syncSource times out on large packages and Archive.copyPackage seems to have some mysterious trouble with copying from stable-proposed to stable-updates (this was confusion due to them landing in the queue; sru-release now prints a message about this as a hint)

  • lp-remove-package

  • queue

    • [jdstrand] while I have moved away from doing NEW reviews on cocoplum itself, I still use queue for the following:

      • to see the list of packages to process (while I prefer a cli view for this, it wouldn't be the worst if I had to use the web frontend)

        • [cjwatson] I intend to add a queue API sufficient to implement a similar CLI client-side, at the cost of a bit of extra download time. I know of no reason it shouldn't have a feature set equivalent to our current CLI tool.

      • to download packages which I then scp locally. While I could also use the web frontend here, I have scripting around this and it would be nice if the new scripts supported downloading locally as well. (The API may already support this; I went with the easiest thing I could find at the moment I wrote my scripts for this)

        • [cjwatson] Ditto.

      • changing overrides in one of the queues *before* deNEWing them. Eg, if I see a NEW package is in binary NEW because it simply added a new binary package, after reviewing the package I will use queue to adjust the overrides in the NEW queue before accepting it. This saves times for everyone involved.

        • [cjwatson] Ditto.

  • unembargo-package

    • [jdstrand] the security team uses the API currently for unembargoing packages (ie, don't have to be a member of ubuntu-archive to publish to the security pocket) via lp:ubuntu-qa-tools/security-tools/unembargo. However, sometimes large packages timeout and an archive admin must use unembargo-package. IMO this should continue to be supported for emergency situations.

      • [cjwatson] Scripts aren't the only answer to timeouts. Jobs are fine for this too, and Archive.copyPackage creates an asynchronously-executed job. Once unembargo can use Archive.copyPackage, unembargo-package will no longer be necessary.

      • [cjwatson] It might be a little more complicated than this, since unembargoing also requires re-uploading files to the unrestricted librarian.

  • point-release-snapshot (could be done on a mirror, especially if we only snapshotted dists/)

  • sync blacklist (should move to blacklist entries in LP proper)

    • [cjwatson] Moving everything in sync-blacklist to blacklist entries in LP turns out to be difficult because not everything in the blacklist has a DistroSeriesDifference, for example because packages are sometimes pre-emptively blacklisted before they exist in Debian, or because the version of the package currently in Debian has already been removed from Ubuntu so it isn't currently considered worth a DSD but may be in future. However, we can meet the goals of this specification by simply moving the blacklist to a branch in Launchpad, and so I've moved it to lp:~ubuntu-archive/+junk/sync-blacklist.

  • new-source (fairly easy to do on top of mass-sync client once implemented)

  • override helpers: new-binary-debian-universe, new-remove-duplicates, kernel-overrides, firefox-overrides, security-kernel-overrides

  • copy-report (easy to do on top of copy-package API client, though should be done in a cron job on snakefruit using a bot LP account in the ubuntu-archive team; see also 616704; impeded by 1006871 and 1006917)

  • publish-queue (easy to do on top of queue API client)

  • populate-archive
  • Fix NewReleaseCycleProcess documentation to say that new-series sanity checking should be done on a mirror

  • Stale files pile up in /srv/launchpad.net/ubuntu-archive/ubuntu-germinate/ from time to time, even for the current series, when seeds or flavours or architectures are removed. At the moment we notice this because the archive reports stop updating. These should be cleared out automatically.


CategorySpec

FoundationsTeam/ReplaceArchiveAdminShellAccess (last edited 2013-10-02 11:45:46 by 82-69-40-219)