Launchpad Entry: foundations-q-replace-archive-admin-shell-access
Packages affected: Launchpad, ubuntu-archive-tools
The Ubuntu archive administration team has always required direct privileged shell access to the ftpmaster system in order to perform many of its routine tasks. This is a security problem, it prevents us from opening some tasks up to those who are not Canonical employees, and it makes it hard for us to improve our own tools. Improve the Launchpad API to handle all our requirements and write suitable API clients.
We will know we have succeeded when archive admins no longer require shell access to do their jobs.
This is not a user-facing feature.
Fixing this has been a long-term goal for many years. Specific problems caused by the current mechanism include:
The archive administration account has a horrendous amount of privilege. While we have not spent much time coming up with specific attack vectors, an Ubuntu archive administrator could almost certainly inject malicious code into Ubuntu bypassing any of the usual auditing mechanisms, could probably forge uploads by other developers, and so on; and they have direct access to the archive signing key (though not the master key used for rollovers). While all our archive admins are highly trustworthy individuals, this means that we are constrained in how we can expand the team because their integrity has to be absolutely unquestionable, and it means that a compromise of an archive admin's local machine could have disastrous consequences if the attacker realised what they'd stumbled on. This is much more privilege than we in fact need.
- Some archive administration tasks cannot be delegated beyond Canonical staff. There are community members who would happily take some of this on if we'd let them, and who could be trusted to do so.
- It is difficult for the archive team to improve their own tools, because they are part of the Launchpad source code. While the basic database operations clearly must be in Launchpad, we could be more efficient if we could more easily build our own user interfaces. The ftpmaster tools have particularly weak UI by comparison with the rest of Launchpad because the Launchpad developers rarely if ever use them outside the test suite.
The direction for improvement is in large part obvious: the Launchpad API should be extended so that all operations currently performed by archive admins using shell access to ftpmaster have been replaced by API clients. At that point our shell access can be removed.
In fact, a substantial part of this work has been done over the years, and particularly recently as part of the derived distributions project, along with some rsync modules that allowed us to move several reporting jobs to another machine. However, there is still a good deal to do, and it seems likely that it will only ever get done if we make a specific coordinated effort to do it. This will require substantial although shallow work on Launchpad, and continued work by the Ubuntu archive admins to migrate our scripts elsewhere.
This is broken down by user. Completing everything in either of the first two categories would allow incremental reductions in archive admin privilege before the entire project is complete.
This user owns the published archive itself. Removing our access to this user and its associated group would remove our direct access to the archive signing key and much of our ability to untraceably modify the archive.
Copying DDTP tarballs to -updates (slightly different from previous bug, although dependent on it)
- [jdstrand] taking publisher offline can be important at certain times, and is currently done by updating the crontab for the lp_publish user. Is this something that we should continue to support or just rely on a losa (or similar)?
- [cjwatson] Specific examples would be useful. The ones I'm aware of are:
Copying custom uploads to -updates
Opening new series
- Avoiding having to wait an hour when in a rush at release times (should be much less of an issue now that the publisher runs every half an hour; otherwise, ask webops to do it)
- Security regressions (IME turning off the publisher is no longer particularly useful there; also see the previous item)
- Arcane situations that require us to do a careful publisher run (these should be fixed in other ways, e.g. adding a way for archive admins to mark a pocket dirty in the API; however, the simple workaround for this is to do some trivial change-override operation in the relevant pocket, such as changing the section of a source publication)
- [cjwatson] I'm inclined to say that in general once we've fixed a few bits of low-hanging fruit it will be tolerable to rely on webops for this, and a worthwhile tradeoff for the gain in datacentre security.
- [cjwatson] Specific examples would be useful. The ones I'm aware of are:
- The archive key is owned by lp_archive, and group lp_publish
This user is responsible for processing incoming uploads. Removing our access to this user would remove our ability to forge uploads by other developers.
Flushing uploads created by sync-source (mostly replaced by Archive.copyPackage; needs ~katie user unsuspending so that auto-sync client can use it, needs auto-sync client finishing, and possibly other odds and ends of API syncs)
Flushing uploads created by backport-source (use backportpackage instead?)
[jdstrand] must work with all pockets as the security team relies on this to fix up various mis-copies prior to USN publication
copy-package (API won't copy to PPAs yet unless the feature flag soyuz.copypackageppa.enabled is set; limitations on copying from PPAs)
[jdstrand] should support copying from an arbitrary PPA to an arbitrary pocket. Eg, the security often copies from ubuntu-mozilla-security or ubuntu-security-proposed to -proposed (this allows us to build packages without -updates enabled in the build, so they can later be copied to -security after going through verification)
[jdstrand] workaround: use unembargo --copy from lp:ubuntu-qa-tools which does work with arbitrary PPAs (only needed for private-to-public copies)
[cjwatson] Archive.syncSource times out on large packages and Archive.copyPackage seems to have some mysterious trouble with copying from stable-proposed to stable-updates (this was confusion due to them landing in the queue; sru-release now prints a message about this as a hint)
[jdstrand] while I have moved away from doing NEW reviews on cocoplum itself, I still use queue for the following:
to see the list of packages to process (while I prefer a cli view for this, it wouldn't be the worst if I had to use the web frontend)
[cjwatson] I intend to add a queue API sufficient to implement a similar CLI client-side, at the cost of a bit of extra download time. I know of no reason it shouldn't have a feature set equivalent to our current CLI tool.
to download packages which I then scp locally. While I could also use the web frontend here, I have scripting around this and it would be nice if the new scripts supported downloading locally as well. (The API may already support this; I went with the easiest thing I could find at the moment I wrote my scripts for this)
changing overrides in one of the queues *before* deNEWing them. Eg, if I see a NEW package is in binary NEW because it simply added a new binary package, after reviewing the package I will use queue to adjust the overrides in the NEW queue before accepting it. This saves times for everyone involved.
[jdstrand] the security team uses the API currently for unembargoing packages (ie, don't have to be a member of ubuntu-archive to publish to the security pocket) via lp:ubuntu-qa-tools/security-tools/unembargo. However, sometimes large packages timeout and an archive admin must use unembargo-package. IMO this should continue to be supported for emergency situations.
[cjwatson] Scripts aren't the only answer to timeouts. Jobs are fine for this too, and Archive.copyPackage creates an asynchronously-executed job. Once unembargo can use Archive.copyPackage, unembargo-package will no longer be necessary.
[cjwatson] It might be a little more complicated than this, since unembargoing also requires re-uploading files to the unrestricted librarian.
point-release-snapshot (could be done on a mirror, especially if we only snapshotted dists/)
sync blacklist (should move to blacklist entries in LP proper)
[cjwatson] Moving everything in sync-blacklist to blacklist entries in LP turns out to be difficult because not everything in the blacklist has a DistroSeriesDifference, for example because packages are sometimes pre-emptively blacklisted before they exist in Debian, or because the version of the package currently in Debian has already been removed from Ubuntu so it isn't currently considered worth a DSD but may be in future. However, we can meet the goals of this specification by simply moving the blacklist to a branch in Launchpad, and so I've moved it to lp:~ubuntu-archive/+junk/sync-blacklist.
new-source (fairly easy to do on top of mass-sync client once implemented)
override helpers: new-binary-debian-universe, new-remove-duplicates, kernel-overrides, firefox-overrides, security-kernel-overrides
copy-report (easy to do on top of copy-package API client, though should be done in a cron job on lillypilly using a bot LP account in the ubuntu-archive team; see also 616704; impeded by 1006871 and 1006917)
publish-queue (easy to do on top of queue API client)
Fix NewReleaseCycleProcess documentation to say that new-series sanity checking should be done on a mirror
Stale files pile up in /srv/launchpad.net/ubuntu-archive/ubuntu-germinate/ from time to time, even for the current series, when seeds or flavours or architectures are removed. At the moment we notice this because the archive reports stop updating. These should be cleared out automatically.