---

• Launchpad Entry: foundations-q-upstart-stateful-re-exec

• Created: 12 November 2012

• Contributors: JamesHunt

• Packages affected: upstart

• Related blueprints: foundations-r-upstart-user-session-enhancements

• Code:

  • TODO:

Summary

This specification describes the plan to implement Enhanced User Session support in Upstart.

Release Note

• TODO:

Rationale

Existing User Sessions

Currently, user sessions are extremely simple, allowing a non-privileged user to create and manage jobs.

Limitations

Existing User sessions suffer from some limitations:

• user jobs cannot share the same name as system jobs.

• initctl aggregates both system jobs and user jobs, but does not distinguish between the two types.

• User jobs output cannot be logged (console log).

Existing Ubuntu Desktop Session

The current desktop startup sequence is quite time consuming and crucially starts a number of services that run as the user which do not necessarily need to be started at user login time.

There is thus an opportunity to enhance the desktop experience for both standard users and those running in resource-constrained environments to:

• defer starting certain desktop services (potentially indefinately).
• dynamically start certain desktop services when appropriate conditions are met.

This fits perfectly into Upstarts event-based model and will allow the desktop startup and shutdown sequence to be made more efficient.

Using Upstart to manage the desktop session has the following additional benefits:

• Provides an attractive side-effect of reducing CPU and memory pressure, thus minimising power consumption and thus maximising battery life.
• Makes entire session flow less opaque.
• Provides a good opportunity to document this aspect of the system.
• Allows users to enhance their own desktop experience using simple Upstart jobs.
• Provides a flexible framework to allow for more fine-grained user job control such as providing a well-known event indicating the user (or the system) wishes the system to move into a low-power state (akin to some sort of "flight-mode" or "minimal-mode").

Terminology

"Upstart"

Unless qualified, this refers to the "/sbin/init" binary.

"User Job"

Job created by a non-privileged user (in $HOME/.init/) that is managed by PID 1 but also controllable by the user via initctl.

"Setuid Job"

Job created by a privileged user (in /etc/init/) that runs as a non-privileged user. Can only be managed by PID 1 and the superuser.

"Session Init"

Supplementary instance of "/sbin/init" (a "sub-init") invoked by a non-privileged user and running as that user for the duration of a login session. Multiple Session Init processes may be running on a system, potentially multiple Session Inits running as the same user.

"Session Job"

Upstart job ($HOME/.init/) managed by the users Session Init and controllable by the user via initctl. This is the new name for a "User Job".

"System Event"

Event emitted by PID 1 or a process running as root.

"User Event"

Event emitted directly by a Session Init or non-privileged user process.

Use Cases

• Han is running Ubuntu on his battered old laptop which has a slow disk and wants to be able to login to his desktop session as quickly as possible.
• Whilst living on Tatooine, Luke likes to use his Ubuntu powered macrobinoculars to read eBooks. Since these are battery powered, and since hotspots in his home area are rare, has no need of networking services when the network is not "up", so doesn't want his eBook experience to be slowed by having to wait for geo-ip services and other networking services to start.
• Yoda wants to minimise the number of processes running as his user and wants services to auto-start on demand.
• Leia wants to start a custom service as herself once a particular file is created.
• Darth is running Ubuntu on his Nexus 7 tablet and wants to be able to maximise battery life to allow him to play games for as long as possible.

• R2D2, after being upgraded to Ubuntu by C3PO, wants to be able to run his holographic projection unit for as long as possible, but wants to disable all other non-essential services such as all networking (including wifi, bluetooth, avahi) and network services (sshd, ntpd, NFS, DropBox, firefox, gwibber, geo-ip, et cetera).

Requirements

Core

• General
  • Minimise changes to existing behaviour.
  • Allow user jobs to 'listen' for system events.
  • Allow Upstart to be started as a non-privileged user and receive events from the system upstart (PID 1).
  • Utilise design compatible with existing Upstart "user sessions", where possible.
  • Ensure Session Inits restart when PID 1 restarted.

    • This can be achieved by having the Session Init respond to the PID 1 "Restarted" D-Bus signal, but it may also be necessary for the Session Init to also auto-restart should their D-Bus connection to PID 1 be severed (although note that PID 1 should not sever its clients D-Bus connection once upstream D-Bus provides "dbus_connection_open_from_fd()".

• Job Configuration Files
  • Allow Upstart to search for Job Configuration Files in multiple directories.

    • (for example "~/.init" and "/etc/xdg/init/" by specifying "init --confdir ~/.init --confdir /etc/xdg/init").

  • Allow override files to apply to any directory that has been specified as potentially containing Job Configuration Files.
• Extra watches/bridges

  • "upstart-file-bridge"

    • Write an inotify bridge.

      • Emit "file" event.

      • Handle create, modify and delete for both files and directories.

    • Document limitations caused by the inability of inotify to handle recursive-watches in a truly race-free manner.

  • "upstart-gsettings-bridge"

    • Write a gsettings bridge.

      • Emit "gsettings" event with the key that changed and the values.

• Jobs
  • Allow user jobs to have same name as system job.
  • Allow user jobs to be distinguished from system jobs.
  • Fully implement job namespacing inside Upstart.
• Events
  • Fully implement event namespacing inside Upstart.
  • Allow system events to be visible to the Session Init.
  • Disallow user events from one Session Init from being visible to another users Session Init.
• Re-exec
  • Allow Session Inits to perform stateful re-exec (requires assertion changes).
• D-Bus

  • Add "EventEmitted" D-Bus signal (specifically for PID 1) to allow Session Inits to proxy system-level events to user jobs.

  • Add "Restarted" D-Bus signal to allow Session Inits to detect that PID 1 has re-execed.

• Desktop
  • Provide an event-based user login experience.
    • Allow for deferred startup of desktop services based on existence of files.

  • Offer a way to change the environment of future children of the Session Init by adding a "SetEnv" or similar D-Bus method.

  • Maintain compatibility with existing FreeDesktop facilities where

    • possible (for example Desktop Entry Specification (or ".desktop") files).

Additional (to be considered in the future but not planned for 13.04)

• Ability for user jobs to detect and react to a "low-power" / minimal
  • / "flight-mode" request from the user via a well-named Upstart event.

  • Needs update to upstart-events(7).

• Ability for system-level jobs to detect and react to "low-power" mode request.
• Allow Session init events to be visible to PID 1 and therefore system jobs.
• Console
  • Allow jobs initiated from a console login to be managed by a users Session Init process (Discuss A11Y aspects with Luke Yelavich).
    • Spawn a Session Init on console login.

Design

Session Init

Invocation

To start Upstart as a Session Init daemon:

  /sbin/init --user

When invoked with the "--user" option, Upstart will run as the user in question and listen for events coming from the system init (PID 1).

D-Bus

Each Session Init daemon will register a D-Bus address of:

"unix:abstract=/com/ubuntu/upstart-session/$PID"

Where "$PID" corresponds to the PID of the Session Init. The path will be exported as UPSTART_SESSION in the environment of all upstart children and initctl will use that variable when upstart can't be found on the session bus.

Bridging Events

System to User

Since user jobs will now run as a child of the Session Init process, it is necessary for the Session Init to connect to PID 1 and bridge all system-level events to the Session Init jobs.

This will necessitate a new D-Bus API for an EventEmitted signal which clients can listen for.

User to System

Not currently a requirement.

Communication with PID 1

PID 1 won't have any explicit knowledge of the Session Init. Session inits will just be acting like a standard process on the system, listening for events coming from the system upstart on the DBUS system bus.

Respawning user jobs

The plan here is to use the new PR_SET_CHILD_SUBREAPER option of prctl().

If Upstart makes use of the new process control option PR_SET_CHILD_SUBREAPER, the Session Init will remain the parent of any double-forking daemons run as the user and to receive appropriate SIGCHLD signals. This simplifies the design significantly such that there are no process state change notifications between PID 1 and the Session Init required, but there are issues associated with using this facility.

Advantages

• Simplicity - it minimises custom-logic to handle user jobs.
• Much more resilient to any possible denial-of-service vectors since only a minimal amount of communication between the Session Init and PID 1 is required.

Disadvantages

• By making use of a Linux-specific prctl(2) call, we effectively tie Upstart to systems running with a Linux kernel. This is a major restriction, but porting to other systems is already complicated by the fact that even the BSDs do not provide a full POSIX environment (missing "waitid(2)" for example).

• This feature may confuse some services which could legitimately check their parent PID and act differently when getppid() shows their parent is PID 1. By using PR_SET_CHILD_SUBREAPER, their parent PID will never be PID 1 so they may fail to work as designed.

• Even though this feature is available in the Linux kernel as of version 3.4, this means it would not work on Precise, the latest Ubuntu LTS release. This may be an issue since

  • Some chroot environments may be running for example Ubuntu Raring, but the "host" environment outside the chroot may be Ubuntu Precise, meaning jobs respawn will not work.

  • Some virtualised environments don't allow users to change kernel version, meaning jobs respawn will not work.

  • During an upgrade, session starting would not work. This means that in the event a user logs out after upgrade without rebooting, they would be getting a working session but any jobs that'd die won't get respawned.

Mechanics

• Have Upstart call prctl(PR_SET_CHILD_SUBREAPER) when running as a non-root user.

• In the event where the kernel doesn't support PR_SET_CHILD_SUBREAPER, upstart will print a warning and simply ignore any "respawn" statement.

Proposed Changes to Events

Upstart allows event names to take any form but to provide full namespacing at the user level (allowing user jobs to distinguish between "system events" and "user events"), an extended syntax is required.

The proposal is to consider colon (':') special iff it appears as the first byte of the event name. This is a behavioural change.

If the first byte of an event name is a colon, the event is treated as below, else the event is treated as it is currently.

New Event Format

Event Format:

    :<type>:<event_name>

Where,

  - <optional_type> is one of:

    - 'sys' for system events.
    - 'user' for user events.
    - the nul string ('').

  - '<event_name>' can be of any format.

When used by PID 1, the event type is a facility to optionally route events to either only System Jobs or only Session Inits (and thus only to User Jobs).

When used by User Jobs, the event type is a way for User Jobs to discriminate between System Events and User Events.

• When running as PID 1, a nul string <type> is an alias for sys.

• When not running as PID 1, a nul string <type> is an alias for user.

System Events

Emitting

$ sudo initctl emit       foo   # emits system event 'foo' (*1)
$ sudo initctl emit     ::foo   # emits system event 'foo' (*2, *B)
$ sudo initctl emit  :sys:foo   # emits system event 'foo' (*2, *B)
$ sudo initctl emit :user:foo   # emits user event 'foo' only

(*B) - Behavioural change.
(*1) - System Event still visible to Session Inits and thus Session Jobs.
(*2) - System Event NOT visible to Session Inits.

Reacting

start on       foo   # reference system event only (*3) implicitly
start on     ::foo   # reference system event semi-explicitly (*B)
start on  :sys:foo   # reference system event explicitly (*B)
start on :user:foo   # reference user event explicitly - INVALID (*3, *B)

(*B) - Behavioural change.
(*3) - Since system jobs have no access to user events.

Restrictions

• If a System Event is emitted that does not specify a type, it is potentially available to both System Jobs and User Jobs assuming a Session Init is running.

• If a System Event is emitted that specifies a type of sys, the event is only visible to System Jobs.

• If a System Event is emitted that specifies a type of user, the event is only visible to User Jobs.

User Events

Emitting

$ initctl emit       foo   # emits user event 'foo'
$ initctl emit     ::foo   # emits user event 'foo' (*B)
$ initctl emit :user:foo   # emits user event 'foo' (*B)
$ initctl emit  :sys:foo   # ERROR: User cannot emit a System Event (*3).

(*B) - Behavioural change.
(*3) - FIXME: alternatively, this could emit ':sys:foo' user event.

Reacting

start on       foo   # reference system or user event implicitly
start on     ::foo   # reference user event semi-explicitly (*B)
start on  :sys:foo   # reference user event explicitly (*B)
start on :user:foo   # reference user event only explicitly (*B)

(*B) - Behavioural change.

Restrictions

• If a User Event is emitted that does not specify a type, it is potentially available to any jobs managed by the Session Init.

• It is an error for a User Event to be emitted with a type of sys.

User Job Logging

The above also simplifies the handling of user job logging since now, the Session Init can handle all logging for jobs started as the user.

Stateful re-exec

When the Session Init detects that PID 1 has re-execed itself (by virtue of the fact that the Session Init D-Bus connection to PID 1 is severed or by receiveing the "Restarted DBUS signal, it must also re-exec itself to ensure that all running inits are running at the same version.

Note that if Upstart is downgraded to a version that does not support "--user", the Session Init will end immediately (since it is not possible to run Upstart as a Session Init, and it is also not possible to detect in a race-free manner that the "new" version of Upstart (having a lower version number than the currently running Session Init) does not support full user sessions.

Desktop Mechanics

The Session Init will be initiated by:

• Modifying /etc/X11/Xsession.d/99x11-common_start to exec "init --user" rather than exec "$STARTUP". This will possibly be limited to only the Unity session, at least initially.

• The Session Init will emit the "desktop-session-start" event.

• A dbus job depending on "desktop-session-start" will trigger and spawn the session bus. The job will also use the "initctl setenv" to push DBUS_SESSION_BUS_ADDRESS into the environment of all subsequent children.

• A gnome-session job will be "start on started dbus" and will spawn the gnome session with the proper DBUS_SESSION_BUS_ADDRESS set.

• Any number of other user jobs will then start.

The Session Init will be terminated by:

• The desktop session sending an exit event "initctl exit" (or similar)

• The Session Instance will react on that event, stop all the jobs and then exit itself, closing the session in the process.

Security

It must not be possible for a User Event to modify the behaviour of System Jobs.

Performance

The impact of introducing an EventEmitted D-Bus signal needs to be measured.

Testing

Auto-package testing needs to be introduced to allow:

• The /sbin/init binary, as built and packaged for production, to be tested.

• Complex scenarios involving PID 1 and Session Inits to be tested (See Security section).

Outstanding Issues

• Is performance impact of adding "EventEmitted" D-Bus signal

  • acceptable?
• Is there anything that should be done when switching users? (Possibly have a consolekit/logind bridge running against the system upstart so system and user jobs can react to a user switch)

Additional Information

• User jobs within a chroot won't be supported, at least not initially.
• A user can have multiple upstart sessions running, the DBUS path for those is based on the PID of upstart so can't possibly conflict.
• Console logins won't start upstart, at least not initially.
• The existing user sessions code will be removed and replaced by the new implementation.

References

• FoundationsTeam/Specs/QuantalUpstartStatefulReexec

• http://standards.freedesktop.org/desktop-entry-spec/latest/

• https://code.launchpad.net/~jamesodhunt/dbus/create-connection-from-fd

• http://upstart.ubuntu.com/cookbook/#ubuntu-well-known-events-ubuntu-specific

• http://upstart.ubuntu.com/cookbook/#job-configuration-file

• http://upstart.ubuntu.com/cookbook/#override-files