Contents

  1. Summary
  2. Release Note
    1. User Job Migration
  3. Rationale
    1. Existing User Sessions
      1. Limitations
    2. Existing Ubuntu Desktop Session
  4. Terminology
  5. Use Cases
  6. Requirements
    1. Core
    2. Additional (to be considered in the future but not planned for 13.04)
  7. Design
    1. Session Init
      1. Invocation
      2. Shell
    2. D-Bus
    3. Bridging Events
      1. System to User
      2. User to System
    4. Communication with PID 1
    5. Respawning user jobs and PID tracking
      1. Advantages
      2. Disadvantages
      3. Mechanics
    6. Proposed Changes to Events
      1. New Event Format
      2. System Events
        1. Emitting
        2. Reacting
        3. Restrictions
      3. User Events
        1. Emitting
        2. Reacting
        3. Restrictions
      4. Internal Changes
    7. Initctl
      1. Effect of UPSTART_SESSION
      2. initctl list
      3. New Commands
        1. List Sessions
        2. Environment Commands
    8. Existing User Jobs
    9. init-checkconf
    10. Configuration Files for User Jobs
    11. User Job Logging
    12. Stateful re-exec
      1. Design Impact
    13. Desktop Mechanics
      1. Desktop Session Startup
      2. Desktop Session Shutdown
        1. Current Ubuntu Architecture
        2. Session Init Ubuntu Architecture
          1. Advice
          2. Implications
          3. Session
          4. PID 1
      3. Desktop Session Lifecycle
    14. Process Overview
      1. Existing Desktop
      2. New Desktop
  8. Default System-Provided Session Init Jobs
  9. Security
  10. Performance
  11. Testing
  12. Outstanding Issues
  13. Additional Information
    1. Alternative to PR_SET_CHILD_SUBREAPER
      1. Advantages
      2. Disadvantages
      3. Mechanics
    2. References

Summary

This specification describes the plan to implement Enhanced User Session support in Upstart.

Release Note

User Job Migration

Any users who ran Upstart User Jobs automatically (without actually having a user login to the system in the conventional fashion), will now have to first start the Session Init manually by invoking "init --user".

Rationale

Existing User Sessions

Currently, user sessions are extremely simple, allowing a non-privileged user to create and manage jobs.

Limitations

Existing User sessions suffer from some limitations:

Existing Ubuntu Desktop Session

The current desktop startup sequence is quite time consuming and crucially starts a number of services that run as the user which do not necessarily need to be started at user login time.

There is thus an opportunity to enhance the desktop experience for both standard users and those running in resource-constrained environments to:

This fits perfectly into Upstart's event-based model and will allow the desktop startup and shutdown sequence to be made more efficient.

Using Upstart to manage the desktop session has the following additional benefits:

Terminology

"Upstart"

Unless qualified, this refers to the "/sbin/init" binary.

"User Job"

Job created by a non-privileged user (in $HOME/.init/) that is managed by PID 1 but also controllable by the user via initctl.

"Setuid Job"

Job created by a privileged user (in /etc/init/) that runs as a non-privileged user. Can only be managed by PID 1 and the superuser.

"Session Init"

Supplementary instance of "/sbin/init" (a "sub-init") invoked by a non-privileged user and running as that user for the duration of a login session.

"Session Init Instance"
Since instance of a Session Init process. Note that multiple Session Init processes may be running on a system, potentially multiple per individual user.
"Session Job"

Upstart job ($HOME/.init/) managed by the users Session Init and controllable by the user via initctl. This is the new name for a "User Job".

"System Event"

Event emitted by PID 1 or a process running as root.

"User Event"
Event emitted directly by a Session Init or non-privileged user process.
"System Directory"

Directory containing Job Configuration Files and Override files that is not a sub-directory of the users home directory ($HOME).

"System Files"
Job Configuration Files and Override Files that live in a System Directory.
"User Directory"

Directory containing Job Configuration Files and Override files that is a sub-directory of the users home directory ($HOME).

"User Files"
Job Configuration Files and Override Files that live in a User Directory.

Use Cases

Requirements

Core

Additional (to be considered in the future but not planned for 13.04)

Design

Session Init

Invocation

To start Upstart as a Session Init daemon: 

  /sbin/init --user

When invoked with the "--user" option, Upstart will run as the user in question, emit the "desktop-session-start" event for the user (see upstart-events(7)) and also listen for events coming from PID 1.

Shell

All jobs run as the user will be run using "/bin/sh -e", as they are for Upstart running as PID 1. To be clear, the users shell as specified in the password database, will not be used.

D-Bus

Each Session Init daemon will register a D-Bus address of:

"unix:abstract=/com/ubuntu/upstart-session/$USER/$PID"

Where "$USER" corresponds to the username and "$PID" corresponds to the PID of the Session Init.

The path will be exported as UPSTART_SESSION in the environment of all child processes of the users Session Init and initctl will use that variable when the Session Init can't be found on the session bus. All child processes will also contain a variable specifying the PID of the Session Init they are being managed by: $UPSTART_SESSION_PID.

The Session Init will store the address of the session in $HOME/.cache/upstart/sessions/$PID.log using familiar shell syntax: 

UPSTART_SESSION=$UPSTART_SESSION

The $PID.log file will be removed on successful Session shutdown.

Bridging Events

System to User

Since user jobs will now run as a child of the Session Init process, it is necessary for a new bridge (upstart-event-bridge) to be written which will be started from the Session Init as an Upstart job. This bridge will connect to the D-Bus System Bus and bridge all system-level events to the Session Init jobs.

This will also necessitate a new D-Bus API for an EventEmitted signal which clients can listen for.

User to System

Not currently a requirement.

Communication with PID 1

PID 1 won't have any explicit knowledge of the Session Init. Session inits will just be acting like a standard process on the system, listening for events coming from the system Upstart on the D-Bus system bus.

Respawning user jobs and PID tracking

The plan is to use the new PR_SET_CHILD_SUBREAPER option of prctl(2).

If Upstart makes use of the new process control option PR_SET_CHILD_SUBREAPER, the Session Init will remain the parent of any double-forking daemons run as the user and to receive appropriate SIGCHLD signals. This simplifies the design significantly such that there are no process state change notifications between PID 1 and the Session Init required, but there are issues associated with using this facility.

Advantages

Disadvantages

Mechanics

Proposed Changes to Events

Upstart allows event names to take any form but to provide full namespacing at the user level (allowing user jobs to distinguish between "system events" and "user events"), an extended syntax is required.

The proposal is to consider colon (':') special iff it appears as the first byte of the event name. This is a behavioural change.

If the first byte of an event name is a colon, the event is treated as below, else the event is treated as it is currently.

New Event Format

Event Format:

    :<optional_type>:<event_name>

Where,

  - <optional_type> is one of:

    - 'sys' for system events.
    - 'user' for user events.
    - the nul string ('').

  - '<event_name>' can be of any format.

When used by PID 1, the event type is a facility to optionally route events to either only System Jobs or only Session Inits (and thus only to User Jobs).

When used by User Jobs, the event type is a way for User Jobs to discriminate between System Events and User Events.

System Events

Emitting

$ sudo initctl emit       foo   # emits system event 'foo' (*1)
$ sudo initctl emit     ::foo   # emits system event 'foo' (*2, *B)
$ sudo initctl emit  :sys:foo   # emits system event 'foo' (*2, *B)
$ sudo initctl emit :user:foo   # emits user event 'foo' only

(*B) - Behavioural change.
(*1) - System Event still visible to Session Inits and thus Session Jobs.
(*2) - System Event NOT visible to Session Inits.

Reacting

start on       foo   # reference system event only (*3) implicitly
start on     ::foo   # reference system event only semi-explicitly (*B)
start on  :sys:foo   # reference system event only explicitly (*B)
start on :user:foo   # reference user event only explicitly - INVALID (*3, *B)

(*B) - Behavioural change.
(*3) - Since system jobs have no access to user events.

Restrictions

User Events

Emitting

$ initctl emit       foo   # emits user event 'foo'
$ initctl emit     ::foo   # emits user event 'foo' (*B)
$ initctl emit :user:foo   # emits user event 'foo' (*B)
$ initctl emit  :sys:foo   # ERROR: User cannot emit a System Event (*3).

(*B) - Behavioural change.
(*3) - FIXME: alternatively, this could emit ':sys:foo' user event.

Reacting

start on       foo   # reference system or user event implicitly
start on     ::foo   # reference user *or* system event semi-explicitly (*B)
start on  :sys:foo   # reference system event only explicitly (*B)
start on :user:foo   # reference user event only explicitly (*B)

(*B) - Behavioural change.

Restrictions

Internal Changes

Initctl

Effect of UPSTART_SESSION

As previously mentioned, initctl will become Session Init-aware such that its behaviour will be modified if the UPSTART_SESSION environment variable is set. The table below summarises the behaviour: 

|---------------------+----------+------------------+-------------+-------------+---------------|
| UPSTART_SESSION set | user     | command prefix   | list        | status      | job cmds (*J) |
|---------------------+----------+------------------+-------------+-------------+---------------|
| yes                 | non-root | initctl          | user        | user        | user          |
| yes                 | non-root | initctl --user   | user        | user        | user          |
| yes                 | non-root | initctl --system | system      | system      | ERROR ! (*2)  |
|---------------------+----------+------------------+-------------+-------------+---------------|
| no                  | non-root | initctl          | system (*B) | system (*B) | ERROR ! (*2)  |
| no                  | non-root | initctl --user   | ERROR ! (*4)| ERROR ! (*4)| ERROR ! (*4)  |
| no                  | non-root | initctl --system | system      | system      | ERROR ! (*2)  |
|---------------------+----------+------------------+-------------+-------------+---------------|
| yes                 | root     | initctl          | user (*1)   | user (*1)   | user (*1)     |
| yes                 | root     | initctl --user   | user (*1)   | user (*1)   | user (*1)     |
| yes                 | root     | initctl --system | system      | system      | system        |
|---------------------+----------+------------------+-------------+-------------+---------------|
| no                  | root     | initctl          | system      | system      | system        |
| no                  | root     | initctl --user   | ERROR ! (*4)| ERROR ! (*4)| ERROR ! (*4)  |
| no                  | root     | initctl --system | system      | system      | system        |
|---------------------+----------+------------------+-------------+-------------+---------------|

Key:

(*B) - Behavioural change: currently shows system+user if user jobs exist and user jobs enabled.
(*J) - job commands being start, stop, restart, reload.
(*1) - Shows 'user' jobs owned by the root user, which itself will have a Session Init running with PID != 1.
       Required to handle possibility of user logging into desktop session as root. Ahem...
(*2) - Normal users cannot manipulate system jobs.
(*3) - Not valid to attempt to start a user job and a system job at the same time.
(*4) - No way to identify current session to know which user jobs to consider.

initctl list

New Commands

List Sessions

A new command "initctl list-sessions" will be added to list all sessions owned by a user.

This will simply iterate through all files in $HOME/.cache/upstart/sessions/$PID.log, and list all running PIDs which correspond to the $PID element of the file and whose name is session-init or whose path is /sbin/init.

Environment Commands

The new environment commands (set-env, get-env, unset-env, list-env and reset-env) allow the environment to be manipulated for all jobs that will run in the future.

The primary reason for introducing these commands is to allow variables to be added to the Session Init to allow for a usable desktop experience. For example, since dbus-daemon will be started as an Upstart job, it is vital that there be a way to ensure the "DBUS_SESSION_BUS" variable be set correctly for all subsequent jobs.

However, there is scope for abusing these commands. As such, it will NOT be possible to use a destructive environment command (set-env, unset-env, reset-env) for PID 1. Further, since these commands can be used to provide a form of IPC between different job processes (such as between the pre-start and the main stanzas), it may well lead to inadvertent abuse where job processes (or even different jobs) communicate by setting variables within the Session Init, but then forget to "clean up" when they finish. As such, if a destructive environment command is used from within a job, unless overridden with an appropriate command-line option (--global), the impact will be to modify ONLY the environment table for that job. This provides the IPC mechanism, but has the attractive benefit of automatically cleaning up after the job has finished. In other words, the Session Init environment table is not polluted.

Existing User Jobs

Enhanced Upstart User Sessions replace the legacy Upstart User Jobs facility entirely. There are no chroot implications to this since it is not possible to run User Jobs from within a chroot environment (since user lookup cannot be performed by Upstart for a user within a chroot as the uids may not match).

The original Upstart "User Jobs" that were supervised by PID 1 created a Session object internally the first time the user ran an "initctl" command. The new architecture does not work this way: a Session object is no longer required since all Session Jobs are managed by the users Session Init process.

This implies that any users who ran User Jobs automatically (without actually having a user login to the system in the conventional fashion), will now have to first start the Session Init manually by invoking "init --user".

init-checkconf

Update such that it will run something like "init --user --list-jobs" which will not run a Session Init, but which will print:

This will allow a user/admin to make sense of the Collision Resolution rules (see below).

Configuration Files for User Jobs

When invoked as a Session Init the following search path will be used:

This is the order job configuration (foo.conf) and override files (foo.override) will be searched in.

The first found job configuration file (foo.conf) will be loaded.

Job files with the same name in subsequent search path directories will be ignored.

An override file (foo.override) is searched in the same order as job configuration file (foo.conf).

The first found override file (foo.override) will be applied against the job configuration file (foo.conf).

Override files with the same name in subsequent search path directories will be ignored.

Exception: override files (foo.override) in the search path directories after the one that has the matching job configuration file (foo.conf) are also ignored.

The job name (foo) will be derived from the job configuration file name (foo.conf)

See upstart cookbook for information about override files.

User Job Logging

The above also simplifies the handling of user job logging since now, the Session Init can handle all logging for jobs started as the user.

Jobs output will be logged to separate files in $XDG_CACHE_HOME (or $HOME/.cache/upstart/logs/ if not set).

Note that the initial implementation will not write to ~/.xsession-errors as has traditionally happened. However, it would be possible to provide this facility in addition to the standard Upstart logging if we introduce the ability to specify multiple keywords to the console stanza (for example, something like, "console log, syslog, xsession").

Stateful re-exec

When the Session Init detects that PID 1 has re-execed itself (by virtue of the fact that the Session Init D-Bus connection to PID 1 is severed or by receiveing the Restarted D-Bus signal, it must also re-exec itself to ensure that all running inits are running at the same version.

Note that if Upstart is downgraded to a version that does not support "--user", the Session Init will end immediately (since it is not possible to run Upstart as a Session Init, and it is also not possible to detect in a race-free manner that the "new" version of Upstart (having a lower version number than the currently running Session Init) does not support full user sessions.

Design Impact

The main impact of allowing Session Inits to re-exec themselves relates to the new "initctl set-env" command. This ephemeral data must be serialised as part of a re-exec to ensure expected behaviour.

Note that no upgrade test is required for this scenario since there is no previous version of Upstart that supports Enhanced User Sessions so there can never be a scenario where the re-exec'ed Upstart is unable to read the set-env environment variables set by the original instance of Upstart.

Note that allowing Upstart to read job configuration files from multiple directories does not have an impact since the stateful re-exec facility will honour /sbin/init command-line variables and will simply re-load those jobs as needed on re-exec. Note that there would be a subtle impact if the user were to change XDG_CONFIG_DIRS and "initctl set-env" were implemented such that the command actually changed Upstarts environment.

Desktop Mechanics

Desktop Session Startup

The Session Init will be initiated by:

Desktop Session Shutdown

Current Ubuntu Architecture

In Ubuntu up to 12.10, exiting the session through the indicator or the power button will do the following:

Session Init Ubuntu Architecture

With Upstart sessions, the Session Init will be terminated by:

In detail:

  1. User selects "Logout", "Restart" or "Shut Down" in the Unity menu.
  2. The following command is run: 
    •       $ initctl shutdown [ --type=$TYPE ] [--wait=$WAIT ]

      Where,
        
        TYPE=[logout|reboot|shutdown]
        WAIT=[<seconds>|forever|-1]

      And,
        '-1' is equivalent to 'forever'.

      Notes:

        - TYPE is not used by Upstart; it is exposed to all jobs
          via the "`session-end`" event though.
  3. The Session Init performs the following actions:

    1. Set shutdown variable.

    2. Disable "respawn" (Bug 750113)

    3. Calculate maximum "kill timeout" value (max_kill_timeout_running) for all currently running job instances.

    4. Generate list of currently-running jobs (running-jobs).

    5. Send appropriate kill signal to all jobs in running-jobs.

    6. Arrange for kill handler to display message if shutdown variable set and SIGKILL required: "Job $job ($instance) being forcibly killed".

      • This will help identify slow-to-kill / problematic jobs.

    7. Emit the event "session-end WAIT=$WAIT TYPE=$TYPE"

    8. Calculate max_initial_delay = max(WAIT, max_kill_timeout_running)

    9. Register periodic timer to fire every second. Each time timer callback is called, iterate running jobs:
      1. If quiescent mode variable set, enter quiescent mode.
      2. If there are no running jobs, enter quiescent mode.

      3. If jobs from running-jobs are still running after max_kill_timeout_running seconds and no other jobs are running, set variable denoting quiescent mode should be entered on next call to timer (to allow SIGKILLs to be sent to all running-jobs).

      4. If after WAIT seconds any jobs not in running-jobs are running:

        1. Calculate max_kill_timeout_new.

        2. Send kill signal to all running jobs not in running-jobs list.

      5. If jobs are still running after max(max_initial_delay, max_kill_timeout_new), set quiescent mode variable.

      1. "quiescent" mode:

        1. Display a message "Entering quiescent mode".

        2. Stop any new jobs from starting.

        3. Send SIGKILL to any remaining job processes.

        4. Emit the "session-exiting" event.

        5. Re-enable new jobs, to allow a globally available "session-exiting" job to run. This job is expected to make the appropriate ConsoleKit call depending on the value of $TYPE.

        6. Perform internal cleanup.

        7. Call exit (0).

Advice

Implications

Session

Session ending could take at worst: 

  ( max_initial_delay + max_kill_timeout_new )

Note that the algorithm above is very simple and ignores some of the pathological scenarios such as handling jobs which start when other jobs end where the latter only start when jobs in running-jobs end.

PID 1

The session shutdown code paves the way for a "pure" Upstart shutdown sequence for PID 1. Due to handling complexities between Upstart and SystemV though, this work will likely be deferred until the next cycle.

Desktop Session Lifecycle

An approximation of a system booting and logging in would then be:

  1. System starting
  2. Lightdm started
  3. User enters credentials
  4. PAM authentication and session creation
  5. Xsession scripts spawned as the user
  6. Upstart Session Init Instance started
    1. desktop-session-start emitted
    2. D-Bus job spawned
    3. Upstart starts listening on session bus
    4. gnome-session spawned
      1. legacy jobs are started by gnome-session
    5. user exits the session
    6. Session Init Instance stops all the jobs
    7. Session Init Instance exits
  7. User gets back to Lightdm

Process Overview

Existing Desktop

init
 |- lightdm
 |   |- Xorg
 |   |- lightdm ---session-child
 |        |- gnome-session --session=ubuntu
 |             |- compiz
 |             |- gwibber
 |             |- nautilus
 |             |- nm-applet
 |             :
 |             :
 |
 |- dbus-daemon --session
 |
 :
 :

New Desktop

init
 |- lightdm
 |   |- Xorg
 |   |- lightdm ---session-child
 |        |- session-init # <-- upstart running as normal user
 |             |- dbus-daemon --session
 |             |- gnome-session --session=ubuntu
 |             |- compiz
 |             |- gwibber
 |             |- nautilus
 |             |- nm-applet
 |             :
 |             :
 :
 :

Default System-Provided Session Init Jobs

To be useful, the system should provide a set of default jobs that Session Inits will have access to.

Such jobs should include:

Security

Performance

The impact of introducing an EventEmitted D-Bus signal needs to be measured.

Testing

Auto-package testing (see bug 1075976) needs to be introduced to allow:

Outstanding Issues

Additional Information

Alternative to PR_SET_CHILD_SUBREAPER

If PR_SET_CHILD_SUBREAPER is not used, it will be necessary for the Session Init to pass a PID tuple to PID 1 via D-Bus for every job the Session Init started, the tuple comprising "(self, child)" with "self" being the PID of the Session Init, and "child" being the primary PID of the process forked from "self". This would allow PID 1 to detect any state changes for processes nominally managed by a Session Init, and communicate those state changes back to the Session Init in question (there may of course be many running on a single system.

Advantages

Disadvantages

Mechanics

References

FoundationsTeam/Specs/RaringUpstartUserSessions (last edited 2013-03-27 14:18:37 by host-92-18-40-67)