SystemdForUpstartUsers

Introduction

This document compares Upstart and systemd with a view to aiding in the transition to the latter.

Support status

First, it is important to note that systemd is only fully supported in Ubuntu 15.04 and later releases. While systemd is available in prior releases through the Ubuntu repositories, there is a deemphasis of support for these releases as noted here. Hence, it is advised to use the default upstart on prior releases.

System Init Daemon

This has changed as part of the Ubuntu 15.04 devel cycle.

Ubuntu 15.04 (using Systemd by default):

  • Systemd runs with PID 1 as /sbin/init.

  • Upstart runs with PID 1 as /sbin/upstart.

Prior versions (using Upstart by default):

  • Upstart runs with PID 1 as /sbin/init.

  • Systemd runs with PID 1 as /lib/systemd/systemd.

Switching init systems

If you are running Ubuntu vivid (15.04), you can easily switch between upstart and systemd at will since both packages are installed at present. As of March 9 2015, vivid was changed to use systemd by default, before that upstart was the default.

Switch to upstart for a single boot

In grub, select "Advanced options for Ubuntu", where you will find an "Ubuntu, with Linux ... (upstart)" entry. This will boot with init=/sbin/upstart.

If you have upstart-sysv installed and thus boot with upstart by default, there will be an "Ubuntu, with Linux ... (systemd)" entry, which will boot with init=/lib/systemd/systemd.

Permanent switch back to upstart

Install the upstart-sysv package, which will remove ubuntu-standard and systemd-sysv (but should not remove anything else -- if it does, yell!), and run sudo update-initramfs -u. After that, grub's "Advanced options" menu will have a corresponding "Ubuntu, with Linux ... (systemd)" entry where you can do an one-time boot with systemd.

If you want to switch back to systemd, install the systemd-sysv and ubuntu-standard packages.

High-level startup concept

Upstart's model for starting processes (jobs) is "greedy event-based", i. e. all available jobs whose startup events happen are started as early as possible. During boot, upstart synthesizes some initial events like startup or rcS as the "tree root", the early services start on those, and later services start when the former are running. A new job merely needs to install its configuration file into /etc/init/ to become active.

systemd's model for starting processes (units) is "lazy dependency-based", i. e. a unit will only start if and when some other starting unit depends on it. During boot, systemd starts a "root unit" (default.target, can be overridden in grub), which then transitively expands and starts its dependencies. A new unit needs to add itself as a dependency of a unit of the boot sequence (commonly multi-user.target) in order to become active.

Job vs. unit keywords

This maps the keywords that can occur in an upstart job to the corresponding ones in a systemd unit. Keywords which don't have a direct equivalent are marked with "-".

Upstart stanza

systemd unit file directive

systemd unit file section

Notes

apparmor load

AppArmorProfile

Available in systemd version 210 and later

apparmor switch

-

author

-

chdir

WorkingDirectory

Service

chroot

RootDirectory

console output

StandardOutput=tty, StandardError=tty

console owner

StandardOutput=tty, StandardError=tty

No real equivalent?

console none

StandardOutput=null, StandardError=null

description

Description

Unit

env

Environment, EnvironmentFile

Service

exec

ExecStart

Service

expect fork

Type=forking

Unit

expect daemon

Type=forking

Unit

expect stop

Type=notify

Unit

Similar, not equivalent. Requires daemon to link to libsystemd-daemon and call sd_notify().

instance

Use "%I" and "%i" in ExecStart, etc to specify an instance

See /lib/systemd/system/getty@.service for an example

kill signal

KillSignal

kill timeout

TimeoutStopSec

limit as

LimitAS

limit core

LimitCORE

limit cpu

LimitCPU

limit data

LimitDATA

limit fsize

LimitFSIZE

limit memlock

LimitMEMLOCK

limit msgqueue

LimitMSGQUEUE

limit nice

LimitNICE

limit nofile

LimitNOFILE

limit nproc

LimitNPROC

limit rss

LimitRSS

limit rtprio

LimitRTPRIO

limit sigpending

LimitSIGPENDING

limit stack

LimitSTACK

manual

No directive(?) - use systemctl disable foo.service

nice

Nice

Unit

normal exit

SuccessExitStatus

oom score

OOMScoreAdjust

post-start exec/script

ExecStartPost

Service

post-stop exec/script

ExecStopPost

Service

pre-start exec/script

ExecStartPre

Service

pre-stop

-

reload signal

ExecReload=/bin/kill -SIGFOO $MAINPID

Service

respawn

Restart=on-failure

Service

respawn limit

RestartSec

script/end script

See shell scripts below

setgid

Group

Service

setuid

User

Service

start on

Wants, Requires, Before, After

Unit

stop on

Conflicts, BindsTo (but not commonly used)

Unit

task

Type=oneshot

Unit

umask

UMask

Unit

usage

Documentation

Unit

no direct equivalent

version

-

Shell scripts

systemd does not provide special support for shell scripts (by design). For short shell commands you can use something like

   ExecStart=/bin/sh -ec 'echo hello'

Longer scripts are usually program logic and should not be directly in a conffile and duplicated between upstart and systemd; factor it out in a proper script in e. g. /usr/share/myapp/ and call it from both the upstart job and the systemd unit.

Automatic starting

As described above, services which want to start during boot (i. e. are not activated through sockets, D-BUS, or similar) need to become a dependency of an existing boot target. Those need an [Install] section with a WantedBy= that specifies the unit which that new service wants to become a dependency of (see man systemd.unit). Very commonly this is multi-user.target, which is roughly equivalent to start on runlevel [2345] in upstart; see man systemd.special for other common targets.

Invalid configurations

  • upstart
    • Refuses to start a job if it contains invalid syntax.
  • systemd
    • Ignores invalid directives and starts the service. Note that you must test your configuration carefully since a typo will not be detected! Use systemd-analyze verify <unit> file to get warnings on typos and badly formatted options.

Override Files

  • upstart
    • Upstart allows part or all of a Job Configuration ("/etc/init/$job.conf") file to be overridden. To create an override file:

    • Create file "/etc/init/$job.override" containing one or more stanzas which take priority over their counterparts from the original "$job.conf" file.

  • systemd
    • Systemd allows a similar facility via "drop-ins": a drop-in allows a directive to be modified without changing the original unit file. To create a drop-in:
    • Create a subdirectory below either "/etc/systemd/system/" or "/lib/systemd/system/" called "${unit}.d/".

    • Create files called <something>.conf in the "${unit}.d/" directory containing the directives that you wish to override.

Behavioral differences (aka Gotchas)

  • systemd will refuse to run a binary (via ExecStart, ExecStartPre, etc) unless the full path to the binary is specified:

    • BAD (systemd will not search for sleep in $PATH):

         ExecStart=sleep 20

      GOOD (absolute path specified):

         ExecStart=/bin/sleep 20

Commands

Note that these are commands for interactive human usage. Package maintainer scripts, ifupdown hooks and similar must always use the init system agnostic abstractions like invoke-rc.d.

Operation

Upstart Command

Systemd equivalent

Notes

Start service

start $job

systemctl start $unit

Stop service

stop $job

systemctl stop $unit

Restart service

restart $job

systemctl restart $unit

See status of services

initctl list

systemctl status

Check configuration is valid

init-checkconf /tmp/foo.conf

systemd-analyze verify <unit_file>

Show job environment

initctl list-env

systemctl show-environment

Set job environment variable

initctl set-env foo=bar

systemctl set-environment foo=bar

Remove job environment variable

initctl unset-env foo

systemctl unset-environment foo

View job log

cat /var/log/upstart/$job.log

sudo journalctl -u $unit

tail -f job log

tail -f /var/log/upstart/$job.log

sudo journalctl -u $unit -f

Show relationship between services

initctl2dot

systemctl list-dependencies --all

Shows pstree-style output.

Example Services

Example Upstart Service

/etc/init/foo.conf:

description "Job that runs the foo daemon"

# start in normal runlevels when disks are mounted and networking is available
start on runlevel [2345]

# stop on shutdown/halt, single-user mode and reboot
stop on runlevel [016]

env statedir=/var/cache/foo

# create a directory needed by the daemon
pre-start exec mkdir -p "$statedir"

exec /usr/bin/foo-daemon --arg1 "hello world" --statedir "$statedir"

Example Systemd service

/lib/systemd/system/foo.service:

[Unit]
Description=Job that runs the foo daemon
Documentation=man:foo(1)

[Service]
Type=forking
Environment=statedir=/var/cache/foo
ExecStartPre=/usr/bin/mkdir -p ${statedir}
ExecStart=/usr/bin/foo-daemon --arg1 "hello world" --statedir ${statedir}

[Install]
WantedBy=multi-user.target

Outstanding Work

If you'd like to help out with the migration, take a look at ...

... and then come and chat to us on #ubuntu-devel.

Common Idioms

Service that specifies shell meta-characters

  • upstart
    • Upstart automatically detects if a job requires a shell to expand meta characters. For example, the following job will automatically be run via "/bin/sh -e":

         exec mydaemon --date $(date)
         
  • systemd
    • systemd does no such detection so the equivalent unit file would need to be:
         [Service]
         ExecStart=/bin/sh -ec "exec /usr/bin/mydaemon --date $(date)"
         

Service that does not fork (runs in foreground)

  • upstart job
    •    description "blah, blah, blah"
         exec sleep 999
         
  • systemd unit
    •    [Unit]
         Description=blah, blah, blah
         
         [Service]
         Type=simple
         # (NOTE: "Type=simple" is the default)
         ExecStart=/usr/bin/sleep 999
         

Do not run service if no daemon configuration file created

  • upstart job
    •    # defines PORT variable
         env config=/etc/default/foo
      
         pre-start script
         [ -e "$config" ] || { stop; exit 0; }
         end script
      
         exec mydaemon --port="$PORT"
         
  • systemd unit
    •    [Service]
         Type=forking
         EnvironmentFile=/etc/default/foo
         ExecStart=/usr/bin/mydaemon --port=$PORT
         

Note: EnvironmentFile=/etc/default/foo is making /etc/default/foo mandatory. If the environment file (and so variables) are optional, you can use: EnvironmentFile=-/etc/default/foo (notice the - in front of the path).

/etc/default files which enable/disable jobs

enable=1|0 type settings in /etc/default files should generally be avoided. The canonical way to enable/disable a service in an init system agnostic way is update-rc.d <service> enable|disable, which will translate to init system specific actions such as adding/removing symlinks (SysV and systemd) or creating/removing job override files (upstart). For systemd in particular, admins also often call systemctl enable|disable <service> directly. Thus these settings are redundant in /etc/default.

There is no clean way to evaluate these in a systemd unit. You can check them in ExecStartPre=, but that would mean that the unit will be in "failed" state if the service gets disabled that way, and so, is not desirable.

For these reasons (confusing/duplication/cannot be modelled in systemd), these settings should be removed. This was done in whoopsie 0.2.42, you can check its diff for a transition which respects the old default setting and removes it on upgrade.

Start a service when a file is created

  • upstart job
    •    start on file FILE=/var/crash/*.crash EVENT=create
         exec crash-handler-daemon
         
  • systemd
    • Two files are required:
    • /lib/systemd/system/foo.service:

         [Service]
         Type=forking
         ExecStart=/usr/bin/crash-handler-daemon
         
    • /lib/systemd/system/foo.path:

         [Path]
         PathExistsGlob=/var/crash/*.crash
         Unit=foo.service
         # (NOTE: the default is "Unit=foo.service" if this is called "foo.path")
         

Start a service when a D-Bus name is acquired

  • upstart job
    •    start on dbus SIGNAL=NameAcquired INTERFACE=... OBJPATH=... SENDER=... DESTINATION=...
         exec mydaemon
         
  • systemd
    • /lib/systemd/system/foo.service:

         [Unit]
         Description=Service that acquires a D-Bus name
         
         [Service]
         Type=dbus
         # (NOTE: systemd will consider this service ready when it claims the 'foo.bar.baz' name on dbus)
         BusName=foo.bar.baz
         ExecStart=/usr/bin/mydaemon
         
    • /lib/systemd/system/bar.service:

         [Unit]
         Description=Service that needs the foo service
         Requires=foo.service
         After=foo.service
         
         [Service]
         ExecStart=/usr/bin/mydaemon2
         

Well-known Sequence Points

  • Upstart provides a set of "well-known" events that jobs can make use of (such as "runlevel" and "started"). These are summarised in the upstart-events(7) manual page.

  • Systemd has a similar concept in the form of "targets". See the systemd.special(7) manual page for details and bootup(7) for the boot sequence.

  • For the particular case of upstart's static-network-up: Our ifupdown integrates that into network-online.target (see man systemd.special(7)). Thus, translate start on static-network-up to Requires/After=network-online.target.

Environment Differences

Both Upstart and systemd provide a very limited environment to the services they run. The environment can be modified by specifying additional stanzas (for Upstart) or directives (for systemd).

However, there are some subtle differences between the default service environment provided by both init systems. The table below shows the major differences:

Description

Upstart

systemd

Environment variables set

PATH, TERM

PATH, all from /etc/default/locale

Standard out and Standard error

terminal (pty pseudo-terminal)

pipes (to journal)

Note that this information is subject to change; to determine a precise difference on your system:

  1. Install the procenv tool:

     $ sudo apt-get -y install procenv
     
  2. Create an Upstart job to run procenv:

     $ cat <<EOT | sudo tee /etc/init/procenv.conf
     description "Display Upstart environment"
     exec procenv --file=/tmp/procenv-upstart.log
     EOT
     
  3. Run procenv under upstart:

     $ sudo start procenv
     
  4. Create a systemd unit file to run procenv:

     $ cat <<EOT | sudo tee /lib/systemd/system/procenv.service
     [Unit]
     Description=Display systemd environment
    
     [Service]
     Type=oneshot
     ExecStart=/usr/bin/procenv --file=/tmp/procenv-systemd.log
     EOT
     
  5. Run procenv under systemd:

     $ sudo systemctl start procenv
     
  6. Compare the environments:
     $ diff /tmp/procenv-upstart.log /tmp/procenv-systemd.log
     

Common Problems

How to identify which init system you are currently booting with

At the time of writing, it is possible to boot an Ubuntu system with either Upstart or systemd since both are installed.

To determine which init daemon you are currently booting with, run:

  $ ps -p1 | grep systemd && echo systemd || echo upstart

Why don't some commands (like grep) work in /lib/systemd/system ?

If you hit this problem...

  $ cd /lib/systemd/system
  $ grep foo *
  grep: invalid option -- '.'
  Usage: grep [OPTION]... PATTERN [FILE]...
  Try 'grep --help' for more information.

... you need to add "--" to the grep(1) call:

  $ cd /lib/systemd/system
  $ grep foo -- *

This is required since systemd provides a file called "-.slice". After the shell has expanded the asterisk ("*"), that file gets passed to the command in question (grep, cat, etc) and that command will then attempt to interpret "-.slice" as a command-line option rather than a filename.

Alternative work-around:

  $ cd /lib/systemd/system
  $ POSIXLY_CORRECT=1 grep foo *

To view the pesky file:

  $ cd /lib/systemd/system
  $ cat -- -.slice

Note that this issue only occurs when your current working directory is the directory containing "-.slice". The simplest work-around is to make the grep or cat call from a different directory:

  $ grep foo /lib/systemd/system/*
  $ cat /lib/systemd/system/-.slice

Debugging

Since this implies the service works under Upstart but is problematic under systemd, details of both systems are provided to allow for some comparison.

Boot Time

Common Setup

Remove the following from the kernel command-line via the grub menu:

  • "quiet"

  • "splash"

Upstart

  • Add "--debug to the kernel command-line via the grub menu.

  • Optionally add "console=ttyS0 to the kernel command-line via the grub menu if you have a serial console.

systemd

  • Add "systemd.log_level=debug  to the kernel command-line via the grub menu.

  • Optionally add one of the following too:
    • "systemd.log_target=kmsg"

    • "systemd.log_target=console"

Starting a rescue shell

  • Run:
    •    $ sudo systemctl enable debug-shell.service
  • Reboot.
  • If the system fails to boot, you can now switch to tty9 (CTRL+ALT+F9) for a getty console login.

From a running system

Upstart

To switch to debug mode for the system init (PID 1):

  $ sudo initctl log-priority debug

To switch to debug mode for a session init (init PID != 1):

  $ initctl log-priority debug
  $ tail -f ~/.xsession-errors

Debian Packaging

Packages installing systemd services should build-depend on dh-systemd and either call dh --with systemd (if they use dh) or call dh_systemd_enable and dh_systemd_start before/after dh_installinit respectively. Files under debian called *.service will be installed analogously to *.upstart files. See dh_systemd_enable(1p) for how to customize the installation.

Further Information

SystemdForUpstartUsers (last edited 2016-10-11 11:33:03 by adrian-bridgett)