Identifying the culprit

Systemd is not especially complex software, but there is some complexity in figuring out what to debug, based on the type of problem at hand.

Pick from the table below the systemd component to look at:

General debugging steps

• In doubt, file a bug report. It's easy to close a bug report if it turns out not to be a bug.
• Attempt to make sure there is as simple as possible a process to reproduce the issue
• Capture as many logs as possible relevant to the issue
• If this is specific to your setup (not reproducible on a new install), make sure any relevant configuration files are included in a bug report -- include contents of /run/systemd/XYZ, as appropriate to the type of issue. Include any files in /etc/systemd you may have changed yourself.

Debugging systemd components

systemd-networkd : network issues

• Include the output of networkctl in bug reports.

• Include the output of ip addr, ip link and ip route as appropriate.

• Check that the output of the commands above is what you expect to see -- if an interface is expected to be up and is down or degraded, or should have a specific address, mention it. The same applies if some route is missing or anything else is amiss.

To further find out what may be going wrong with systemd-networkd, you can also stop it, and restart it in debug mode, like so:

sudo systemctl stop systemd-networkd
SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd

The daemon will start in the foreground and output everything on console, with many details. These can be used by developers to identify what is going wrong.

systemd-resolved : DNS resolution issues

• If nslookup XYZ or dig XYZ fail, also try systemd-resolve XYZ to attempt resolving the address, note if one works and the other command fails.

• Check that /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf. It should be the case in most installations.

• Inspect the output of systemd-resolve --status, make sure expected DNS servers are showing there, and are showing for the expected interfaces. If necessary, verify that the right search domains are listed.

To further find out what may be going wrong with systemd-resolved, you can also stop it, and restart it in debug mode, like so:

sudo systemctl stop systemd-resolved
SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-resolved

Then attempt to resolve the address again, watching for the messages displayed on the console; there will be lots of information that a developer can use to make sense of the issue, whether it is a bug, DNSSEC problem, etc.

systemd-timesyncd : time synchronization issues

You can restart systemd-timesyncd in debug mode and watch for the messages for issues:

sudo systemctl stop systemd-timesyncd
SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-timesyncd

systemd-udevd

You can attempt to reproduce the issue by unplugging the affected device; then starting the udev monitor:

udevadm monitor

Replug the affected device, and make sure to include the output in your bug report.

systemd-journald

TBD

systemd-logind

• Verify that the output of loginctl is as expected.

• As necessary, dig further in loginctl show-session X, loginctl show-seat Y, or loginctl show-user Z.

systemd : daemon, app startup issues

• Check the output of systemd-analyze and systemd-analyze critical-chain to identify unexpected issues at startup.

• Check the output of systemd-analyze blame to identify unexpected delays at startup.

Verify whether the system's state is running according to systemd:

systemctl is-system-running

List the systemd units on the system, including any possible failed ones (which might not be an issue at all):

systemctl list-units

Check the logs for a systemd unit:

journalctl -u  xyz