This page is part of the debugging series — pages with debugging details for a variety of Ubuntu packages. |
Identifying the culprit
Systemd is not especially complex software, but there is some complexity in figuring out what to debug, based on the type of problem at hand.
Pick from the table below the systemd component to look at:
Type of problem |
systemd component |
network addresses (missing IP, can't ping 8.8.8.8. etc.) |
systemd-networkd |
cannot resolve addresses (nslookup google.com fails) |
systemd-resolved |
incorrect time, time not syncing to NTP |
systemd-timesyncd |
devices not showing up as expected |
systemd-udevd |
system log issues |
systemd-journald |
issues in text-based terminal (VTs, not in graphical mode) |
systemd-logind |
daemons, applications not starting at boot or at login |
systemd |
General debugging steps
- In doubt, file a bug report. It's easy to close a bug report if it turns out not to be a bug.
- Attempt to make sure there is as simple as possible a process to reproduce the issue
- Capture as many logs as possible relevant to the issue
- If this is specific to your setup (not reproducible on a new install), make sure any relevant configuration files are included in a bug report -- include contents of /run/systemd/XYZ, as appropriate to the type of issue. Include any files in /etc/systemd you may have changed yourself.
Debugging systemd components
systemd-networkd : network issues
Include the output of networkctl in bug reports.
Include the output of ip addr, ip link and ip route as appropriate.
- Check that the output of the commands above is what you expect to see -- if an interface is expected to be up and is down or degraded, or should have a specific address, mention it. The same applies if some route is missing or anything else is amiss.
To further find out what may be going wrong with systemd-networkd, you can also stop it, and restart it in debug mode, like so:
sudo systemctl stop systemd-networkd SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
The daemon will start in the foreground and output everything on console, with many details. These can be used by developers to identify what is going wrong.
systemd-resolved : DNS resolution issues
If nslookup XYZ or dig XYZ fail, also try systemd-resolve XYZ to attempt resolving the address, note if one works and the other command fails.
- Check that /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf. It should be the case in most installations.
Inspect the output of systemd-resolve --status, make sure expected DNS servers are showing there, and are showing for the expected interfaces. If necessary, verify that the right search domains are listed.
To further find out what may be going wrong with systemd-resolved, you can also stop it, and restart it in debug mode, like so:
sudo systemctl stop systemd-resolved SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-resolved
Then attempt to resolve the address again, watching for the messages displayed on the console; there will be lots of information that a developer can use to make sense of the issue, whether it is a bug, DNSSEC problem, etc.
systemd-timesyncd : time synchronization issues
You can restart systemd-timesyncd in debug mode and watch for the messages for issues:
sudo systemctl stop systemd-timesyncd SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-timesyncd
systemd-udevd
You can attempt to reproduce the issue by unplugging the affected device; then starting the udev monitor:
udevadm monitor
Replug the affected device, and make sure to include the output in your bug report.
systemd-journald
TBD
systemd-logind
Verify that the output of loginctl is as expected.
As necessary, dig further in loginctl show-session X, loginctl show-seat Y, or loginctl show-user Z.
systemd : daemon, app startup issues
Check the output of systemd-analyze and systemd-analyze critical-chain to identify unexpected issues at startup.
Check the output of systemd-analyze blame to identify unexpected delays at startup.
Verify whether the system's state is running according to systemd:
systemctl is-system-running
List the systemd units on the system, including any possible failed ones (which might not be an issue at all):
systemctl list-units
Check the logs for a systemd unit:
journalctl -u xyz