The boot sequence hangs or a service refuses to start
You reboot your Fedora machine after a routine dnf upgrade --refresh. The desktop environment loads, but your custom web server or database daemon never comes up. Or maybe you are SSH'd into a headless box and systemctl status just shows active (exited) when it should be active (running). The system is not broken. The service just hit a wall during initialization and systemd caught it.
What systemd is actually doing behind the scenes
Systemd does not just run commands. It manages a dependency graph. Every service unit declares what it needs before it starts and what depends on it after. When a service fails, systemd records the exit code, captures the standard output and standard error, and marks the unit as failed. It does not retry automatically. It waits for you to look at the logs, fix the configuration, and tell it to try again. Think of it like a factory assembly line. If one station jams, the line stops. The supervisor logs the jam, waits for a mechanic, and only moves forward when the station is cleared.
The unit file itself is just a configuration document. It tells systemd how to start the process, how to detect when it is ready, and what to do when it crashes. Most packages ship their unit files to /usr/lib/systemd/system/. Those files are managed by the package manager. You should never edit them directly. Package updates will overwrite your changes and the next dnf upgrade will silently break your configuration. Instead, systemd provides an override mechanism that keeps your modifications in /etc/systemd/system/. The override layer always wins. This separation keeps the system upgradeable and your customizations intact.
Read the journal before you guess
The first step is always the journal. Do not guess which configuration file is wrong. Run the journal query that isolates the unit and the current boot. Here is how to pull the exact log lines for a specific service without drowning in unrelated system noise.
# -u filters by unit name. Replace postgresql.service with your target.
# -b limits output to the current boot cycle.
# -x adds explanatory text for common error codes.
# -e jumps to the end of the output so you see the failure first.
journalctl -u postgresql.service -b -xe
The journal stores logs in binary format and indexes them by unit, priority, and timestamp. The -x flag is a community standard for quick debugging. It appends a short explanation to known error codes and signal names. The -e flag opens the pager at the bottom. Press q to exit. If the output is too long, add -n 20 to show only the last twenty lines.
You will typically see a sequence of events. Systemd sends the start signal. The process forks or executes. The process exits with a non-zero code or receives a termination signal. Systemd logs the failure and changes the unit state. Look for lines that mention configuration parsing errors, missing files, permission denials, or dependency timeouts.
# Example of a real systemd failure sequence
Oct 14 09:22:11 fedora-workstation systemd[1]: Starting PostgreSQL database server...
Oct 14 09:22:11 fedora-workstation postgresql-setup[4821]: Initializing database...
Oct 14 09:22:12 fedora-workstation postgresql-setup[4821]: FATAL: data directory "/var/lib/pgsql/data" has wrong ownership
Oct 14 09:22:12 fedora-workstation postgresql-setup[4821]: The files belonging to this database system will be owned by user "postgres".
Oct 14 09:22:12 fedora-workstation systemd[1]: postgresql.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 09:22:12 fedora-workstation systemd[1]: postgresql.service: Failed with result 'exit-code'.
Oct 14 09:22:12 fedora-workstation systemd[1]: Failed to start PostgreSQL database server.
The error is explicit. The data directory has wrong ownership. The fix is a single chown command. Run journalctl first. Read the actual error before guessing.
Reset the failed state and restart
Once you fix the configuration or dependency, the unit stays in the failed state until you explicitly clear it. Systemd caches the failure to prevent immediate restart loops. You need to reset the state, then start it. Here is the exact sequence to clear the failure flag and launch the service cleanly.
# reset-failed clears the failed state for the specified unit.
# It does not stop or start the service. It only removes the failure marker.
sudo systemctl reset-failed postgresql.service
# start launches the unit. Systemd will evaluate dependencies,
# apply overrides, and execute the ExecStart command.
sudo systemctl start postgresql.service
The reset-failed command is often overlooked. Without it, systemctl start will refuse to run and print a warning that the unit is marked as failed. Some administrators use systemctl restart instead, which works, but it also kills the process first. If the process is already dead, restart adds unnecessary overhead. reset-failed followed by start is the cleanest path. Clear the failure flag before you restart. Systemd will not retry a dead unit on its own.
Verify the service is healthy
Check the status. Look at the active state, the main PID, and the recent log lines. Here is how to confirm the service is running and bound to the correct socket or port.
# status shows the unit state, main PID, memory usage, and recent journal lines.
# It combines systemctl and journalctl into one view.
systemctl status postgresql.service
The output contains three critical sections. The top line shows the load state and the active state. loaded means systemd found the unit file. active (running) means the process is alive and healthy. The second line shows the documentation URL. The third line shows the main PID and the memory footprint. The bottom section shows the last five journal entries for that unit. If you see active (exited), the service ran and finished normally. That is correct for one-shot tasks like systemd-tmpfiles-setup.service. It is wrong for daemons that should stay alive. Trust the status output. If it says inactive, the process is not running.
Common pitfalls and what the error looks like
Most service failures fall into three categories. Configuration syntax errors, missing dependencies, and security module denials. Each one leaves a distinct fingerprint in the journal.
Configuration errors usually appear as parsing failures or missing file references. If you edit a service override and forget a space after the = sign, systemd will refuse to load the unit. The journal will show Failed to parse configuration file or Invalid section header. Always validate your overrides before reloading. Run systemd-analyze verify <unit> to catch syntax errors without starting the service.
Missing dependencies show up as timeout errors. Systemd waits for a target or another service before starting the dependent unit. If the dependency never becomes active, the dependent unit times out and fails. The journal prints Dependency failed for <unit> or Job <unit> failed because a timeout was exceeded. Check the Requires= or After= directives in the unit file. Install the missing package or enable the missing service.
Security module denials are the most common silent killer. SELinux blocks access to files, ports, or network namespaces that the service is not allowed to touch. The service exits immediately, but the journal might only show a generic permission error. You need to query the SELinux audit stream. Run journalctl -t setroubleshoot to see the one-line summary. The summary tells you exactly which process was blocked and what file or port it tried to access. Fix the context with restorecon or adjust the policy with semanage. Never disable SELinux to work around a service failure. Edit /etc/ overrides, never /usr/lib/ defaults. Package updates will overwrite your manual changes.
When to use this approach versus other tools
Use journalctl -u when you need to isolate logs for a single unit and filter by boot cycle. Use systemctl reset-failed when a service is stuck in a failed state and refuses to start. Use systemctl edit --full when you need to override package defaults without touching the original unit file. Use systemd-analyze verify when you suspect a dependency loop or invalid unit syntax. Use journalctl -t setroubleshoot when a service exits immediately with a permission error and standard logs show nothing useful. Stay on systemctl status for quick health checks before you attempt a restart.