systemd in practice: writing a service, a timer, and how to debug failures

On Fedora, “make this start at boot” is not a line in /etc/rc.local — it is a unit file. Scheduled jobs work the same way: systemd timers are replacing cron (F45 minimal images do not even ship crontab).

What follows was measured on F45.

Version and default target

systemctl --version | head -2
systemctl get-default

systemd 261 (261.2-1.fc45)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +IPE +SMACK +SECCOMP ...

multi-user.target

A command-line Fedora defaults to multi-user.target (no graphical session); the desktop editions use graphical.target.

How many units are there:

systemctl list-unit-files --type=service --no-pager | tail -1
systemctl list-unit-files --type=timer --no-pager | tail -1

278 unit files listed.
12 unit files listed.

Where unit files go

Location	Written by	Priority	Overwritten on upgrade
/usr/lib/systemd/system/	the package manager	lowest	yes
/run/systemd/system/	generated at runtime	middle	lost on reboot
/etc/systemd/system/	you	highest	no

Put your own units in /etc/systemd/system/. Do not edit files under /usr/lib — upgrades overwrite them and later nobody can tell who changed what.

Writing a service

sudo tee /etc/systemd/system/lab-hello.service >/dev/null <<'EOF'
[Unit]
Description=Lab hello service
After=network-online.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'date >> /var/log/lab-hello.log; echo hello >> /var/log/lab-hello.log'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl start lab-hello.service
systemctl status lab-hello.service --no-pager | head -6

● lab-hello.service - Lab hello service
     Loaded: loaded (/etc/systemd/system/lab-hello.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Tue 2026-10-27 17:41:46 UTC; 9ms ago
 Invocation: 45ec53c7e0f24c72883a6d099d034a05

Three details worth reading carefully:

• disabled — the file exists but you never enabled it, so it will not start at boot;
• active (exited) — the normal state for oneshot with RemainAfterExit=yes: the script finished, the service still counts as active;
• Drop-In — those lines are not yours; the distribution adds a shared timeout policy.

Choosing Type

Type	Counts as “started” when	Suits
simple (default)	the process has been spawned	long-running foreground processes
oneshot	the process has finished	one-off scripts, initialisation
forking	the parent exits	classic daemons that daemonise themselves
notify	the program calls sd_notify	when “actually ready” matters

Caution

For Type=simple, “started successfully” only means the process exists — not that it works. Use Type=notify, or an ExecStartPost= probe, if readiness matters.

Writing a timer (the cron replacement)

A timer needs a pair: a .service that does the work and a .timer that schedules it.

sudo tee /etc/systemd/system/lab-tick.service >/dev/null <<'EOF'
[Unit]
Description=Lab tick
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo tick $(date +%s) >> /var/log/lab-tick.log'
EOF

sudo tee /etc/systemd/system/lab-tick.timer >/dev/null <<'EOF'
[Unit]
Description=Lab tick every minute
[Timer]
OnCalendar=*:0/1
Persistent=true
[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now lab-tick.timer

Created symlink '/etc/systemd/system/timers.target.wants/lab-tick.timer' → '/etc/systemd/system/lab-tick.timer'.

enable --now creates a symlink under timers.target.wants/ — that symlink is what “start at boot” actually means. It is not an entry in some configuration file.

When does it run next:

systemctl list-timers lab-tick.timer --no-pager

NEXT                        LEFT LAST PASSED UNIT           ACTIVATES
Tue 2026-10-27 17:43:00 UTC  29s -         - lab-tick.timer lab-tick.service

The ACTIVATES column names the service it triggers.

Persistent=true means a run missed while the machine was off is caught up on the next boot — something cron’s @reboot cannot do.

OnCalendar forms

Form	Meaning
*:0/1	every minute
hourly / daily / weekly	fixed periods
Mon..Fri 02:00	weekdays at 02:00
*-*-* 03:30:00	every day at 03:30
OnUnitActiveSec=5min	5 minutes after the last run (relative)

Validate an expression without waiting for it:

systemd-analyze calendar "*:0/1"

Debugging a failure

Deliberately failing a service produces:

● lab-fail.service - Lab failing service
     Loaded: loaded (/etc/systemd/system/lab-fail.service; static)
     Active: failed (Result: exit-code) since Tue 2026-10-27 17:43:03 UTC
    Process: 24880 ExecStart=/bin/false (code=exited, status=1/FAILURE)
   Main PID: 24880 (code=exited, status=1/FAILURE)
   Mem peak: 1.2M

The Process: lines are where debugging starts — they give an exit code instead of a vague “failed to start”.

The order to follow:

systemctl list-units --failed          # 1) what failed this boot
systemctl status <unit> --no-pager     # 2) exit code and reason
journalctl -u <unit> -n 50 --no-pager  # 3) the actual log
journalctl -xeu <unit>                 # 4) log with systemd's explanations

  UNIT                  LOAD   ACTIVE SUB    DESCRIPTION
● dnf-makecache.service loaded failed failed dnf5 makecache
● lab-fail.service      loaded failed failed Lab failing service

Tip

journalctl -xeu <unit> is the cheapest single command: -x inserts explanations for systemd state transitions and -e jumps to the end. Much faster than paging through everything.

Changing someone else’s unit: use a drop-in

Never edit the file under /usr/lib. Instead:

sudo systemctl edit <unit>          # creates /etc/systemd/system/<unit>.d/override.conf
sudo systemctl daemon-reload

edit opens an empty file; you write only what you want to override or append:

[Service]
Environment=HTTP_PROXY=http://proxy.internal:3128
TimeoutStartSec=180

Upgrades keep the distribution’s changes and yours survive alongside them.

Slow boot? Measure it

systemd-analyze

Startup finished in 1.502s (kernel) + 2.497s (initrd) + 7.888s (userspace) = 11.888s

The three parts are kernel, initrd and userspace. If userspace dominates, look at blame:

systemd-analyze blame | head -3

11.713s unbound-anchor.service
 3.798s sys-module-fuse.device
 3.763s dev-vport1p1.device

unbound-anchor.service on top is exactly the service that kept DNS resolution failing on this machine — it was stuck waiting to refresh the upstream root key. The “slow boot” and the “weird network” were the same root cause.

Note

systemd-analyze blame lists per-unit durations, but a long unit is not automatically the problem — check whether it is waiting on the network. systemd-analyze critical-chain shows which link in the dependency chain actually stalls.

Everyday commands

Goal	Command
state	systemctl status <unit> --no-pager
start / stop	systemctl start / stop
start at boot	systemctl enable --now <unit>
stop starting at boot	systemctl disable --now <unit>
block entirely	systemctl mask <unit>
list failures	systemctl list-units --failed
show the unit file	systemctl cat <unit>
show effective values	systemctl show <unit> -p <property>
reload after edits	systemctl daemon-reload
timers	systemctl list-timers --all

Related pages

• Backup and restore — scheduling backups with a timer
• firewalld — opening the port a service listens on
• SELinux — the other reason a service gets refused
• Storage and filesystems — space for logs and data
• Error lookup — Unit could not be found, is masked, Access denied
• Cheat sheet — every systemd command
• Official: systemd.service, systemd.timer