How to Set Up Health Checks and Auto-Restart for Services

Services fail — a memory spike, an unhandled exception, a dependency hiccup. The difference between a blip and an outage is whether the service comes back on its own or waits for you to notice. This guide sets up automatic recovery with systemd's built-in restart policies, watchdogs, and health checks, so your apps self-heal.

The foundation: systemd Restart policies

If your service runs under systemd (it should), a few directives make it restart automatically on failure. In your unit file (e.g. /etc/systemd/system/myapp.service):

[Service]
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60
User=appuser
EnvironmentFile=/etc/myapp/env

• Restart=on-failure restarts if the process exits non-zero or is killed.
• RestartSec=5 waits 5 seconds between attempts, avoiding a tight crash loop.
• StartLimitBurst/IntervalSec stop endless restarts if something is fundamentally broken.

Use Restart=always if you want restarts even on clean exits (e.g. a worker that shouldn't ever stop). Load secrets via EnvironmentFile as in manage secrets and environment variables.

Apply and verify

sudo systemctl daemon-reload
sudo systemctl enable --now myapp
sudo systemctl status myapp

Test it: kill the process and watch systemd bring it back:

sudo pkill -f myapp
journalctl -u myapp -f

{{SCREENSHOT}}

Restarting on hangs, not just crashes

A process can be "running" but frozen — restart-on-failure won't catch that because the process hasn't exited. Two answers:

Systemd watchdog

If your app supports sd_notify, systemd can restart it when it stops sending heartbeats:

[Service]
WatchdogSec=30
Restart=on-watchdog

The app must ping the watchdog within 30s or systemd kills and restarts it.

External health-check timer

For apps that can't do sd_notify, run a periodic check. Create a script that probes the app's health endpoint and restarts it on failure:

#!/usr/bin/env bash
if ! curl -fsS --max-time 5 http://localhost:3000/health > /dev/null; then
  systemctl restart myapp
  logger "myapp health check failed — restarted"
fi

Run the check on a schedule with systemd timers

Systemd timers are cleaner than cron for this. Create healthcheck.service (Type=oneshot running the script) and healthcheck.timer:

[Timer]
OnBootSec=1min
OnUnitActiveSec=1min

[Install]
WantedBy=timers.target

Enable it:

sudo systemctl enable --now healthcheck.timer

Now the endpoint is checked every minute and the service self-heals within ~60 seconds.

Add a real health endpoint

Auto-restart is only as good as your health signal. A good /health endpoint checks the things that actually matter — database connectivity, critical dependencies — not just "the web server responds." Return non-200 when the app can't do its job.

Don't forget alerting

Auto-restart hides transient failures, which is good — but a service restarting every minute is a problem you need to know about. Log restarts and surface them through monitoring and journalctl, so recovery buys you time without hiding chronic issues. Game servers benefit from the same pattern — see automate game server restarts.

Get the startup order right

Auto-restart is only half the reliability story — services also need to start in the correct sequence, especially after a reboot. If your app starts before its database is ready, it'll crash-loop needlessly until the restart policy happens to catch it running late. systemd's dependency directives express these relationships explicitly:

[Unit]
Description=My App
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

• After= controls *ordering* — start this unit after those are up.
• Requires= creates a hard dependency — if the database is stopped, stop this too.
• Wants= is a softer version that doesn't fail your service if the dependency fails.

For a service that genuinely needs to wait for a dependency to be *usable* (not just started), a short startup health check in an ExecStartPre= line — polling the database until it accepts connections — prevents the crash-loop entirely and makes reboots clean.

Test failure, don't just hope

The reliability equivalent of an untested backup is an untested restart policy. Deliberately break things in a safe window and confirm recovery works as designed:

sudo systemctl kill --signal=SIGKILL myapp    # simulate a hard crash
sudo reboot                                    # confirm clean startup ordering

Watch journalctl -u myapp -f and verify the service comes back, in the right order, without manual help. It's far better to discover a misconfigured dependency or a too-aggressive start limit during a planned test than during a real 3am incident. Chaos-testing your own recovery — even this lightly — is what turns "I think it'll restart" into "I've watched it restart." Pair the auto-restart with alerting so silent flapping still reaches you, and you have infrastructure that genuinely heals itself while keeping you honest about why it needed to.

FAQ

Restart=on-failure vs always — which should I use?

Use on-failure for most services (restart only on crashes). Use always for processes that should never stop, even on a clean exit, like a persistent worker.

Why does my service stop restarting after a few tries?

The start limit kicked in — systemd stops restarting after too many failures in a window to avoid a crash loop. Reset with systemctl reset-failed <service> and fix the root cause.

How do I catch a frozen (not crashed) process?

Restart-on-failure won't help since the process is still alive. Use a systemd watchdog (sd_notify) or an external health-check timer that probes a /health endpoint.

Should I still alert if things auto-restart?

Yes. Auto-restart handles blips, but frequent restarts signal a real problem. Log every restart and alert on the pattern so recovery doesn't mask chronic failures.

Self-healing service checklist

A service that recovers on its own comes from combining a few systemd features:

• Set Restart=on-failure (or always) with a sensible RestartSec.
• Bound the restarts with StartLimitBurst/StartLimitIntervalSec to avoid a crash loop.
• Catch hangs, not just crashes, with a systemd watchdog (sd_notify) or an external health-check timer.
• Expose a real /health endpoint that checks dependencies, not just "the process is up".
• Get startup ordering right with After=, Requires=, and Wants= so services start in sequence.
• Alert on restarts so silent flapping still reaches you.
• Test failure deliberately — kill the process and reboot to confirm recovery works.

The mindset shift is treating reliability like backups: it isn't real until you've tested it. Kill a process in a safe window and watch it come back; reboot and confirm the ordering holds. It's far better to discover a misconfigured dependency or an over-aggressive start limit during a planned test than during a real incident. Get this right and routine failures become invisible blips that resolve themselves, while genuine chronic problems still surface through your alerts rather than hiding behind constant restarts.

Self-healing services turn 2am pages into next-morning tickets. Build reliable infrastructure on a VPS you fully control — see Nxeon VPS hosting and Nxeon for developers.