OPINIONATED OMARCHY
TROUBLESHOOTING CORPUS
790
RECORDS
790
AUDITED
0
UNCHECKED
2388
SOURCES
LIVE
apps-services medium common DISPUTE

Recover a service stuck in "Start request repeated too quickly"

omarchyarchendeavouroscachyosmanjarosystemdlaptopdesktop
CORRECTED An auditor found the fix wrong or incomplete and rewrote it.
AUDIT NOTE

Checked against man systemd.unit(5) and man systemctl(1) on this machine (systemd 261.2-1), systemd v261's own parser table, and live units here. What held: both quoted messages are real in v261 (log_unit_warning(u, "Start request repeated too quickly.") at src/core/unit.c:1882, and [SERVICE_FAILURE_START_LIMIT_HIT] = "start-limit-hit" at src/core/service.c:6296). The 5-starts-in-10s default is confirmed twice, from the commented #DefaultStartLimitIntervalSec=10s / #DefaultStartLimitBurst=5 at /etc/systemd/system.conf:56-57 and from live StartLimitIntervalUSec=10s StartLimitBurst=5 on sshd.service. Reset-failed flushing the start rate limit counter, StartLimit* being documented [Unit] keys, 0 disabling rate limiting, and drop-ins being preferred over editing packaged units are all confirmed in the man pages. The framing that the rate limit is the symptom and not the fault is right and is kept verbatim. Three defects, all confirmed on this machine. (1) systemctl show -p StartLimitIntervalSec and -p RestartSec return NOTHING: the D-Bus properties are StartLimitIntervalUSec and RestartUSec, and systemctl show -p silently skips a name it does not know, so the record's diagnostic line lost two of five properties and its verify step could not show the interval the reader had just set. That is the worst of the three because it sat in verify. (2) The claim that systemd warns about StartLimit* in [Service] is true for exactly one key. systemd v261's src/core/load-fragment-gperf.gperf.in still carries Service.StartLimitBurst, Service.StartLimitInterval and Service.StartLimitAction as compat entries writing into the same Unit fields, and has no Service.StartLimitIntervalSec. systemd-analyze verify on a throwaway file in /tmp confirmed it: only StartLimitIntervalSec= in [Service] warns, while StartLimitBurst=, StartLimitInterval= and StartLimitAction= there produce no message and are applied. So the record promised feedback that does not arrive for the key most likely to be misplaced, and the real trap is a burst applied over the default 10s window. (3) man systemctl says systemctl edit reloads configuration when the editor exits, so the record's extra daemon-reload was redundant. The danger was half wrong and contradicted the record's own fix: reset-failed clears the recorded exit status but does not touch the journal, so "capture the journal output before resetting" was unnecessary alarm, while the fix itself correctly ran journalctl after resetting. The unbounded-restart-loop half of the danger is sound and is kept. Added an Omarchy branch because the record claims applies_to: omarchy and had no Omarchy content: six user units under /usr/share/omarchy/default/systemd/user/ set Restart= with RestartSec=2 or 5 and set no StartLimit*, and five of them are loaded here showing StartLimitIntervalUSec=10s StartLimitBurst=5. NOT exercised: I did not trip a real start limit, edit, reset or restart any unit on this machine, so the reset-failed-then-start recovery sequence is confirmed from the man page and the parser source rather than by running it. severity medium and frequency common left alone, both look right for a generic systemd shape.

The cause above was not rewritten and may still contain the error described. The fix below is the corrected version.
RISK

systemctl reset-failed clears the recorded exit code and status that systemctl status was reporting, and resets the start rate limit counter and the service restart counter, so the previous failure stops being shown once the unit is started again. It does not touch the journal: journalctl -u foo.service -b still holds every line from the failed attempts, which is why reading the journal after resetting is safe and is the order given above.

Raising StartLimitBurst or setting StartLimitIntervalSec=0 on a service that crashes on startup turns it into an unbounded restart loop that can spin a CPU core and flood the journal. The start limiter is the one thing stopping that, so widen the interval instead of removing it, and only after the underlying failure is understood.

Symptom

A service dies and then refuses to come back at all:


foo.service: Start request repeated too quickly.
foo.service: Failed with result 'start-limit-hit'.
Failed to start foo.service.

systemctl restart foo.service returns the same thing immediately, without even trying to run the binary. The journal shows the service starting and exiting five times in a few seconds just before this.

Cause

With Restart=always/on-failure, a service that fails instantly gets restarted instantly, and systemd's start rate limiter cuts in: more than StartLimitBurst starts inside StartLimitIntervalSec (5 in 10s by default on Arch) and the unit is refused any further start until the interval passes. The rate limit is the *symptom*. The real failure is whatever made the service exit in the first place, and once the limiter trips, systemctl restart no longer tells you anything about it.

Fix

Clear the counter so you can try again and see the real error:


sudo systemctl reset-failed foo.service
sudo systemctl start foo.service
sudo journalctl -u foo.service -b --no-pager -n 100

For a user unit:


systemctl --user reset-failed foo.service
systemctl --user start foo.service
journalctl --user -u foo.service -b --no-pager -n 100

Find out what it actually runs and with what limits before guessing. Mind the property names: systemctl show reports the interval and the restart delay as StartLimitIntervalUSec and RestartUSec, not with the ...Sec spellings you write in a unit file. systemctl show -p ignores a property name it does not recognise and prints nothing for it instead of erroring, so asking for -p StartLimitIntervalSec returns an empty result and looks like the unit has no limit:


systemctl cat foo.service
systemctl show foo.service -p ExecStart -p Restart -p RestartUSec \
  -p StartLimitBurst -p StartLimitIntervalUSec -p StartLimitAction

Fix the underlying failure. If the service is legitimately expected to flap (a network-dependent daemon, a tunnel), widen the window and slow the restarts down so it backs off instead of burning through the limit, with a drop-in rather than by editing the packaged unit:


sudo systemctl edit foo.service

[Unit]
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=15

systemctl edit reloads the manager itself once the editor exits, equivalent to systemctl daemon-reload, so no separate reload is needed:


sudo systemctl restart foo.service

That allows 5 starts per 5 minutes with 15 seconds between attempts. To disable rate limiting entirely (rarely a good idea) set StartLimitIntervalSec=0.

StartLimit* belong in [Unit], and putting them in [Service] fails with almost no feedback. Measured with systemd-analyze verify on systemd 261:


$ systemd-analyze verify ./t.service        # StartLimitIntervalSec in [Service]
./t.service:7: Unknown key 'StartLimitIntervalSec' in section [Service], ignoring.

$ systemd-analyze verify ./t.service        # StartLimitBurst in [Service]
                                            # no output at all

StartLimitBurst=, StartLimitInterval= (the old spelling, no Sec) and StartLimitAction= are still accepted inside [Service] as backward-compatible aliases and are applied silently, while StartLimitIntervalSec= is not a [Service] key at all and is dropped with a warning. So the whole block pasted into [Service] leaves you with the burst you asked for over the default 10 second window, and only one line in the journal hints at it. Put them in [Unit] and confirm with the systemctl show line above.

The defaults live in /etc/systemd/system.conf as DefaultStartLimitIntervalSec= and DefaultStartLimitBurst=, and in /etc/systemd/user.conf for user units. Both ship commented out at their stock values of 10s and 5.

On Omarchy this is a live shape rather than a hypothetical. Six of the user units Omarchy ships under /usr/share/omarchy/default/systemd/user/ set Restart= with a short RestartSec= and none of them set StartLimit*, so they all inherit 5 starts per 10 seconds:


systemctl --user show omarchy-fcitx5.service omarchy-sleep-lock.service \
  omarchy-crash-watch.service omarchy-tailscale-receive.service bt-agent.service \
  -p Restart -p RestartUSec -p StartLimitBurst -p StartLimitIntervalUSec

With RestartSec=2 against a 10 second window, a binary that fails immediately exhausts the burst in roughly ten seconds, which is why one of these units going bad shows up as start-limit-hit rather than as a visible crash loop.

Verify

systemctl status foo.service shows active (running) and the journal no longer contains start-limit-hit. Confirm the drop-in took effect with the property names systemctl show actually uses, because the ...Sec spellings return nothing:


systemctl show foo.service -p StartLimitBurst -p StartLimitIntervalUSec -p RestartUSec

On a system unit with a 300 second window and a 15 second restart delay that prints StartLimitBurst=5, StartLimitIntervalUSec=5min and RestartUSec=15s. systemctl cat foo.service should also list your drop-in file under the unit.

Sources

unit-start-limit-hit-restart-loop