OPINIONATED OMARCHY
TROUBLESHOOTING CORPUS
790
RECORDS
790
AUDITED
0
UNCHECKED
2388
SOURCES
LIVE
apps-services low common DISPUTE

Stop journald silently dropping a chatty service's log lines

omarchyarchendeavouroscachyosmanjarosystemdlaptopdesktop
CORRECTED An auditor found the fix wrong or incomplete and rewrote it.
AUDIT NOTE

Checked on this workstation (omarchy 4.0.2-1, systemd 261.2-1, kernel 7.1.9) and against systemd's own source and man pages. Confirmed on this machine that Omarchy 4 sets nothing for journald: /etc/systemd/journald.conf.d/ and /usr/lib/systemd/journald.conf.d/ do not exist, systemctl cat systemd-journald shows no drop-in, /etc/systemd/journald.conf is owned by systemd 261.2-1 with #RateLimitIntervalSec=30s and #RateLimitBurst=10000 both commented, and grep -rniE 'journal' /usr/share/omarchy/ matches only an unrelated skill document and a reserved-usernames list. So the record does not tell the reader to change something Omarchy already sets, and it needs no Omarchy-versus-Arch branch. The core mechanics all held at systemd 261: man journald.conf here still says rate limiting is per-service, "Defaults to 10000 messages in 30s", "To turn off any kind of rate limiting, set either value to 0", and that the effective limit is multiplied by a base 2 logarithm factor with the table topping the excerpt out at 1 TB, and that LogRateLimitIntervalSec=/LogRateLimitBurst= in systemd.exec(5) override it. Source agrees: src/journal/journald-rate-limit.c has if (rl_interval == 0 || rl_burst == 0) return 1; and burst_modulate() computing burst * (log2u64(available) - 16) / 4, and src/journal/journald-context.c lines 187 to 190 seed the per-client limits from the global config and flag when a unit overrides them. Four corrections. First, the symptom quoted a message format that systemd has not produced since version 240: src/journal/journald-manager.c line 1294 logs "Suppressed %i messages from %s" with c->unit, which journald-context.c line 315 fills from cg_path_get_unit(), and that helper skips the slices and returns the bare unit name (src/basic/cgroup-util.c, cg_path_get_unit_full). So the line reads "from nginx.service", not "from /system.slice/nginx.service". I kept the old form as a labelled aside so a reader who found it in older advice recognises it. Second, the fix said "bypass the journal entirely and watch the process directly" and then gave a journalctl -f command, which is still the journal and still rate-limited, so it did not do what its own sentence promised. I kept the command with an honest label and added the real bypass: man systemd.exec states outright that "if you connect a service's stderr directly to a file via StandardOutput=file:... or a similar setting, the rate limiting will not be applied to messages written that way (but it will be enforced for messages generated via syslog(3) and similar functions)", so I gave the append: drop-in plus that caveat. StandardError= accepting the same values as StandardOutput= is confirmed in the same man page. Third, the fix claimed "the multiplier only reaches its maximum with tens of GB available". There is no maximum: burst_modulate() grows with log2 of the free space without a ceiling, giving 4x at 4 GB, 5x at 64 GB and 6x at 1 TB, so tens of GB is only 5x and not a plateau. I replaced the clause with the actual figures. Fourth, the danger said nothing about journalctl --vacuum-size=1G permanently deleting archived journal files, which is a real data loss in a record whose whole subject is missing log lines. I added it. Two things I verified rather than corrected: journalctl --rotate before --vacuum-size is necessary and the record already had the order right, because man journalctl says vacuum "removes the oldest archived journal files" and the Arch wiki Systemd/Journal page says files "must have been rotated out and made inactive before they can be trimmed". The second is that systemctl edit already reloads configuration, so the record's extra daemon-reload was harmless, and I dropped it only because I was rewriting the block. One source-scope note without a removal: wiki.archlinux.org/title/Systemd/Journal resolves (HTTP 200) but says nothing about rate limiting anywhere, so it supports the drop-in directory pattern, the rotate-then-vacuum ordering and SystemMaxUse= but not the rate-limit claims it sat next to. man.archlinux.org/man/journald.conf.5.en resolves and carries the text quoted above. I left severity: low and frequency: common alone rather than manufacture a change. Not exercised: I have no sudo, so I could not write a drop-in, run systemctl edit, restart journald, or trigger a burst. journalctl -b | grep -c 'Suppressed' returns 0 on this boot, so I never saw a real suppression line on this machine and the message format finding rests on source reading, not observation.

The cause above was rewritten on 2026-09-11 to match this note; the fix was corrected by the audit itself.
RISK

Turning rate limiting off system-wide is how a single looping service fills the root filesystem in minutes and takes down pacman, Docker and anything else that needs to write. Prefer the per-unit LogRateLimit* override, keep SystemMaxUse= set, and revert the change once you have the logs you needed. journalctl --rotate followed by journalctl --vacuum-size=1G permanently deletes archived journal files, which can include the record of an earlier incident you have not read yet, so check journalctl --disk-usage and export anything you still need before vacuuming rather than after.

Symptom

Logs from one service have holes in them. The interesting lines are simply missing, and the journal contains entries from journald itself like:


systemd-journald[318]: Suppressed 4079 messages from nginx.service

Debugging a crash is impossible because the burst right before the failure is exactly the part that was dropped. Advice written for systemd older than version 240 quotes the control group path instead of the unit name (Suppressed 4079 messages from /system.slice/nginx.service) and is describing the same thing.

Cause

journald rate-limits per service: more than RateLimitBurst messages inside RateLimitIntervalSec (10000 in 30s by default) and everything else in that window is discarded, with only a summary line kept. The count is held in five separate buckets per service, grouped by priority, so a flood of info lines does not on its own suppress that service's err lines. The effective burst is additionally scaled by how much free disk space the journal has, using the base 2 logarithm of the space available, so a nearly-full disk gets roughly the nominal limit while a disk with tens of GB free gets five times it. That is why a nearly-full disk drops far more than the nominal figure suggests.

Fix

See how much is being dropped and by whom:


journalctl -b | grep -i 'Suppressed .* messages from'
journalctl -b -u systemd-journald --no-pager
journalctl --disk-usage
df -h /var

Omarchy 4 ships no journald configuration of its own. There is no drop-in under /etc/systemd/journald.conf.d/ or /usr/lib/systemd/journald.conf.d/, and /etc/systemd/journald.conf is systemd's stock file with every setting commented out, so what is in force are the compile time defaults. Confirm that before changing anything:


systemd-analyze cat-config systemd/journald.conf

Raise (or disable) the limit for the whole system with a drop-in:


sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/10-ratelimit.conf >/dev/null <<'EOF'
[Journal]
RateLimitIntervalSec=30s
RateLimitBurst=50000
EOF
sudo systemctl restart systemd-journald.service

Better: lift it only for the one noisy unit, using its own per-service limits, so the rest of the system stays protected:


sudo systemctl edit nginx.service

[Service]
LogRateLimitIntervalSec=0
LogRateLimitBurst=0

sudo systemctl restart nginx.service

LogRateLimit* in the unit override the global RateLimit* for that service, and 0 for either value turns rate limiting off for it. systemctl edit reloads the configuration itself, so no separate daemon-reload is needed.

Because the effective burst is multiplied by a factor derived from the free space for the journal, free space up as well. The factor is the base 2 logarithm of the space available, so it keeps climbing rather than topping out: roughly 4 times the nominal burst at 4 GB free, 5 times at 64 GB, 6 times at 1 TB. Vacuuming only touches journal files that have already been rotated out, so rotate first:


sudo journalctl --rotate
sudo journalctl --vacuum-size=1G

To follow a unit live while you reproduce the problem:


sudo journalctl -u nginx.service -f -o short-precise

That is still the journal, so it is still subject to the rate limit. It shows the suppression line as it happens, which confirms the diagnosis, but it does not recover the dropped messages.

To take one unit out of the journal for a debugging session, send its output straight to a file. Rate limiting is not applied to what is written that way:


sudo systemctl edit nginx.service

[Service]
StandardOutput=append:/var/log/nginx-debug.log
StandardError=append:/var/log/nginx-debug.log

sudo systemctl restart nginx.service
tail -f /var/log/nginx-debug.log

That covers only what the process writes to stdout and stderr. A service that logs through syslog(3) or sd_journal_send() still goes through journald and is still rate-limited, so for those the per-unit LogRateLimit* override above is the only route. Remove whichever override you added once you have the logs you needed.

Verify

Reproduce the burst: journalctl -b | grep -c 'Suppressed .* messages' stays at its previous value (no new suppression lines) and the previously missing lines now appear in journalctl -u <unit>.

Sources

journald-rate-limit-suppressed-messages