Get the journal to survive reboots so you can read a crash's logs
Everything of substance in this record held, on Omarchy 4 and on plain Arch, and the only defect is one command in the danger field that cannot run on Omarchy. Confirmed on this machine (omarchy 4.0.2-1, systemd 261.2-1): systemd-analyze cat-config systemd/journald.conf shows the package-stock file only, with #Storage=persistent as the compile-time default, and /etc/systemd/journald.conf.d does not exist, so Omarchy overrides nothing about journald and the record's drop-in advice neither duplicates nor fights it. pacman -Qo /var/log/journal returns "owned by systemd 261.2-1", and pacman -Ql systemd lists /var/log/journal/, confirming the cause's claim that the directory ships with the package and that its absence means somebody removed it. The journal is persistent here, which is the premise the record depends on: journalctl --header reports a file under /var/log/journal/2e5dbe305d304555aa01d578380cb2e8/, journalctl --list-boots lists three boots, journalctl --disk-usage reports 679.8M in the file system, and /run/log/journal is empty. The quoted error text is exact rather than paraphrased: "Specifying boot ID or boot offset has no effect, no persistent journal was found." appears at line 82 of the cited journalctl-util.c and byte-identically in strings /usr/bin/journalctl on systemd 261.2-1, so it has not drifted. The recovery sequence is right, and its choice of systemd-tmpfiles --create --prefix /var/log/journal over a hand-written chmod is better than the record claims, because on this btrfs root that prefix picks up both /usr/lib/tmpfiles.d/systemd.conf line 28 (z /var/log/journal 2755 root systemd-journal) and /usr/lib/tmpfiles.d/journal-nocow.conf line 25 (h /var/log/journal - - - - +C), restoring the NOCOW attribute as well as the mode. --create, --prefix, --flush, --list-boots and --disk-usage all exist on the installed versions, checked against systemd-tmpfiles --help and journalctl --help. findmnt /var/log is the right tmpfs check and on Omarchy 4 it usefully shows the @log subvolume the installer creates. All four cited URLs resolved: journald.conf(5) and journalctl(1) both returned real Arch manual pages, the raw journalctl-util.c returned HTTP 200, and Arch's Systemd/Journal page line 141 supports the Storage=persistent default while line 145 supports the 10% and 4 GiB soft cap quoted in the danger. The local man 5 journald.conf confirms the auto-is-the-trap explanation. Nothing was removed from sources. The one correction: the danger says an oversized journal "breaks pacman -Syu", and on Omarchy 4 that command never gets as far as a disk check, because /usr/share/libalpm/hooks/00-omarchy-update-guard.hook runs omarchy-update-pacman-guard with AbortOnFail and that script aborts whenever both S and u are present, identical at upstream tag v4.0.3. The real Omarchy symptom is omarchy-update-requires-free-space printing "You need at least 10 GiB free to safely update Omarchy." I rewrote the danger only and left symptom, cause, fix and verify untouched, since a rewrite of correct text would itself be a defect. Severity and frequency left alone. Not exercised: I have no sudo, so I did not delete /var/log/journal, did not run the recovery sequence, did not restart journald, and did not reboot to prove journalctl -b -1 comes back. I also never observed the volatile failure state itself, only the healthy state it contrasts with.
Persistent journals grow. The default cap is 10% of the filesystem with a soft cap at 4 GiB, which is exactly how a small root partition fills up and starts blocking updates. On plain Arch that shows up as pacman -Syu failing on disk space. On Omarchy 4 you see omarchy update refuse with You need at least 10 GiB free to safely update Omarchy. instead, and pacman -Syu is no test either way, because Omarchy's update guard hook aborts it whatever the disk looks like. Set SystemMaxUse= at the same time as you enable persistence.
Do not reclaim space by deleting /var/log/journal itself, which is what turns persistence off in the first place. Trim the files and leave the directory in place:
sudo journalctl --rotate
sudo journalctl --vacuum-size=200M
Symptom
After a freeze or hard reboot, journalctl -b -1 prints only:
Specifying boot ID or boot offset has no effect, no persistent journal was found.
and shows the current boot instead. journalctl --list-boots lists exactly one boot, and journalctl --disk-usage reports usage under /run/log/journal rather than /var/log/journal. So there is no way to see what happened before the crash.
Cause
journald is only writing to the in-memory runtime journal, which is discarded on every boot. On Arch the default is Storage=persistent and /var/log/journal/ ships with the systemd package, so this state almost always means the directory was deleted (often while reclaiming disk space, or by rm -rf /var/log/journal), or Storage= was set to volatile/auto in a config drop-in, or /var/log is on a tmpfs.
Fix
Check where you stand:
journalctl --list-boots
journalctl --disk-usage
ls -ld /var/log/journal
systemd-analyze cat-config systemd/journald.conf | grep -i -E '^Storage|^#Storage'
findmnt /var/log
Re-create the directory with the right ownership and mode, then tell journald to flush:
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald.service
sudo journalctl --flush
Be explicit about the mode so nothing can silently fall back again. Use a drop-in rather than editing the packaged journald.conf:
sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/00-persistent.conf >/dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=500M
SystemMaxFileSize=50M
EOF
sudo systemctl restart systemd-journald.service
Storage=auto is the trap: it behaves like volatile whenever /var/log/journal does not exist, so the mere absence of the directory silently disables persistence. Storage=persistent creates it.
Reboot once, then confirm the previous boot is readable:
journalctl --list-boots
journalctl -b -1 -p err..alert --no-pager
journalctl -b -1 -k --no-pager | tail -50
Useful follow-ups for an actual crash post-mortem:
journalctl -b -1 -u <suspect-unit> --no-pager
journalctl -b -1 --since '10 min ago' --until 'now' # relative to that boot's end
coredumpctl list
Verify
ls /var/log/journal/ contains a machine-id directory with .journal files, journalctl --disk-usage reports usage under /var/log/journal, and after a reboot journalctl --list-boots lists at least two boots and journalctl -b -1 shows real log lines.
Sources
- https://man.archlinux.org/man/journald.conf.5.en
- https://man.archlinux.org/man/journalctl.1.en
- https://wiki.archlinux.org/title/Systemd/Journal
- https://raw.githubusercontent.com/systemd/systemd/main/src/journal/journalctl-util.c
- https://github.com/omacom/omarchy/blob/v4.0.3/bin/omarchy-update-pacman-guard
- https://github.com/omacom/omarchy/blob/v4.0.3/bin/omarchy-update-requires-free-space