OPINIONATED OMARCHY
TROUBLESHOOTING CORPUS
790
RECORDS
790
AUDITED
0
UNCHECKED
2388
SOURCES
LIVE
apps-services high occasional DISPUTE

Fix a scheduled restic/borg backup that skips runs and then fails on a stale lock

omarchyarchendeavouroscachyosmanjarosystemdlaptopdesktop
CORRECTED An auditor found the fix wrong or incomplete and rewrote it.
AUDIT NOTE

Checked on this workstation (omarchy 4.0.2-1, systemd 261.2-1, kernel 7.1.9) and against primary sources. Three findings. First, the cause was wrong about suspend: man systemd.timer says Persistent= is "useful to catch up on missed runs of the service when the system was powered down", and systemd's own src/core/timer.c arms t->wake_system ? CLOCK_REALTIME_ALARM : CLOCK_REALTIME, so a realtime timer that elapses during suspend fires shortly after resume without Persistent= at all. Only power-off needs the catch-up. Second, the fix omitted WakeSystem=true, which is the only setting that makes the timer actually run at 03:00 on the suspended laptop the symptom describes, so the record did not fully answer its own scenario. I added it with the Arch wiki Systemd/Timers caveat that it can fail with "Failed to enter waiting state: Operation not supported". Third, confirmed on this machine that neither tool ships with Omarchy: pacman -Q restic borg reports both not found, and grep -rniE 'restic|borg' /usr/share/omarchy/ matches nothing, while /usr/share/omarchy/install/omarchy-other.packages carries snapper and limine-snapper-sync instead (both installed here, snapper 0.13.1-3 and limine-snapper-sync 1.31.0-1). Both tools are in Arch extra (restic 0.19.1-1, borg 1.4.5-1), so the record stays correct for an Omarchy user who installs one, and applies_to keeps omarchy, but I added the install step and the snapper note and dropped frequency from common to occasional, because the problem needs a third-party tool Omarchy does not ship plus a hand-written system timer. severity: high stays: a backup that silently has not run for weeks is exactly that bad. What held, and where: the pgrep concurrency guard, restic unlock before the backup, restic check --with-cache --read-data-subset=5G, chmod 744 and chmod 700 are all verbatim from the Arch wiki Restic page, and --read-data-subset accepting a size is confirmed in restic's own "Working with repositories" page. restic unlock removing only stale locks and --remove-all removing all is confirmed in cmd/restic/cmd_unlock.go ("remove all locks, even non-stale ones"), and the staleness rule is internal/repository/lock_file.go lines 246 to 248: "stale if the timestamp is older than 30 minutes or if it was created on the current machine and the process isn't alive any more". That makes plain unlock sufficient for the symptom, which I folded into the fix and the danger to sharpen the gate on --remove-all. The symptom's quoted restic error is current: lock_file.go line 378 formats exactly "PID %d on %s by %s (UID %d, GID %d)". The danger held and is now backed by borg's own break-lock page, "Please use with care and only when no borg process (on any machine) is trying to access the cache or the repository", and by restic's troubleshooting page confirming an interrupted backup does not damage the repository but may need a manual unlock. ExecCondition= exiting 1 through 254 not marking the unit failed, and RequiresMountsFor= adding Requires= and After=, are both confirmed in man systemd.service and man systemd.unit here. Bare systemd-inhibit in ExecStart= resolves, confirmed in man systemd.service: a first argument without slashes is searched in a fixed path including /usr/bin/, and /usr/bin/systemd-inhibit exists here. Two corrections to the previous audit note rather than to the record: it credited the Arch wiki Restic page with the Persistent=true and RandomizedDelaySec pattern, but that page's timer is monotonic (OnBootSec=5min, OnUnitActiveSec=15min), and the pattern comes from Systemd/Timers and man systemd.timer. The cited wiki.archlinux.org/title/Borg_backup page resolves but contains no mention of locks at all, so it did not support the borg claim it was cited for. I kept it rather than removing it, because it does support the borg install step I added, and added borg's own break-lock page for the lock claim. Not exercised: I have no sudo, so I could not install restic, create a repository, run a backup, break a lock, start a timer, or test WakeSystem=true on real firmware. The claim that a root system service can take a block inhibitor is from source reading only: /usr/share/polkit-1/actions/org.freedesktop.login1.policy rates inhibit-block-sleep and inhibit-block-shutdown as auth_admin_keep, but systemd's src/shared/bus-polkit.c skips the polkit query for a privileged caller ("Don't query PK if client is privileged"), and a root service holds CAP_SYS_ADMIN, so the Arch wiki's note that systemd-inhibit "is only available when running in a user session" is misleading for a root system unit. I could not confirm that by running it.

The cause above was rewritten on 2026-09-11 to match this note; the fix was corrected by the audit itself.
RISK

restic unlock --remove-all and borg break-lock remove locks belonging to processes that may still be running. Borg's own documentation says to use break-lock with care and only when no borg process, on any machine, is trying to access the cache or the repository. Doing either while a backup or a prune is genuinely in progress can corrupt the repository. Plain restic unlock is the safe form and clears the usual stale lock, so treat --remove-all as a last resort and confirm nothing is running on any host that touches the repo first. restic forget --prune permanently deletes snapshots: test your retention flags with restic forget --dry-run before putting them in a timer. restic backup / with a missing or empty exclude file walks /proc, /sys and /dev, so write the exclude list before the first scheduled run. And note that an automated backup necessarily has the repository password available to root in plain text (the --password-command script), so protect it with chmod 700 and remember that anyone with root can read your backups.

Symptom

A nightly backup timer set for 03:00 has not run in days on a laptop. systemctl list-timers shows a NEXT time but LAST is n/a or weeks old. When it does eventually run it fails with restic's


Fatal: unable to create lock in backend: repository is already locked exclusively by PID 1234 on host by user (UID 0, GID 0)

or borg's Failed to create/acquire the lock, and every subsequent run fails the same way.

Cause

Two compounding problems. The schedule: a realtime OnCalendar= timer without Persistent=true does not record its last trigger time, so any occurrence that fell while the machine was powered off is dropped instead of caught up. Suspend behaves differently. A realtime timer that elapses during suspend does fire shortly after resume, because systemd arms CLOCK_REALTIME for it unless WakeSystem=true is set, so a laptop suspended at 03:00 gets a late backup on the next resume while a laptop shut down at 03:00 gets none at all. The lock: when a run is cut short by suspend, shutdown, an OOM kill or an unplugged drive, the repository lock it created is never released, and every later run is refused by a lock whose owning process is long gone. restic counts a lock stale once its creating process is dead on the same host, or once any lock is more than 30 minutes old without a refresh, which is why restic unlock clears the usual case and --remove-all is needed only for a live process or a recent lock from another host.

Fix

Neither restic nor borg is part of Omarchy. Install the one you want first, after an update so the package databases are current:


omarchy update
sudo pacman -S --needed restic     # or: sudo pacman -S --needed borg

pacman -S <pkg> on its own is not blocked by Omarchy's ALPM guard, which aborts only when the pacman command line carries both a sync and a sysupgrade flag. Do not reach for pacman -Sy restic, which is a partial upgrade. Omarchy's own packaged snapshot tooling is snapper with limine-snapper-sync, which takes btrfs snapshots on the same disk and is not a replacement for a backup repository on separate storage.

Make the schedule catch up after downtime:


# /etc/systemd/system/restic-backup.timer
[Unit]
Description=Timer for full system backups

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
RandomizedDelaySec=15m
Unit=restic-backup.service

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer
systemctl list-timers restic-backup.timer
systemd-analyze calendar "*-*-* 03:00:00"

Persistent=true stores the last trigger time on disk and runs the job immediately after boot if the scheduled time passed while the machine was powered down. It does not cover suspend and does not need to: a realtime timer that elapses while the machine is suspended fires shortly after resume on its own.

If you want the backup to run at 03:00 on a laptop that is suspended at 03:00, rather than whenever you next open the lid, add WakeSystem=true to the [Timer] section:


[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
RandomizedDelaySec=15m
WakeSystem=true
Unit=restic-backup.service

That switches the timer to CLOCK_REALTIME_ALARM and resumes the machine. It needs hardware and firmware support, and a timer that cannot get it fails to start with Failed to enter waiting state: Operation not supported. It also does not suspend the machine again once the backup finishes.

Stop the run being cut short in the middle, and clear a stale lock before starting. This is what the recommended wrapper script does:


# /etc/systemd/system/restic-backup.service
[Unit]
Description=Backup system

[Service]
Type=oneshot
ExecStart=systemd-inhibit --what=sleep:shutdown --why="restic backup" /usr/local/bin/restic-backup

#!/bin/bash
# /usr/local/bin/restic-backup
if pgrep -f 'restic backup' > /dev/null; then
  echo 'restic is already running...' 1>&2
  exit 0
fi

set -e
export RESTIC_REPOSITORY='/mnt/restic'
export RESTIC_PASSWORD_COMMAND='/usr/local/bin/get-restic-password'
export RESTIC_CACHE_DIR=/root/.cache/restic
mkdir -p "$RESTIC_CACHE_DIR"

restic unlock
restic backup / --exclude-file=/etc/restic/excludes.txt --tag scheduled
restic check --with-cache --read-data-subset=5G
restic forget --prune --keep-daily 30 --keep-weekly 4 --keep-monthly 6 --keep-yearly 3

sudo chmod 744 /usr/local/bin/restic-backup
sudo chmod 700 /usr/local/bin/get-restic-password

/etc/restic/excludes.txt has to exist and has to exclude the pseudo filesystems before the first run, or restic backup / walks /proc, /sys, /dev and everything mounted under /mnt. The Arch wiki's Restic page carries a working list. Adding --one-file-system to the restic backup line is the alternative, and it keeps the mount point directories themselves in the snapshot.

To clear a stuck lock by hand:


# restic: clears locks whose process is dead on this host, and any lock over 30 minutes old
restic -r /mnt/restic unlock

# restic: clears live locks too. Only if you are certain nothing is running
restic -r /mnt/restic unlock --remove-all

# borg
borg break-lock /mnt/borgrepo

Plain restic unlock is enough for the case in the symptom, because a lock left behind by a killed backup on the same machine already counts as stale. Reach for --remove-all only when the lock belongs to a process that is still alive, or came from another host less than 30 minutes ago. Check for a run in progress on any machine that shares the repository before breaking a lock:


pgrep -a -f 'restic|borg'
systemctl is-active restic-backup.service

If the destination is an external drive that is not always attached, gate the service on the mount rather than letting it fail. Put it in a drop-in so the unit file above stays intact:


sudo systemctl edit restic-backup.service

[Unit]
RequiresMountsFor=/mnt/restic

[Service]
ExecCondition=/usr/bin/mountpoint -q /mnt/restic

An ExecCondition= that exits 1 through 254 skips the remaining commands without marking the unit failed, so a detached drive produces a clean no-op instead of a failed service and an alert.

Use a **system** timer (under /etc/systemd/system/) for a whole-system backup. A user timer only exists while your user instance does, unless you run loginctl enable-linger.

Verify

systemctl list-timers restic-backup.timer shows a LAST timestamp that advances daily, including after the machine has been powered off across 03:00, and ls -l /var/lib/systemd/timers/stamp-restic-backup.timer shows the stamp file that Persistent=true maintains, which is the mechanism that makes the catch-up happen. restic -r /mnt/restic snapshots lists a new snapshot per day, and journalctl -u restic-backup.service -n 50 shows clean runs with no unable to create lock errors.

Sources

scheduled-backup-skipped-and-repo-locked