OPINIONATED OMARCHY
TROUBLESHOOTING CORPUS
790
RECORDS
790
AUDITED
0
UNCHECKED
2388
SOURCES
LIVE
pacman-aur medium occasional DISPUTE

Finding truncated or missing package files after a crash or bad disk with pacman -Qkk / paccheck

archomarchyendeavouroscachyosmanjarodesktoplaptop
CORRECTED An auditor found the fix wrong or incomplete and rewrote it.
AUDIT NOTE

Checked on this Omarchy 4 workstation (pacman 7.1.0.r9.g54d9411-2, omarchy-settings 4.0.2-1): pacman -Qk omarchy-settings prints omarchy-settings: 628 total files, 4 missing files and -Qkk prints 629 total files, 4 altered files, both with (Permission denied) warnings for the four /etc/sudoers.d/omarchy-* files when run unprivileged, so the ' 0 missing files' and ' 0 altered files' filters match the real strings (also confirmed in src/pacman/check.c, which uses the _n plural forms). pacman -Qkk pacman on this fresh install already prints three backup file: pacman: /etc/pacman.conf (...) mismatch lines with 0 altered files, because Omarchy's post-install copies its channel template over that file: the record's 'those are your own edits' is wrong on Omarchy 4, so the /etc caveat was rewritten, and the difference between a backup file: line and a real altered file is now spelled out. pacutils is in extra at 0.15.0-2 and NOT installed here, and paccheck(1) from man.archlinux.org documents --sha256sum, --files, --file-properties and --quiet as used. The guard claim held: /usr/share/libalpm/hooks/00-omarchy-update-guard.hook fires on Operation = Upgrade (which includes a reinstall, per lib/libalpm/hook.c oldpkg matching), but /usr/bin/omarchy-update-pacman-guard exits 0 unless the pacman command line has both -S and -u, so per-package pacman -S is not blocked. Added one precondition to step 4: omarchy-update-keyring runs pacman -Sy before the -Syu, so if the update stops afterwards the sync databases are ahead of the system and a single-package pacman -S becomes a partial upgrade. checkupdates is available (pacman-contrib 1.13.1-1 is installed here). yay --rebuildall confirmed in yay 13.0.1 help. Not exercised: paccheck itself (not installed, no sudo), the reinstall loop, smartctl and btrfs scrub.

The cause above was not rewritten and may still contain the error described. The fix below is the corrected version.
RISK

If the local database itself was damaged (/var/lib/pacman/local), -Qkk reports nonsense and reinstalling on top of it makes things worse. Restore the database first. Do not run a blanket pacman -Qqn | pacman -S - on a system whose disk is still failing. You will only write more corrupt files. Reinstalling packages overwrites files under /etc that the package owns and does not list as backup files, so read the reinstall list first. If you suspect a compromise rather than a crash, run these checks from a live USB against hashes from an independent source. A rootkit can rewrite the local mtree data that -Qkk and paccheck compare against.

Symptom

"The machine hard-locked / the disk threw errors / I pulled the plug during an update. It mostly boots, but random things segfault, an icon theme is half missing, and a binary is 0 bytes." Typical runtime fallout:


/usr/bin/foo: error while loading shared libraries: libbar.so.5: file too short
bash: /usr/bin/foo: cannot execute binary file: Exec format error

pacman -Syu says the system is fully up to date and refuses to help.

Cause

A crash, a filesystem that filled mid-write, or a failing SSD can leave files that pacman installed truncated, zero-length or missing entirely, while /var/lib/pacman/local still records the package as installed at the current version. pacman -Syu only compares *versions*. It never looks at the files on disk, so it will never notice and never re-extract them.

Fix

**1. Find missing files across every installed package:**


sudo pacman -Qk $(pacman -Qsq) | grep -v ' 0 missing files'

Output looks like foo: 231 total files, 3 missing files. Nothing printed means nothing is missing. Run it with sudo: as an unprivileged user every root-only file (for example /etc/sudoers.d/*) prints a (Permission denied) warning and is counted as missing.

**2. Find altered files too** (size, mtime, permissions). This is what catches truncation:


sudo pacman -Qkk $(pacman -Qsq) | grep -v ' 0 altered files'

Expect false positives under /etc. Config files are backup file: lines, reported but not counted as altered, and they mismatch whenever anything edited them. On Omarchy 4 that includes the installer and migrations, so a fresh install already prints:


backup file: pacman: /etc/pacman.conf (Modification time mismatch)
backup file: pacman: /etc/pacman.conf (Size mismatch)
backup file: pacman: /etc/pacman.conf (SHA256 checksum mismatch)
pacman: 426 total files, 0 altered files

A backup file: line is not damage. A non-backup line such as foo: /usr/lib/libbar.so.5 (Size mismatch) is.

**3. Real content verification against the packaged hashes**, using paccheck from pacutils (extra, not installed by default on Omarchy):


sudo pacman -S --needed pacutils
sudo paccheck --sha256sum --quiet
sudo paccheck --files --file-properties --sha256sum --quiet   # thorough

**4. Reinstall everything that came back damaged.** First make sure the sync databases are not ahead of the installed system, or a single-package pacman -S pulls a newer version and creates a partial upgrade:


checkupdates              # must print nothing; if it prints anything, run omarchy update (Arch: sudo pacman -Syu) first

Then:


sudo bash -c 'LC_ALL=C.UTF-8 pacman -Qk 2>/dev/null | grep -v " 0 missing files" | cut -d: -f1 | \
  while read -r p; do pacman -S --noconfirm "$p"; done'

On Omarchy the update guard hook (/usr/share/libalpm/hooks/00-omarchy-update-guard.hook) runs on every reinstall, but /usr/bin/omarchy-update-pacman-guard aborts only when the pacman command line carries both -S and -u, so plain pacman -S pkg goes through. If it fires anyway, prefix the command with env OMARCHY_ALLOW_DIRECT_PACMAN=1.

**5. Foreign (AUR) packages are in no repo and must be rebuilt:**


pacman -Qqm                                    # list them
yay -S --rebuildall $(pacman -Qqm | tr '\n' ' ')

**6. Check the hardware before you trust the repair:**


sudo pacman -S --needed smartmontools
sudo smartctl -a /dev/nvme0n1 | grep -iE 'health|media|error'
sudo btrfs scrub start -B /                    # btrfs root
sudo btrfs device stats /

Verify

A second sudo pacman -Qk $(pacman -Qsq) | grep -v ' 0 missing files' prints nothing, and sudo paccheck --sha256sum --quiet prints nothing. sudo btrfs scrub status / reports 0 uncorrectable errors.

Sources

verify-installed-files-against-package-database