OPINIONATED OMARCHY
TROUBLESHOOTING CORPUS
790
RECORDS
790
AUDITED
0
UNCHECKED
2388
SOURCES
LIVE
power-suspend medium common DISPUTE

Work out which device is failing or blocking suspend

omarchyarchendeavouroscachyosmanjarolaptopdesktopamdintel
CORRECTED An auditor found the fix wrong or incomplete and rewrote it.
AUDIT NOTE

The kernel interfaces all check out and two factual claims about them do not, plus the record is silent on the Omarchy 4 trap that makes this recipe dangerous to run over ssh. Confirmed on this workstation (kernel 7.1.9-arch1-2, Intel, omarchy 4.0.2-1): /sys/power/ holds pm_debug_messages, pm_print_times, pm_async, pm_test, pm_wakeup_irq and suspend_stats/, all root-writable only, and cat /sys/power/pm_test prints [none] core processors platform devices freezer. The pm_test modes, the freezer to devices to platform to processors to core order and the roughly 5 second return are verbatim from docs.kernel.org/power/basic-pm-debugging.html. The step 1 echo pair is the kernel's own recipe in docs.kernel.org/arch/x86/amd-debugging.html. amd-s2idle test really takes --count, --duration and --format with html, txt or md, and states the 6.1 kernel floor, per docs/amd-s2idle.md on superm1/amd-debug-tools master, and the package is amd-debug-tools 0.2.21-1 in extra. Defect one: the # kernel 6.9+ comment on /sys/power/suspend_stats/* is wrong. Documentation/ABI/testing/sysfs-power dates the directory and fail, success, last_failed_dev and last_failed_step to July 2019, and suspend_stats_show still exists under CONFIG_DEBUG_FS in kernel/power/main.c, so the debugfs copy is current rather than a legacy fallback. Defect two: the record presents last_hw_sleep as an AMD check, but suspend_attr_is_visible() in kernel/power/main.c hides last_hw_sleep, total_hw_sleep and max_hw_sleep unless acpi_gbl_FADT.flags & ACPI_FADT_LOW_POWER_S0, and its ABI date is June 2023. On this Intel desktop /sys/power/mem_sleep reads s2idle [deep] and the three files are absent, so a reader following the verify step gets No such file and no explanation. Defect three, and the reason this is not an ok: sudo systemctl suspend on Omarchy 4 fires the omarchy-sleep-lock.service delay inhibitor and locks the session through Quickshell, and that ext-session-lock surface has no unlock() IPC and outlives its client, so a suspend that hangs on a remote machine is unrecoverable without the physical console. The record also never says the shipped InhibitDelayMaxSec=15 makes a broken lock look like a slow suspend. Added a step 0 for that, added /sys/power/pm_wakeup_irq which the AMD debugging page names first for a spurious wakeup, fixed the wakeup_sources sort, which the record ran as sort -k3 on event_count with the header line dragged in. The header printed by wakeup_sources_stats_seq_start() in drivers/base/power/wakeup.c puts wakeup_count, the count of wakeups from system sleep, in field 4, so the fix now drops the header and sorts on -k4, and split the Limine step into an Omarchy drop-in (/etc/limine-entry-tool.d/99-pm-debug.conf, since /etc/default/limine is owned by no package here) and a plain Arch branch, noting that omarchy-defaults.conf sets quiet splash loglevel=0. Not exercised, deliberately: nothing was suspended, nothing under /sys/power/ was written, and /sys/kernel/debug/ was not readable unprivileged, so the debugfs and wakeup_sources commands are from the sources rather than run. The Limine menu e editing claim for a UKI entry could not be checked because /boot is a vfat ESP mounted dmask=0077, so the fix now hedges it and recommends the drop-in. The record was not GNOME or KDE mis-specialised, and its journalctl -b -k is right because a resume is the same boot.

The cause above was not rewritten and may still contain the error described. The fix below is the corrected version.
RISK

Leaving /sys/power/pm_test set to anything but none is the classic self-inflicted wound: every subsequent systemctl suspend will appear to work and immediately return without the machine ever sleeping, so a laptop in a bag stays fully awake. Always reset it. It does not survive reboot, so reboot if in doubt. no_console_suspend keeps the console powered across suspend and slightly increases sleep-state power draw, so remove it once you are done. Writing to /sys/power/state directly bypasses logind, so nothing locks your screen and no sleep hooks run, and on Omarchy 4 that means omarchy-sleep-lock.service never fires and the session is exposed. Conversely, testing suspend through logind on Omarchy 4 does lock the screen, and that lock has no unlock() IPC and outlives its client, so a failed suspend on a headless or remote machine can leave you with no way back in short of the physical console or a power cycle. Run omarchy-toggle-idle stay-awake before you start and remember to undo it.

Symptom

Suspend either fails outright, takes forever, or the machine wakes straight back up, and all I get is one useless line:


kernel: PM: Some devices failed to suspend, or early wake event detected

I don't know which device to blame, and every guide tells me to try a random kernel parameter.

Cause

By default the kernel logs almost nothing about the suspend path, and the console is blanked before the interesting messages appear, so a failure looks anonymous. The kernel does have a full debugging interface: per-device timing, a failure-counter directory, a staged test mode that stops before the risky part, and a wakeup-source accounting table. It is just off unless you turn it on.

Fix

**0. On Omarchy 4, take the lock screen out of the loop first**

systemctl suspend goes through logind, which fires the omarchy-sleep-lock.service delay inhibitor and locks the session through Quickshell before sleeping. Omarchy 4's lock is an ext-session-lock surface with no unlock() IPC and it survives its own client dying, so a suspend test that hangs can leave a machine you can only recover by typing the password at the physical console or power cycling. Before testing suspend on a machine you reach over ssh:


omarchy-toggle-idle stay-awake     # no idle lock while you work
omarchy-hyprland-session-locked; echo $?   # 0 locked, 1 unlocked, 2 undetermined

Note also that InhibitDelayMaxSec=15 on Omarchy (/etc/systemd/logind.conf.d/20-inhibit-delay.conf), so if locking is broken systemctl suspend sits for up to 15 seconds before it really suspends. That is the inhibitor window expiring, not a slow device.

**1. Turn on verbose PM logging, then suspend**


echo 1 | sudo tee /sys/power/pm_debug_messages
echo 1 | sudo tee /sys/power/pm_print_times
sudo systemctl suspend

After you resume:


sudo journalctl -b -k --no-pager | grep -iE 'PM: suspend (entry|exit)|calling |late |noirq|failed to suspend'

-b is correct here: a resume is the same boot as the suspend. If the machine never resumed and you power cycled it, read the previous boot instead, and accept that anything the kernel printed after the journal stopped flushing is gone:


journalctl --list-boots
sudo journalctl -b -1 -k --no-pager | tail -200

pm_print_times gives a calling <device>+ @ ... / call <device>+ returned N after M usecs pair per device, so the last device before the hang is named explicitly.

**2. Read the failure counters, they name the guilty device directly**


grep -H '' /sys/power/suspend_stats/*

This directory has existed since 2019, not only on recent kernels. On kernel 7.1 it holds 13 files: fail, success, failed_freeze, failed_prepare, failed_suspend, failed_suspend_late, failed_suspend_noirq, failed_resume, failed_resume_early, failed_resume_noirq, last_failed_dev, last_failed_errno, last_failed_step. last_failed_dev is the device and last_failed_step the phase (suspend, suspend_late, suspend_noirq, freeze, prepare).

The same data is also exported through debugfs on current kernels, which is the only place it lives on anything older than 5.4:


sudo cat /sys/kernel/debug/suspend_stats

**3. Bisect with pm_test, which stages the suspend and returns without really sleeping**


for stage in freezer devices platform processors core; do
  echo "== $stage"
  echo $stage | sudo tee /sys/power/pm_test
  echo mem | sudo tee /sys/power/state      # returns after ~5s
  sudo dmesg | tail -30
done
echo none | sudo tee /sys/power/pm_test     # MUST reset

The first stage that misbehaves tells you the layer: freezer means a userspace process will not freeze, devices means a driver's suspend callback, platform and core mean ACPI or firmware. Reading the file shows the available modes with the current one bracketed:


$ cat /sys/power/pm_test
[none] core processors platform devices freezer

**4. If a driver hangs, serialise the suspend so the log ordering is trustworthy**


echo 0 | sudo tee /sys/power/pm_async

Per-device, once you know the suspect:


echo 0 | sudo tee /sys/devices/pci0000:00/0000:00:14.0/power/async

**5. Machine wakes straight back up? Find the wakeup source**

Start with the IRQ the kernel recorded, which is the cheapest answer:


cat /sys/power/pm_wakeup_irq
grep -w "$(cat /sys/power/pm_wakeup_irq)" /proc/interrupts

Then rank the wakeup sources:


# columns: name active_count event_count wakeup_count expire_count active_since
#          total_time max_time last_change prevent_suspend_time
# wakeup_count (field 4) is the one that counts wakeups from system sleep
sudo cat /sys/kernel/debug/wakeup_sources | tail -n +2 | sort -k4 -n -r | head -20
grep -H '' /sys/class/wakeup/*/device/power/wakeup_count
cat /proc/acpi/wakeup
# dmidecode's guess, unreliable on many boards
sudo dmidecode -t system | grep -P '\tWake-up Type: '

Snapshot wakeup_count before suspending and diff after resume. The counter that moved is the trigger.

**6. Keep the console alive so you can read a hang that never resumes**

Temporarily, edit the entry in the Limine menu (e) if your entry exposes an editable cmdline. Omarchy 4 boots a UKI by default (ENABLE_UKI=yes in /etc/limine-entry-tool.d/omarchy-uki.conf), which carries its cmdline inside the image, so prefer a drop-in you can delete afterwards.

Omarchy 4, /etc/limine-entry-tool.d/99-pm-debug.conf:


KERNEL_CMDLINE[default]+=" no_console_suspend ignore_loglevel"

Plain Arch with limine-mkinitcpio-hook and no drop-in directory, /etc/default/limine:


KERNEL_CMDLINE[default]+=" no_console_suspend ignore_loglevel"

sudo limine-update

On Omarchy this deliberately fights the defaults that /etc/limine-entry-tool.d/omarchy-defaults.conf sets (quiet splash loglevel=0 systemd.show_status=false rd.udev.log_level=0), which is the point: you want the messages back. Delete the drop-in and re-run limine-update when you are done.

**7. On AMD s2idle systems, use the purpose-built analyser**


sudo pacman -S amd-debug-tools      # Omarchy: omarchy pkg install amd-debug-tools
sudo amd-s2idle test --count 3 --duration 30 --format txt

It needs kernel 6.1 or later, drives the suspend cycles itself, and prints the blocking IP block, GPIO or missing hardware-sleep residency. Check residency by hand with:


cat /sys/power/suspend_stats/last_hw_sleep

A value of 0 means the platform never reached hardware sleep, so the battery drained because the machine only ever froze the CPUs.

last_hw_sleep, total_hw_sleep and max_hw_sleep are only created when the firmware advertises ACPI Low Power S0 Idle. On a machine without it, typically a desktop reporting s2idle [deep] in /sys/power/mem_sleep, the files are simply absent and cat returns "No such file or directory". That is expected, not a fault, and it means this whole step does not apply.

Verify

grep -H '' /sys/power/suspend_stats/* shows fail and last_failed_dev populated after a failed attempt, and cat /sys/power/pm_test prints the mode list with [none] bracketed when you are finished. cat /sys/power/pm_async reads 1 again once you have undone step 4. On an AMD or Intel machine that advertises ACPI Low Power S0 Idle, cat /sys/power/suspend_stats/last_hw_sleep is non-zero after a successful s2idle cycle. On a machine without it that file does not exist at all. On Omarchy 4, run omarchy-toggle-idle allow-idle afterwards to put idle locking back.

Sources

find-which-device-fails-suspend