Omarchy / Arch troubleshooting research
A corpus of real, reported problems that Omarchy and Arch-based desktop/laptop users hit, each paired with a concrete fix and the source it came from.
The goal is practical coverage, not a headcount. A record only earns its place if the fix field is something you could paste into a shell or a config file.
What's in here now
790 problems across 13 categories, drawn from 2,388 distinct sources. Every record carries at least one real, fetched source URL, and no two records share a slug.
| audit status | count | meaning |
|---|---|---|
ok | 169 | audited and confirmed accurate |
corrected | 621 | problem real, fix (and sometimes cause, symptom or danger) rewritten by the audit |
unaudited | 0 | audit returned no verdict. The last 4 were audited on 2026-09-06 |
So all 790 records have been through an adversarial audit. That audit checked each record against its cited sources, which is a weaker claim than it reads as. 159 records have since been checked against what Omarchy 4 actually ships, and 157 of them needed correcting: 10 boot-kernel and 22 pacman-aur on 2026-09-06 and 2026-09-07, then 14 gpu-drivers, 23 apps-services and 15 network on 2026-09-11, then 16 power-suspend on 2026-09-12 and 15 omarchy-theming, 13 hyprland-config, 18 across wayland-compat and audio-input, and a final 13 across omarchy-core and display-monitors on 2026-09-13. That is every ok record carrying a danger that applies to Omarchy, so this pass is complete. Two records passed: mt7921e-dead-after-suspend-aspm and shell-section-override-ignored-without-colors-toml. The 100th was a reject, meaning the problem it describes does not exist. It was kept rather than retired, because its page is published, and rewritten by hand to say so and to warn against the fix that circulates for it. Its slug still names the non-defect, which no field can change. The failures were rarely the problem being unreal. They were generic Arch advice mis-specialised to this distribution: a fix that rebuilds the initramfs with a command that writes nothing here, a kernel parameter written to a file that is regenerated, a setting the distribution already ships, or a diagnostic that reports a fault on a healthy machine. A separate 36 records harvested from the upstream issue tracker were audited the same day and all 36 needed correcting. See "Refreshing the corpus" for the re-audit path. The last gapfill-unaudited records were audited on 2026-09-01. That status is still a value the schema and merge_gapfill.py can produce, but no record currently carries it.
This was built in three passes. The first harvested 314 records but hit the account spend limit, which killed the gap-fill stage and left apps-services unaudited. The second pass closed both: it audited those 26 records and filled 143 new records against the 117 gaps the first pass's auditors had named. Only two of the second pass's audits failed (wayland-compat, network: API errors rather than budget), leaving 28 records harvested but never reviewed.
The third pass, on 2026-09-01, audited exactly those 28: 12 ok, 15 corrected, and 1 rejected and removed, which is why the first 456 records were 456 rather than 457. Since then no record carries gapfill-unaudited.
The fourth pass, on 2026-10-04 and 2026-10-05, added 285 records. A harvester per category read the existing records first and returned only problems they did not cover, a first auditor reviewed each category's batch, and a gap-fill agent covered what that auditor named as missing. That produced 341 records, of which 56 duplicated another new record and were removed before merging. The 285 that remained then had a second audit, six to eight records per auditor, against their sources, the upstream quattro tree and what Omarchy 4.0.4-1 ships, and 182 of the 285 needed correcting, including 74 that the first auditor had passed as ok. 90 of those had their cause rewritten. 278 of the 285 now carry checked_against naming the 4.0.4-1 install they were read against. Neither audit rejected a single record, which no earlier pass managed, so read that as a limit of the audits rather than a property of the harvest.
The second pass also fixed a flaw in the first: its auditors could return a corrected_cause, not just a corrected fix. 20 records had a disproved cause replaced rather than left standing. A later pass on 2026-08-30 closed the same gap for the first pass's 130 corrected records: all were reviewed and 22 had their cause rewritten to match their own audit note. The third pass added 7 more, for 29 stamped records across two dates. See the trust model below.
Datastore design
SQLite fits, but only for one job, so it isn't the source of truth:
data/problems.jsonlis authoritative. One JSON object per line. It diffs cleanly in git, appends without rewriting, and stays readable when the tooling isn't around. Hand-edit this.data/problems.dbis a derived index. Delete it any time.build_db.pyrebuilds it. SQLite earns its keep because the real access pattern is "a user describes a symptom in their own words → find the matching record", which is a ranked full-text query. FTS5 + bm25 does that in one statement, and the relational tables handle the tag/source fan-out that flat JSONL can't query.docs/*.mdis derived too. One page per category, sorted by severity. This is what a human or an agent reads directly, with no tooling required.
So: JSONL for truth and diffs, SQLite for search, markdown for reading. Anything generated can be thrown away and rebuilt.
Layout
research/
data/problems.jsonl source of truth — edit this
data/categories.json category key -> display label
data/problems.db generated FTS5 index
docs/ generated markdown, one page per category
raw/ unprocessed workflow output, kept for provenance
bench/ skill-efficacy measurements — hand-written, NOT generated
validation/ records exercised on a real VM — a SEPARATE signal from the
audit, and never merged into audit_status (see below)
tests/ stdlib-unittest tests for the corpus writers; ./tests/run.sh
tools/lint_corpus.py flags known-bad shapes; data/lint-baseline.json is the known set
tools/corpus.py the record schema (FIELDS) + the only reader/writer
tools/build_db.py JSONL -> DB + markdown
tools/ask.py symptom search
tools/schema.sql DB schema
tools/ingest.py REPLACES the corpus, refuses unless empty or --replace
tools/merge_gapfill.py EXTENDS it in place, and applies audit verdicts
tools/harvest-workflow.js the agent workflow that produced the corpus
tools/gapfill-workflow.js harvests new records against auditor-named gaps
tools/audit-existing-workflow.js audits records already in the corpus, harvests nothing
Usage
python3 tools/build_db.py # rebuild after editing JSONL
python3 tools/lint_corpus.py # records carrying shapes wrong on Omarchy 4
./tests/run.sh # 18 tests, stdlib only
python3 tools/ask.py "screen share is black in zoom" # search by symptom
python3 tools/ask.py "wifi keeps dropping" -v # include verify steps + sources
python3 tools/ask.py --tag nvidia --tag laptop --list # filter by tags
python3 tools/ask.py --slug some-problem-slug # exact lookup
The scripts are executable and carry a python3 shebang, so ./tools/ask.py ... works too.
Search is deliberately forgiving: query text is tokenised and OR-matched, so a user's phrasing finds a record even when it shares no exact wording. Any FTS5 operators in the query are quoted into literals rather than parsed.
Colour is emitted only when stdout is a terminal, and suppressed by NO_COLOR or TERM=dumb. Redirecting to a file gives clean text. The !! RISK and !! NOT INDEPENDENTLY AUDITED markers are ASCII prefixes, so a warning is never carried by colour alone.
Record shape
{
"slug": "nvidia-black-screen-after-suspend", // unique, kebab-case
"title": "Black screen after resume on NVIDIA",
"category": "gpu-drivers",
"symptom": "...", // as a USER would describe it, incl. literal error text
"cause": "...",
"fix": "...", // copy-pasteable commands / config, markdown fenced
"verify": "...", // how to confirm it worked
"applies_to": ["arch", "omarchy", "nvidia", "laptop"],
"severity": "critical|high|medium|low",
"frequency": "very-common|common|occasional|rare",
"danger": "", // non-empty when the fix risks data loss or boot
"audit_status": "ok|corrected|unaudited|gapfill-unaudited",
"audit_confidence": "high|medium|low",
"cause_reconciled": "2026-08-30", // optional; see the trust model below
"sources": ["https://..."]
}
Trust model: read this before running anything
These fixes were gathered by agents from wikis, issue trackers, and forums, then put through a second adversarial audit pass against the Arch and Hyprland wikis. The audit rejects fixes that are wrong, obsolete, dangerous, or citing sources that don't hold up, and corrects ones that are salvageable. Every record carries what it survived:
audit_status | meaning |
|---|---|
ok | audited and confirmed accurate |
corrected | problem was real, fix was wrong, and the audited version is stored |
unaudited | the auditor never returned a verdict for it |
gapfill-unaudited | added in a later gap-fill pass, never audited. None remain, but merge_gapfill.py still assigns it when an audit agent dies |
unaudited and gapfill-unaudited records are flagged in both ask.py output and the generated markdown. Treat them as leads, not instructions.
One limitation of corrected records from the first pass. Those audits rewrote only the fix field. Where the auditor's objection was really about the cause (a wrong mechanism, an outdated architecture claim), that stale text was left sitting in cause. The second pass fixed this going forward: its auditors supply corrected_cause, and 20 causes were replaced. The first pass's 130 corrected records were not covered by that.
All 130 were reviewed on 2026-08-30, and 22 were found to have a cause its own audit note contradicts. Those 22 have been rewritten from the note (the auditor had already done the source work, and the first pass simply had nowhere to put the result) and each is stamped cause_reconciled. The other 108 were left alone: their notes affirm the cause ("the diagnosis is right", "cause verified exactly") and object only to the fix. The worst offenders were the Omarchy 3 → 4 tree split (~/.local/share/omarchy git checkout vs the pacman-owned /usr/share/omarchy) and fabricated precision: a version boundary or a config filename asserted with more confidence than the source supports.
So the "~130 possibly-stale causes" figure that appeared in earlier notes was a worst-case bound on an unreviewed population, not a count of defects. The real number is 22.
The 2026-09-01 audit stamped a further 7, and the 2026-09-06 audit of the last four unaudited records, the first VM-validated record and ten boot-kernel records re-audited for Omarchy 4 stamped 12 more, and the 2026-09-07 re-audit of all 22 pacman-aur records 14 more, the 2026-09-11 audit of the 36 records harvested from the issue tracker 25 more, and the same day's re-audits of the 14 gpu-drivers records 11 more and the 23 apps-services records 16 more and the 15 network records 12 more, and the 2026-09-12 re-audit of the 16 power-suspend records 11 more and the 2026-09-13 re-audits of the 15 omarchy-theming records 13 more and the 13 hyprland-config records 11 more and the 18 wayland-compat and audio-input records 13 more and the final 13 omarchy-core and display-monitors records 10 more, so 177 records carry cause_reconciled across seven dates. From that pass onward the stamp is applied by merge_gapfill.py itself whenever an auditor supplies a corrected_cause. It previously rewrote the cause and left the field unset, which made the renderers below assert the opposite of what had happened. See ../writeups/2026-09-01-merge-gapfill-silent-defects.md.
Both ask.py and the generated markdown still print the full audit note directly beneath the cause. The line that follows it now says which case you are in: cause rewritten to match the note, or cause not rewritten and possibly still wrong. Read the note before trusting the cause on any corrected record. The fix itself is always the audited version.
This is a research corpus, not a warranty. It is worth reading danger and confirming against the cited source before running anything as root, particularly for anything touching pacman, the bootloader, initramfs, or partitions.
Exercising a record on a real machine is a different signal
validation/ induces a record's problem on a throwaway Omarchy VM, applies a fix, and asserts on the machine. It answers a question the audit cannot: does this work on Omarchy 4?
It never touches audit_status, and the two must not be conflated. audit_status means "checked against its sources". One VM agreeing is not a source confirming. A fix can pass by accident, pass only on that hardware, or pass while its stated cause is wrong. Results live in validation/runs.jsonl, an append-only log, because a record has one audit but many runs, each with its own date and Omarchy version.
Being source-audited does not mean being right about the machine, and the first run proved it. mkinitcpio-pacnew-unhandled-breaks-next-boot is audit_status: ok and its remediation advice checks out, but three of its specifics are false on Omarchy 4: /etc/default/limine is owned by no package and so can never produce the .pacnew its symptom block quotes, /etc/mkinitcpio.conf is [unmodified] on a stock install and so cannot either, and overwriting that file does not remove the encryption/plymouth/btrfs hooks, because those are set by a package-owned drop-in sourced afterwards that assigns HOOKS= wholesale. Generic Arch advice mis-specialised to Omarchy, the same family as the Omarchy 3 → 4 tree split, and invisible to an auditor reading sources.
Scale expectations accordingly: every scenario needs a hand-written seed and assertions, and roughly a third of the corpus is out of reach of these VMs anyway (GPU, laptop and most network records). This is spot-check and bench-source, not corpus validation.
The record schema has one definition
tools/corpus.py owns FIELDS and the only read_jsonl / write_jsonl. Both writers import it. Before 2026-09-01 each kept a private copy, and ingest.py's was missing cause_reconciled, so the replace path would have dropped that provenance silently.
FIELDS order is load-bearing: it is the key order of every line in problems.jsonl. Append, never reorder. An unrecognised key raises rather than being dropped. Harvest working notes the corpus deliberately discards (cause_note, cause_extra, verify_note) are enumerated in WORKFLOW_ONLY. Adding a schema field still means editing four things by hand (schema.sql, build_db.py, ask.py, corpus.py), but tests/ now asserts FIELDS against both schema.sql and the live corpus, so the 2026-08-30 mistake fails the suite instead of destroying data.
Security records and disclosure
The security category carries a gate the other twelve do not. This repository is public and the corpus is published as a website, so a security record is a disclosure whatever else it is. It holds 16 records as of 2026-09-18, the first harvest against these gates, and every one of the 16 was corrected by the audit that followed.
Three gates. Gate 1 decides whether a topic may be written at all. Gates 2 and 3 decide what the record may say, so a draft that fails one of those is rewritten rather than abandoned.
- The weakness is already public, and we are not the ones who made it public. Either it appears in an upstream issue, an upstream code discussion, or a third party's report, published before this repository wrote anything about it, or the behaviour is the documented default, stated in the software's own documentation, its manual page, or the Arch wiki, and cited. Our own earlier publication counts as neither. Treating it as public turns a disclosure we chose to make into a fact we merely observed.
The second clause carries most of this category, and it is not a loophole. Nobody "reported" that systemd-resolved ships with DNSSEC off, or that Avahi advertises a laptop on a hotel network. Those are documented defaults. A gate written only for discovered defects would block two thirds of the work while stopping none of the laundering it exists to stop.
- No field states a condition, input or sequence that its cited source does not. The bound is on the whole record rather than on
causealone, because a mechanism moved intosymptom,title,dangerorverifyis published just the same. Mechanism means the conditions or steps by which the weakness is used. One sentence of consequence in general terms, who can do what, is not mechanism. A record that cannot say why a setting matters is useless. - The record gives a mitigation, not a reproduction. It stops at the defect: name the setting that is missing, the port that is open, the code path that does not validate its input. Never a walkthrough, never a working exploit, never a step that only helps somebody attacking a machine they do not own.
Verification is bound by the same rule as the text. Checking a security claim means reading the setting or reading the code, on this workstation or a test VM. It never means exercising the weakness. verify confirms the control is now present, not that an attack now fails.
A candidate that fails gate 1 does not go in the corpus, and does not go in the journal either. Report it privately, record the send date, channel and window, and hold the record somewhere unpublished until upstream ships a fix or the window closes. The default window is 90 days, stated in the report itself. Public text comes from the report's own paste-ready version rather than being rewritten afterwards from memory.
Note what these gates do not ask: whose fault the defect is. A permissive ufw rule a user typed and a permissive ufw rule the installer wrote are the same disclosure to the same reader. Sorting by blame gives the wrong answer for the most common case in this category, which is a bad default that upstream chose and a user can change.
Evidence before assertion. Confirm the defect at a pinned upstream commit and on an installed version, and record both. checked_against is where the installed version goes. An open issue is not evidence that a problem is current: two of the first fourteen candidates for this category were already fixed in code while their issues stayed open. Read the code, not the issue state.
The precedent, stated accurately, because it is not a clean one. This repository published the mechanism of an Omarchy enterprise Wi-Fi weakness in wpa2-enterprise-8021x-connect-from-cli, citing the two upstream source files that implement it, and described it again in JOURNAL.md. That went live at 2026-09-12 03:55 UTC. The first private report to upstream followed at 2026-09-12 15:34 UTC, eleven hours and forty minutes later. A second report on the same defect went on 2026-09-14. Both are still in triage and neither names this repository's own earlier publication. writeups/upstream/05-WITHHELD.md records the caveat honestly: only the exploitability analysis was ever private. The record stays up, because withdrawing a published mitigation takes away a protection readers already have and advertises the thing the withdrawal is meant to keep quiet. Gate 1 exists so that this order does not happen again.
What is tracked, and what you build
data/problems.jsonl is the source of truth and is tracked. docs/*.md is generated and is also tracked, because the reason that directory exists is that someone can read a category page straight out of a clone without running anything. data/problems.db is generated and is not tracked: it is a 4 MB binary rewritten in full on every build, so each commit of it would add another blob that size to history.
So a fresh clone has the corpus and the reading copy, but no search index. Build it once:
cd research && python3 tools/build_db.py
ask.py refuses to run without it and prints that exact command, so the failure is loud.
The rule that follows: any commit touching data/problems.jsonl must carry the regenerated docs/ with it. Build, then confirm nothing is left unstaged:
python3 tools/build_db.py
git diff --exit-code docs/
Nothing hand-written may live in docs/
write_docs() begins with for old in DOCS.glob("*.md"): old.unlink(). That deletes every markdown file in docs/, including docs/README.md, before regenerating from the JSONL. Anything hand-written there survives exactly until the next build, silently.
That is why bench/ is a sibling of docs/ rather than a page inside it. It holds skill-efficacy measurements (whether giving a model the Omarchy skill measurably improves its answers), which is research but not corpus: no audit_status, no slug, no source list, and no tooling reads it. It is hand-written and stays that way.
Refreshing the corpus
Three workflow scripts live in tools/. Two of them built this, and either can be re-run with the Workflow tool pointed at its script path:
# full harvest: one harvester per category, each audited, appended to the corpus
python3 tools/merge_gapfill.py raw/harvest-result.json # after tools/harvest-workflow.js
python3 tools/build_db.py
# extend an existing corpus: audit unaudited categories, fill named gaps
python3 tools/merge_gapfill.py raw/gapfill-result.json # after tools/gapfill-workflow.js
python3 tools/build_db.py
merge_gapfill.py extends the corpus in place and takes either workflow's result. A harvest result is appended as its own audit left it: existing records are not touched, a colliding slug gets a numeric suffix, and a new record that resembles an existing one is named for review but kept. ingest.py builds a corpus from nothing and refuses to overwrite one that exists unless given --replace, which discards every correction.
To re-audit records that already carry ok against what Omarchy 4 actually ships, use tools/reaudit-brief.md: one agent per one or two records, each writing a verdict-<slug>.json. Assemble the verdicts into a payload with one {"category": ..., "audit": {"verdicts": [...]}} entry per category, scoped to exactly the slugs audited, dry-run merge_gapfill.py on a copy, diff the copy against the corpus to confirm only those records changed, then run it for real. A verdict may replace fix, cause, symptom, danger, verify, severity and frequency, its sources are appended to the record, and its sources_remove drops a citation that does not resolve or does not say what the record claims. The last four keys were added on 2026-09-11, after an audit hit their absence and both changes had to be applied by hand.
To build records from the upstream issue tracker rather than the web, see tools/issue-harvest-brief.md and tools/issue_candidates.py, which lists issues worth reading. A record exists only where the thread carries a fix a maintainer or a second reporter confirmed. Harvest output lands in raw/ and needs deduplication and an audit pass before it reaches the corpus.
tools/lint_corpus.py lists the records still carrying shapes the re-audit found wrong (mkinitcpio -P, sudo pacman -Syu, /boot/limine.conf edits and eight more). It is a candidate list for that brief, not an audit. --check fails on any hit outside data/lint-baseline.json, and so does the test suite, so a new record cannot bring one back unnoticed. When a re-audit clears a record, remove it from the baseline in the same commit.
Pick the workflow by what the records need, not by category. gapfill-workflow.js harvests new records against named gaps, and it audits nothing that already exists. To audit records already in the corpus, use tools/audit-existing-workflow.js and edit its BATCHES. Pointing GAP_CATEGORIES at an unaudited category instead re-harvests the same topics as -2 suffixed duplicates and audits none of the originals. The 28 gapfill-unaudited records that used to be outstanding here were closed on 2026-09-01.
A note on resuming: resumeFromRunId did not cleanly replay only the failed agents here: stalled-agent retries caused it to re-run work and burn budget, and it had to be stopped. Prefer editing the script to target just the failing categories over resuming.
raw/deep-research-report.json is a separate, more heavily verified pass over the same territory: 13 findings that each survived 3-vote adversarial verification, plus 12 refuted claims. The refuted list is worth reading on its own. It is mostly widely repeated folk fixes that primary sources actually contradict.