OPINIONATED OMARCHY
TROUBLESHOOTING CORPUS
790
RECORDS
790
AUDITED
0
UNCHECKED
2388
SOURCES
LIVE

Local model feasibility on the agentic lane

Measured 2026-09-02 on omarchy-agentic-root-config (run 21), bare (variant: none), one repeat per model, all 14 tool-capable models the host serves.

Read this before concluding a bench is too easy. A model that cannot drive the agent loop and a model that finds the task trivial produce different scores, but a model that runs out of VRAM, one that times out, and one that emits its tool call as prose all produce the same score: the untouched floor. The score column alone cannot tell you which you are looking at, and that difference decides whether the fix is a harder bench or a different model.

The floor is 5/8, not 0

omarchy-agentic-root-config has 8 post assertions. A model that does nothing scores 5/8, because five of them are "you did not break anything" checks. Those are pacman -Qkk clean, the local customisation still present, and so on. The three that actually move are: adopt the new upstream setting, remove the .pacnew, regenerate the boot config.

So 5/8 means "did nothing" and 8/8 means "did the whole job". There is very little in between, which is itself the finding.

Four gates, and they fail differently

  1. Tool support. capabilities must include tools. Without it pi fails instantly with HTTP 400 does not support tools. Check with python3 skillbench/tools/probe_models.py; no run needed.
  2. VRAM. Weights plus a 32K KV cache must fit the 24 GiB card. A model that does not fit does not fail cleanly: Ollama spills the remainder to CPU and the case crawls into the agent timeout, which reads as a capability failure. Confirm with curl -s localhost:11434/api/ps mid-run and look at size_vram against size.
  3. Harness compatibility. The model must survive pi's OpenAI-compat request shape. qwen3.8:27b looked incapable here and was not. See "The blocked model was capable" below. Record a harness failure as blocked, never as a capability judgement.
  4. Agentic competence. Declared tool support is not competence. Most models that pass gate 1 still fail here, and a skill cannot fix any of them: they do not lose on Omarchy knowledge, they lose before knowledge is reachable.

Results

The "what happened" column is not re-derivable, and that is a defect in how this was recorded rather than a doubt about the findings. Each entry was read from a live transcript on 2026-09-02. app/runner.py invokes pi without --mode json, so the database kept only the final text, a median of 16 characters across roughly 600 agentic cases, and no token accounting at all. So "emits pseudo-XML instead of a tool call" and "empty transcript" are honest reports of something nobody can now go back and check.

Treat the score and timing columns as evidence: all 14 cases of run 21 are banked in results/ and reproduce this table exactly. The params, disk and on-GPU columns came from probe_models.py and ollama rather than the run, so they are reproducible only while those models are still pulled. Treat the failure attributions as a contemporaneous note. If this table is ever re-measured, pass --mode json first and the distinction disappears.

modelparamsdiskon GPUscoretimeverdictwhat happened
qwen3-coder:30b30.5B18.6 GB100%8/842 s✓ capableSolves it.
gemma4:26b25.8B18.0 GB100%8/897 s✓ capableSolves it.
devstral-small-2:24b24.0B15.2 GB100%8/886 s✓ capableSolves it.
qwen3.8:27b27.3B17.7 GB100%8/8378 s✓ capableSolves it (run 22, 2 of 3 clean; 1 run hit an intermittent 500 at 7/8).
qwen3:32b32.8B20.2 GB90%5/8237 s✗ floorWeights + 32K KV exceed the card: 10% on CPU, then stalls.
muse-glimmer:30b27.9B18.2 GB100%5/8604 s✗ floorFits entirely on GPU; still exceeded the 600 s budget.
mistral-small3.2:24b24.0B15.2 GB100%5/819 s✗ floorActs, hits a permission error, gives up.
gpt-oss:20b20.9B13.8 GB100%5/861 s✗ floorEmpty transcript.
gemma48.0B9.6 GB100%5/840 s✗ floorClaims success falsely: reports the merge done, changed nothing.
qwen3:14b14.8B9.3 GB100%5/826 s✗ floorEmpty transcript.
qwen2.5-coder:14b14.8B9.0 GB100%5/815 s✗ floorEmits the tool call as fenced json text.
qwen2.5vl:7b8.3B6.0 GBn/an/an/a✗ no toolsNo tool support: HTTP 400, cannot run.
granite3.3:8b8.2B4.9 GB100%5/815 s✗ floorEmits pseudo-XML <file name=…> instead of a tool call.
llama3.1:8b8.0B4.9 GB100%5/810 s✗ floorEmits the tool call as JSON text.
qwen2.57.6B4.7 GB100%5/881 s✗ floorNarrates shell in markdown; never executes.

What this means for measuring skills

Four of fourteen local models can run this bench at all, and all four already score 8/8 bare. There is therefore no headroom on this bench for any skill to demonstrate a lift, and run 18 confirmed it directly: devstral-small-2:24b, none vs skill:omarchy, came out 0.958 vs 0.958 (identical to three decimals). That run also reported the skill at 3.0× the latency, a figure not quoted here: until run 26 the clock started before the case acquired a VM, so the second variant banked the queue wait.

The band is narrow and it is not obvious from the outside: every model capable enough to drive the loop is also capable enough to finish the task. Writing harder agentic tasks is therefore not obviously the fix. The failing models will keep scoring the floor for reasons that have nothing to do with what a skill could tell them. That was the assumption behind the old backlog item and this table is the evidence against it.

The most dangerous single result is gemma4 (8B), which reported the merge complete while changing nothing. Any grader that trusted the transcript would have scored it a success. This is exactly why the agentic lane asserts on the machine and carries no transcript checks.

Reproducing


python3 skillbench/tools/probe_models.py          # gates 1 and 2, no run needed
tools/provision-bench-vm.sh 1 2                   # REQUIRED after adding a model

Re-provision after pulling a model. pi's ~/.pi/agent/models.json is written by provisioning. pi still runs an unlisted model. It warns not found for provider ollama. Using custom model id and carries on with its own defaults, so an unlisted model is not configured identically to a listed one, and nothing surfaces as an error. The list is derived from Ollama's tool-capable models, so pulling plus re-provisioning is enough; you never edit the list by hand.

Then launch a bare scan across every model from the UI, or:


curl -sX POST http://127.0.0.1:8878/api/runs -H 'Content-Type: application/json' \
  -d '{"bench":"omarchy-agentic-root-config","variants":["none"],"repeats":1,
       "models":["<model>", "..."]}'

Caveats

The blocked model was capable: read this before trusting any single scan

qwen3.8:27b was recorded on 2026-09-02 as blocked: one bare case, pi exited 1, a transcript containing only 500: {"message":"no user query found in messages"}, and a 5/8 floor score. On a one-repeat scan that is indistinguishable from a model that cannot act.

It can. Re-run with 3 repeats (run 22) it scored 8/8, 8/8, and 7/8, and the 7/8 came from a case that had already done most of the work before the error. Run directly it produced the most sophisticated solution of any model tested: it used Omarchy's own omarchy-refresh-limine rather than calling limine-update by hand, and noticed that OMARCHY_PATH is not passed through sudo, then corrected for it.

Root cause. The string no user query found in messages is compiled into the Ollama binary, next to its reasoning-effort strings. qwen3.8:27b reports family qwen35 and template {{ .Prompt }} (a stub), so Ollama renders it with a built-in renderer for that family, and that renderer requires at least one user message. Reproduce it in one call, no agent involved:


# 200 OK
curl -s localhost:11434/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8:27b","max_tokens":8,
       "messages":[{"role":"system","content":"s"},{"role":"user","content":"hi"}]}'
# 500 no user query found in messages
curl -s localhost:11434/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8:27b","max_tokens":8,"messages":[{"role":"system","content":"s"}]}'

system + assistant fails the same way; system + user + assistant + tool is fine. devstral-small-2:24b accepts all of them, which is why only this model trips it. So somewhere in a long agent loop pi sends an array with no surviving user turn and Ollama rejects the whole request. It is intermittent (~1 in 3) and it is an Ollama-side constraint, not a pi bug and not a model capability.

The lesson for this table: a one-repeat scan cannot separate "cannot act" from "hit an intermittent harness failure". Anything recorded as blocked deserves a re-run before it is believed, and any model here that scores the floor with an error set should be treated as unmeasured rather than incapable.

skillbench/MODELS.md · view on github