0.37.1 makes the Supervisor stop believing the agent. When a coding agent reports done, 0.35.0's verify-gate already checked that claim against the recorded log of tool calls it actually made. 0.37.1 goes one level deeper: it checks the claim against the ground truth — the real working-tree diff and the real command output — adds two free deterministic pre-gates that never spend a model call, and stops trusting tool names altogether: whether your change was actually verified is now observed from the working tree itself. One release, one theme: the agent's narrative is no longer evidence.
The self-verification dilemma
The failure this release attacks is the confident wrong done. The agent says "fixed the bug in src/auth/session.rs:142, tests pass" — and none of it is true. The line doesn't exist, the edit landed in a different file, the test command errored and the agent narrated past it. Left unchecked, that fiction gets learned as a success and repeated.
The obvious fix — ask the model to check its own work — is the weakest one. A model grading its own output inherits its own blind spots and rubber-stamps them; the research consensus of the past year is blunt about it. Two things actually move the needle, and both are in Octomind's gate design: verify with a different model family than the one that generated the work (a same-family judge shares the same failure modes — that's why the gate's verifier_model is required and deliberately separate), and ground the verdict in runtime state the model cannot narrate away. 0.35.0 shipped the first half of that grounding with the evidence ledger. This is the second half.
Ground truth outranks the narrative
When the agent claims completion, the gate now assembles a GROUND TRUTH block from state the agent physically cannot edit, and the verifier is told it outranks everything else:
- The working-tree diff (
git diff HEAD) of exactly the files this task's successful mutations touched. A claimed change that isn't in the diff is a gap. - The head of new or untracked files the diff can't cover, and a
MISSINGnote for any file the agent said it wrote that doesn't exist on disk. - The last command's recorded output tail — the 2,000 characters where build and test summaries actually land. A "tests pass" claim is judged against that output, not the sentence next to it.
This is deterministic and built by the runtime, not by a model — the agent's story cannot alter a single byte of it. And it fails open: no git repo, no HEAD, nothing mutated, and the block is simply empty, so the gate falls back to judging the result text on its own terms. The Supervisor never blocks an agent on infrastructure it doesn't have.
Two free pre-gates: no model call, no excuses
Before the LLM verify-gate runs at all, 0.37.1 adds deterministic pre-gates that catch the cheapest classes of false completion for zero tokens.
file:line references must hold on disk. Every file:line citation in the answer is checked against the filesystem — the file must exist and the line must be within the file. A reference to a missing file, or a line past end-of-file, is flagged and re-grounded through the gate. The matcher is high-precision by construction: only real paths (a slash and a file extension) are checked, so URLs and version strings like v1.2/3.4:56 never trip it. This rides the existing claim_check and needs no new configuration.
The plan checklist must be finished. Agents that decompose a task into a plan have a signature failure — drift by omission: they complete three of five steps, quietly declare victory, and the two dropped items never surface. The new require_plan_complete pre-gate refuses a done claim while the live plan still has open items. The agent has to finish them, or explicitly close them out via the plan tool, first — and it gets an advisory listing exactly which items are still open. It sits right next to the pre-existing require_check_after_mutation gate, which refuses done when code changed but no build/test/lint ran since — and that gate just got a serious upgrade of its own. Both are on by default; both cost nothing.
Verified means observed, not inferred
The check-after-mutation gate used to reason by shape (d9614d2c fixes this): classify each tool call as a mutation or a check by what it looks like, and let any successful non-mutation call clear the "unverified change" flag. That guesses at effects. 0.37.1 measures them instead: the runtime fingerprints the working tree around every tool round — a hash of git status --porcelain -uall folded with the size and mtime of every dirty file — and "is there an unverified change?" becomes a comparison of two numbers: the tree now versus the tree at the last clean verification.
Two consequences fall out of measuring instead of guessing:
- Every mutation is caught, whatever made it. An edit through the editor tool, a
sed -iburied in a shell command, a file a sub-agent wrote — no name table could classify all of those, and now none has to. The tree moved; that is the mutation. - A verification only counts when it proves something. A round verifies only when a verifier-shaped call succeeded on a tree the round itself did not change. Run the tests in the same parallel batch as an edit and the gate shrugs — those tests checked an ambiguous state. And reads never clear the flag anymore: looking at code is not checking it, which was exactly the hole in the old shape-based rule.
What counts as "verifier-shaped" is also decided from what the runtime knows, not a name list: the call must carry a string command parameter — the execution signature shells, runners, and remote executors all share — and must not belong to one of octomind's own control-plane tools (the plan tool takes a command too; the runtime knows it executes nothing). Outside a git repo the fingerprint is unavailable and the gate degrades to the shape-based signal — fails open, same philosophy as the ground-truth block.
Prohibitions are requirements too
"Do not modify the tests." Every model reads that at turn one, and some fraction abandons it by turn thirty as attention decays — the negative instruction is the first thing a long session forgets. 0.37.1 attacks this from both ends. The Supervisor now deterministically extracts explicit prohibitions from the request — short, non-question sentences carrying a strong negative imperative ("do not…", "never…", "must not…") — and recites them verbatim at the context tail from the very first turn, capped at eight so precision beats recall. And the verify-gate prompt now treats each prohibition as a requirement in its own right: a mutation recorded against something the request said not to touch, or a forbidden change visible in the ground-truth diff, is a gap even when all the requested work is complete.
Compression keeps the goal, not just the tokens
The other half of c8515976 is continuity. Session compression can now refresh the session's intent and open requests as it compacts, instead of freezing a stale goal from the start of a long session — so after an hour of work the verifier is still checking against what you actually asked for. The gate also carries previously flagged gaps forward across its re-run loop: a gap it found last pass must be closed with concrete evidence or credibly rebutted, not simply outlasted.
Compression's own price tag is no longer invisible either (baf3e1d5). The compression decision runs on its own model, and /info now breaks out exactly what it consumed — input tokens, output tokens, and cost — including the calls that decided not to compress. The session total was always correct; now you can see which slice of it bought you continuity.
The benchmark didn't move — so we made it harder
Verification changes are exactly where you fear a regression: stricter gates can turn into nagging that burns turns and tokens without solving more. So this release went through the same SWE-bench-Live harness we've published against since 0.33.0, measured HEAD-only against the frozen 0.35.0 baseline. It held the success guardrail with no per-instance regression — nothing the baseline solved got broken. That's the honest headline: these gates add refusals only on unproven completions, and everything the agent could already prove, it still proves. The expected direction from blocking premature done claims is fewer false completions at equal-or-lower cost; the guardrail confirms it isn't worse, and by construction it's more likely better.
Then, with the gates in, we froze a new baseline to judge future work against (862bca89) — and deliberately made it harder. The suite grows from the 7-instance pilot to 15 instances (aiogram, cfn-lint, conan, xarray, matplotlib, streamlink, tox, and falcon join), each run at k=4 — 60 runs on the same binary, config, and day, all clean. The headline: 42/60 solved (70%) at $3.39 per solved instance, median 1.95M tokens per solved run, with six instances at a perfect 4/4. Yes, 70% is lower than the old 93% on 7 instances — by design. The added instances are harder (two of the unsolved ones fail on collateral test regressions, not the target fix), and a baseline you can only regress from is a vanity metric. Every future release gets judged per-instance against this: any instance the baseline solves that HEAD drops to zero is a regression, full stop.
Under the hood
0.37.1 also folds in a dependency and protocol pass (8749f30d): the Agent Client Protocol schema moves to v1, and rmcp, tokenizers, and tungstenite are upgraded, with the internal Content type migrated to ContentBlock and the jsonrpcmsg dependency dropped. That migration then got a follow-up sweep (cd5c9065): the outgoing ACP requests octomind's agent tool sends are now built by dedicated functions validated in tests against the actual agent-client-protocol schema types — which caught protocolVersion still going out as the string "0.1.0" where v1 wants the integer 1. Plus the 0.36.1 point release in between, which bumped the core library and trimmed a config reference. Housekeeping — but the kind that keeps the wire protocol current.
Upgrade
# Homebrew (macOS / Linux)
brew install muvon/tap/octomind
# or from crates.io
cargo install octomindPrebuilt binaries for every platform are on the releases page. No migration steps — the new pre-gates are on by default in [supervisor.gate]; set require_plan_complete = false or require_check_after_mutation = false if you want the older, looser behavior.
0.35.0 taught the agent to read less. 0.36.0 made it a better citizen of your repo. 0.37.1 makes it harder for the agent to lie to itself about being finished — and it costs you nothing to keep it honest.
