DeepSeek V4 Flash on 50 Real Pull Requests: Two Harnesses, One Ceiling

Two weeks ago we published a same-model A/B where the harness decided everything: 24 of 25 tasks solved in octomind, 19 of 25 in opencode, at double the cost. We just re-ran that experiment with DeepSeek V4 Flash, on a stricter benchmark, twice the tasks — and the model refused to make it dramatic. 45 versus 43.

On 50 real pull requests harvested from merged fixes across C++, JavaScript, PHP, Python and Rust, deepseek-v4-flash solved 45/50 in octomind and 43/50 in opencode — same model, same system prompt, sealed network, about three cents a task either way. The model is strong enough to be harness-neutral. What's left for a harness to prove is how often it finishes.

What we ran

octobench cases aren't synthetic puzzles. Each one is a real bug or feature that a maintainer actually merged, replayed at the parent commit, and graded by the project's own held-out tests plus a judge panel scoring 0–100. Fifty cases, ten per language.

Since the glm-5.2 run, we rebuilt the methodology, because the first version had a hole you could drive a truck through: the bench is built from merged PRs, and nothing stopped an agent from going and reading the merged PR. In early rounds opencode fetched the upstream fix from raw.githubusercontent.com in 13 of 49 cases and copied it. octomind's stock role was subtler but worse in spirit — it ordered the agent to websearch the upstream issue and "mirror its approach and API contracts EXACTLY." On a benchmark made of merged PRs, that's an instruction to read the answer key.

So this campaign runs clean:

• One shared system prompt for both harnesses, distilled from octomind's role with the client-specific parts stripped. A sync script fails the build if they drift.
• No route to the answer. Web search and web fetch disabled in both clients; GitHub unreachable from the agent's container once setup finishes.
• Same model, same endpoint — official deepseek-v4-flash on api.deepseek.com, deepseek:deepseek-v4-flash in octomind, deepseek/deepseek-v4-flash in opencode.

One measurement note that changed our read of things: we started on a third-party token-plan endpoint and switched to the official API mid-campaign. The official endpoint is roughly 3× faster per request. A chunk of what we'd previously logged as "slow client" turned out to be slow provider.

The scoreboard

Two tasks and three judge points apart. For calibration: our measured judge noise on this suite is about 1.7 points, so the gap is real but roughly double the noise floor — not the five-task blowout the glm run produced. Costs are a coin flip.

And because aggregate numbers hide the interesting stuff, here is every case. ✅/❌ is the verdict from the project's own held-out tests; j = judge score 0–100; then cost, wall time, steps, tokens (in/out/reasoning) and cache read.

The new Flash in numbers

The official V4 Flash went live July 31, and the release is the real thing: DeepSeek's own GA numbers have it beating the V4 Pro preview on agentic benchmarks — Terminal Bench 2.1 at 82.7, DeepSWE at 54.4, Toolathlon verified at 70.3 — from a Mixture-of-Experts with 284B total and 13B activated parameters and a 1M-token context window (specs via its OpenRouter listing).

What that translates to on real-PR work: twelve minutes and three cents per landed fix, a ~94 judge average on cases it passes, in either harness. No babysitting, no mystery timeouts. For the routine 90% of maintainer work, this model at flash pricing is simply a solved problem.

And here's the interesting part — it's harness-neutral. The glm-5.2 run showed a five-task gap between the same model in two harnesses. V4 Flash lands two tasks apart. Two. Part of that is the model carrying more of its own discipline: fewer wasted reads, fewer early victory laps, less need for a supervisor to keep it honest. Part of it is that this benchmark is stricter with both clients — we sealed the answer key, and that cut against octomind's stock role as much as anyone's web access. We made the test harder for ourselves and the gap narrowed anyway. Both things are true.

One ceiling, five walls

The five cases octomind fails are the same five opencode fails: yaml-cpp's binary emit styles, pino-pretty's control stripping, commonmark's fence tabs, rustls's misplaced extensions, and React's hidden hydration hang.

Identical walls across harnesses is what a model blind spot looks like — not a harness artifact. These are the composition-heavy fixes where the model keeps missing a sibling path or a per-message rule no matter how it's prompted. Four instrumented reruns of pino-pretty all missed the same sibling. That's the model, not the steering.

The honest read of the 86–90% band: the last 10% is where model quality ends and everything else begins.

What the harness still decides is the finish

octomind has zero unique failures. opencode's two extra losses are both finishes that never landed: Guzzle's cookie prefixes, where opencode spent 17 minutes and failed while octomind passed in 4; Monolog's trace length, where opencode bailed in 3 minutes with a judge score of 36 while octomind landed a 94.67 in 2.

The React case is the same story at maximum volume. It's a hang-by-design bug — every wrong attempt blocks instead of failing — on a giant repo. opencode stopped at 63 minutes and 394 steps, scoring 36.67. octomind ground for 271 minutes and 1,322 steps — for the same $0.32 — to a fix that passes 66 of 67 hidden tests. We record it as a failure, because it is one, but the judge scored the near-miss at 41.67. (One run died at minute 213 on a provider quota error and was resumed from a restored session to finish at all.)

That's finalization: not better luck, more refusal to stop early. Strip the React outlier and octomind also averages about 7 minutes a case to opencode's 9.

The tax is real, though. octomind wrote 2.6× the output tokens — 1.5M versus 573K. It talks more. And on the React-sized repos its fine-grained tool loop against a fat per-call context is a genuine speed ceiling; coarser steps and context slimming are next on our own list. A harness that finishes everything will still cost you in words.

The model sets the ceiling. The harness sets the floor.

After the glm run we wrote that model quality is table stakes and the harness is where the leverage is. V4 Flash tightens that into something more precise: a model this good raises the ceiling for everyone — pick the model you like, the ceiling barely moves between harnesses. What the harness decides is how often you actually reach it, and whether the last five percent lands or gets declared done at minute 63.

glm-5.3 just dropped. Same bench, next question.

Yesterday Zhipu released GLM-5.3: same base model as 5.2, with the gains coming from post-training alone. Zhipu claims a 50% coding improvement over 5.2 and calls it the strongest open-weights coding model. It's live through their coding plan now. Open weights are promised in about two weeks, behind a safety review.

This matters to us more than most releases, because glm-5.2 is the model that started this story. In the first run, glm-5.2 in octomind beat Claude Code running claude-opus-5 (24/25 solved against 23/25, $63 against $82, 3.6 hours against 6.7) — while paying full freight: that run went through an endpoint with no prompt caching, so every turn re-bought its whole context at list price.

So the next campaign writes itself. glm-5.3 versus glm-5.2 on the same sealed 50-case bench, in both harnesses, answer key locked. A claimed 50% post-training jump is exactly the kind of number vendor benchmarks love and real pull requests interrogate. And there's a sharper question underneath: V4 Flash just showed that a strong model narrows the harness gap to two tasks. If 5.3 really is that much better than a model already beating Opus, does the floor rise with the ceiling — or does the last 10% stay exactly right where it was? Worth finding out.

Reproduction steps and the raw per-case artifacts are pinned at the commit this post describes. GLM-5.3 versus 5.2 is next on the calendar, with Claude Code and Codex columns on the same sealed bench right behind it.