In early 2026, OpenAI stopped reporting SWE-bench Verified scores. Not because their models got worse — because the benchmark failed an audit: 138 of its tasks had problems, and over 60% were unsolvable as written, no matter how strong the model.

That's the headline you need to internalize before you trust any agent leaderboard. If a benchmark's tests are broken, a 90% score doesn't mean "solves 90% of real problems." It means "solves 90% of whatever survived the audit."

SWE-bench stopped being a useful signal in 2026. Here's what broke, and what we built instead, because we needed the answer for ourselves.

What SWE-bench Was Supposed to Be

The idea was sound. Take real GitHub issues from open-source Python repos, give an agent the repo and the issue, and check whether the agent's patch makes the repo's own tests pass. Verified — a curated 500-task subset with human-checked tests — was the tightened version meant to fix the noise in the original.

For a while it worked. Then three failure modes compounded, and each one made the scores less meaningful than the last.

Failure Mode 1: The Tests Themselves Are Broken

The 2026 audit that preceded OpenAI's exit found 138 problematic tasks in SWE-bench Verified. More than 60% were unsolvable as written — environment breakage, dependencies that no longer resolve, tests that fail for reasons unrelated to the issue.

When the yardstick is warped, every measurement inherits the warp. An agent scores well partly by lucking into the tasks that work. Two agents with a 5-point gap might be identical, or miles apart — you can't tell from the score, which is the definition of a useless metric.

Failure Mode 2: Contamination

SWE-bench is built from public GitHub issues and public PRs. The models were trained on the public internet. The overlap isn't a risk — it's a certainty.

The fixers of the original issues are in the training data. The patch discussions are in the training data. In several documented cases, the exact issue text is in the training data. So when a model "solves" a SWE-bench task, you can't distinguish reasoning from recall. It's the take-home exam where the answer key leaked years ago, and everyone kept grading it as if it hadn't.

Failure Mode 3: Goodhart, on Schedule

Once a benchmark becomes the leaderboard, agents get optimized for it. Temperature settings, prompt templates, harness quirks — all tuned to squeeze points out of the suite. Some of that is legitimate engineering. But when the suite itself is warped, you're tuning into the warp.

The result is scores that climb while the thing you actually care about — "can this agent fix a real bug in my repo" — stays unmeasured. A number that goes up every month and tells you nothing is worse than no number, because it feels like knowledge.

What a Real Benchmark Looks Like

When we couldn't find one we trusted, we built octobench. The design rules are the ones the audits point to:

Real PRs, recent enough to be uncontaminated. The 25 tasks come from pull requests merged in 2026, across Python, PHP, Rust, C++ and JavaScript. Post-training-cutoff work, pulled from real repos, not synthetic puzzles.

Held-out tests. The tests that judge success aren't the ones that shipped with the fix — they're held out, so an agent can't pattern-match its way to green.

One judge, applied evenly. A judge model scores correctness 0–100 per task, identically for every agent. Same rubric, no per-tool charity.

Cost and time measured, not just correctness. A solve that costs $12 and four hours is a different product than a solve that costs $0.70 and ten minutes. Leaderboards that hide this are hiding the part you pay for.

The results surprised us in both directions. The full runs, harness configs and judge settings are pinned in the repo's BENCHMARK.md — public and reproducible. We publish our own losses in there, which is the point: a benchmark you can't fail publicly isn't a benchmark, it's an ad.

What to Ask Any Leaderboard Now

Three questions, and if the answers are vague, close the tab.

Where do the tasks come from, and could they be in the training data? Who wrote the tests, and are they held out? And where's the cost column?

A vendor who can't answer those crisply is quoting you a number they don't understand either. SWE-bench had a good run — it taught the field what agent benchmarks could be. But in 2026, a SWE-bench score is a participation trophy. Ask for better.

Get Octomind — and read BENCHMARK.md before you believe us, too.

FAQ

What is SWE-bench? SWE-bench is a benchmark that tests AI coding agents on real GitHub issues from open-source Python repositories: the agent gets the repo and the issue, and passes if its patch turns the repo's tests green. The "Verified" subset human-checked 500 tasks for quality. It was the field's standard leaderboard until a 2026 audit found 138 flawed tasks — over 60% unsolvable as written — and OpenAI stopped reporting scores.

Why did OpenAI stop reporting SWE-bench scores? OpenAI dropped SWE-bench Verified from its evaluations in early 2026 after an audit found 138 of its tasks had problems — environment breakage and tests failing for unrelated reasons — leaving over 60% unsolvable as written. OpenAI's own recommendation was to move to alternatives like SWE-bench Pro. Contamination concerns compounded the problem: the public issues and patches are in every frontier model's training data.

What replaced SWE-bench? No single standard has replaced it. OpenAI recommended SWE-bench Pro; we built octobench — 25 tasks from real 2026 PRs across five languages, held-out tests, one judge model, with cost and wall-clock time published alongside correctness. The general rule: trust benchmarks with post-cutoff tasks, held-out tests, and public, reproducible runs.