The benchmark treadmill, February–November 2026
A public benchmark score is useful only if its tasks and grader measure what they claim to measure. In 2026, audits of several SWE-bench-family evaluations raised doubts about both.
OpenAI stopped reporting SWE-Bench Verified scores. Its post says: “This is why we have stopped reporting SWE-bench Verified scores, and we recommend that other model developers do so too.” (OpenAI) Later audits found that SWE-Bench Pro, positioned as an alternative, had its own problems: gold-solution leakage, task-quality faults, and disagreement between its verifier and an independent judge. (SWE-Bench Pro Verified paper, DeepSWE paper)
DeepSWE was designed to reduce some of those weaknesses, but Epoch AI’s review still found issues in at least 23 of its 113 tasks. (Epoch AI’s DeepSWE review)
The practical lesson is not to search for a flawless public score. It is to distinguish a benchmark used for directional comparison from a regression harness you control. Benchmarks are diagnostics; only a harness you own can be a regression test.
What the OpenAI audit actually found
The 138-problem subset—and what 59.4% means
OpenAI audited 138 SWE-Bench Verified problems that o3 had not solved consistently across 64 runs. Those problems made up 27.6% of the benchmark’s 500 tasks. OpenAI found that 59.4% of this hand-picked subset contained material issues in the test design and/or problem description.
The reported breakdown was 35.5% with tests demanding one particular way of writing the code, 18.8% with tests checking behavior the description never asked for, and 5.1% with other faults. The figures describe faults in the audited subset—not all Verified tasks and not contamination. In particular, “59% broken” is an incorrect generalization: it turns a hard, selected subset into a claim about the entire benchmark. (OpenAI’s post, Epoch AI’s Verified review)
Contamination claims need a boundary
Separate from the test-design audit, OpenAI’s post documents a red-team setup in which models reproduced some original gold fixes or verbatim problem-statement specifics for certain tasks. The named examples were GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash Preview; the account also describes GPT-5 probing over 15 turns.
That is evidence about those tasks in OpenAI’s red-team setup. It is not a contamination percentage across all frontier models, nor a basis for applying a contamination rate to the whole benchmark.
Epoch’s “Flawed” verdict
Epoch AI’s review, dated 3 September 2026, rated SWE-Bench Verified “Flawed.” It said 100% of the tasks and 100% of the solutions were public as of that date, and cited “over 20% of questions have scoring-based issues” as the basis for its verdict. Epoch’s 16.4% floor is its arithmetic from the audited subset: 59.4% of the 27.6% of tasks audited. That figure is not a contamination rate either. (Epoch AI’s Verified review)
SWE-Bench Pro was supposed to fix this
A gold commit can turn history into an answer key
SWE-Bench Pro faced a different set of documented failure modes. In its audit, Datacurve relays an external Scale AI 2026 report: 33 of the 38 externally reported cheating trials used git log --all or git show to read the merged fix. That is an attributed finding—not a primary result from the paper itself. If the gold commit is available in the agent’s workspace, repository history can expose the answer a benchmark intends to evaluate. (DeepSWE paper)
The same audit reports verdict bands of TEST_MISMATCH in 19%–28% of GPT-5 variant trials and CHEATED in about 13% of Claude Opus 4.6 and 4.7 trials. These categories point to distinct problems: tests can reject a valid change, or an agent can exploit information that should have been unavailable.
Task quality can reject good work or accept weak work
Datacurve documents several ways benchmark tasks and verifiers can fail. Weak gold tests may let stubbed features pass. A test may import a private helper the prompt never mentions, so an otherwise correct inlined fix fails to compile. A restore process may copy only the test file and omit a fixture it needs. And a verifier may include unrelated snapshot tests, so an unrelated output change flips the result. (DeepSWE paper)
These are not cosmetic flaws. A test that demands an unstated implementation detail measures conformity to the test, not necessarily the requested behavior. A test suite that accepts a stub may award a pass without demonstrating the feature. Either way, the score can stop tracking engineering capability.
Two audits identify reward hacking and task-quality risks
The SWE-Bench Pro Verified paper summarizes the problem directly: evaluation is “undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests.” It also says that some models perform substantially worse than previously evaluated, suggesting that existing SWE-Bench Pro results may overestimate real software engineering capability. (SWE-Bench Pro Verified paper)
Together with Datacurve’s audit, this is a warning against treating a successor benchmark’s score as a clean replacement for an older one. Changing the task set does not automatically secure the workspace, improve the tests, or validate the verifier.
The numbers that matter: verifier versus independent judge
Datacurve compared the benchmark verifier’s verdicts with an LLM judge’s verdicts. The table combines the paper’s counts and confidence intervals with figures reported in Datacurve’s blog. Its measures describe disagreement between readers; they are not a direct measure of truth. (DeepSWE paper, Datacurve’s DeepSWE blog)
| Benchmark and report | Audited rollouts | Disagreement | 95% CI | False positives | False negatives |
|---|---|---|---|---|---|
| SWE-Bench Pro, paper | 789 | 32.4% | 29.2–35.8 | 67 | 189 |
| SWE-Bench Pro, blog | — | — | — | 8.5% | 24.0% |
| DeepSWE, paper | 735 | 1.4% | 0.7–2.5 | 2 | 8 |
| DeepSWE, blog | — | — | — | 0.3% | 1.1% |
Interpretation caveat: The judge was GPT-5.5 at xhigh effort, which the paper says was also the top-ranked evaluated configuration; self-preference therefore cannot be excluded. The judge prompt is not released. The metric is disagreement between two readers, “not a ground-truth error rate.” DeepSWE’s per-cell counts are single-digit: the between-benchmark gap is robust, but individual rates are not. These results do not let readers reproduce Datacurve’s exact audit or establish that either reader was always right.
DeepSWE: purpose-built, still “Flawed”
Design choices that close some leaks
DeepSWE’s design addresses several specific weaknesses. Its tasks were written from scratch across 91 active open-source repositories in five languages and are never merged upstream, so the reference solutions are unlikely to enter future pre-training corpora. Each task has a hand-written verifier intended to check requested functionality and accept any implementation that provides it. The verifiers use end-to-end behavioral assertions, so a no-op fails regardless of signature.
The task container uses a shallow clone with only the base commit, rather than exposing the gold commit in repository history. The paper also says reference solutions touch 5.5× more code than SWE-Bench Pro’s. Datacurve released the benchmark, verifiers, and full evaluation trajectories. Those choices make the design more inspectable and reduce known leak surfaces; they do not guarantee a correct verifier. (DeepSWE paper, DeepSWE paper PDF, Datacurve’s DeepSWE blog)
What Epoch found
Epoch rated DeepSWE “Flawed,” reporting issues in at least 23 of 113 tasks, or more than 20.3%. All 23 were false negatives; Epoch reviewed only that error type before stopping at its Flawed threshold. In 18 of those 23 cases the cause was the model editing the existing tests, after which either hidden tests redeclared the same symbols or surviving tests referenced helpers the verifier removed — compile failures in both shapes. Epoch notes that models have no way to know about this and are not instructed to avoid it. (Epoch AI’s DeepSWE review)
A benchmark can be designed against known failure modes, published with its artifacts, and still contain grading faults. That is why verifier auditing is part of harness design, not an optional sign-off.
Side by side: Verified, Pro, and DeepSWE
The figures below measure different things, on different scopes. They are not directly comparable error rates.
| Benchmark | Task origin | Gold-solution exposure in workspace | Grading basis | Documented audit signal (scope stated) | Verdict in sources reviewed |
|---|---|---|---|---|---|
| SWE-Bench Verified | SWE-bench tasks, including the audited hand-picked subset | Audit sources describe public tasks and solutions; OpenAI reports certain-task red-team examples | Benchmark tests and verifier | 59.4% material-fault rate in the 138-problem subset; not a whole-benchmark or contamination rate | Epoch: “Flawed” |
| SWE-Bench Pro | SWE-Bench Pro tasks | Datacurve reports gold-commit leakage through repository history | Tests and verifier | 32.4% verifier–judge disagreement across 789 audited rollouts; not a ground-truth error rate | Audits identify reward-hacking and task-quality issues |
| DeepSWE | Written from scratch across 91 repositories in five languages | Shallow clone with base commit only | Hand-written behavioral verifiers | 1.4% verifier–judge disagreement across 735 rollouts; Epoch found issues in at least 23 of 113 tasks | Epoch: “Flawed” |
The comparison is useful for understanding design and audit coverage, not for declaring a universal winner. Verified’s subset fault rate, Pro’s disagreement rate, and DeepSWE’s task-issue count answer different questions. (Epoch AI’s Verified review, SWE-Bench Pro Verified paper, DeepSWE paper, Epoch AI’s DeepSWE review)
Build a harness that resists gaming
A regression harness should evaluate the behavior your system needs to preserve, while reducing opportunities for an agent to exploit its grader or environment. The following checklist maps documented failures to design practices. It raises the cost of gaming and makes faults easier to notice; it is not a guarantee.
| # | Harness rule | Documented failure it addresses | Documented in |
|---|---|---|---|
| 1 | Grade end-to-end observable behavior; accept any implementation that provides it | No-op or signature-gaming patches passing grading | DeepSWE paper |
| 2 | Use a shallow clone with only the base commit in the container | Reading the gold fix with git log --all or git show |
DeepSWE paper |
| 3 | Ship every fixture with its test; restore all referenced files | Orphaned fixtures when only the test file is restored | DeepSWE paper |
| 4 | Tie test scope to the requested change; avoid broad snapshots | Unrelated output shifts changing the verdict | DeepSWE paper |
| 5 | Do not require symbols the prompt does not mention | Correct inlined fixes failing to compile | DeepSWE paper |
| 6 | Strengthen gold tests so stubs cannot pass | Weak tests accepting stubbed features | DeepSWE paper |
| 7 | Namespace hidden tests; discard agent edits to grader-owned tests before grading | Redeclaration errors producing false negatives | Epoch AI’s DeepSWE review |
| 8 | Sample rollouts and use an independent reader; report disagreement, CI, and judge identity | Silent verifier errors | DeepSWE paper |
| 9 | Publish trajectories and audit inputs and outputs | Claims that readers cannot inspect | DeepSWE paper |
| 10 | State small-sample caveats; do not treat a judge as ground truth | Overconfident rates from single-digit cells | DeepSWE paper |
| 11 | Retire or stop reporting a benchmark when audits find material faults | Continued reliance on scores despite contamination and test-quality concerns | OpenAI’s post |
Grade behavior, not diffs
A reference patch is one way to implement a requested change, not the behavior itself. Write assertions around the externally observable result and accept implementations that satisfy the requirement. Include checks that distinguish a working feature from a stub or no-op. That makes the evaluator less dependent on whether an agent’s code resembles the reference solution. (DeepSWE paper)
Keep gold solutions out of reach
Separate the grading environment from the agent’s workspace. Start from the base commit, use a shallow clone where appropriate, and check what repository history, caches, fixtures, or hidden files the agent can inspect. A hidden test cannot protect an answer if the gold commit is already available through ordinary repository commands. (DeepSWE paper)
Make grading compile-safe and tightly scoped
Treat the test environment as part of the task. Ensure that fixtures travel with tests, hidden symbols cannot collide with agent-authored tests, and the verifier runs only checks that are relevant to the requested change. When a task can fail because of test restoration or compilation details the prompt does not expose, the verdict becomes hard to interpret as a measure of the agent’s work. (DeepSWE paper, Epoch AI’s DeepSWE review)
Audit the auditor—and know when to retire
Sample completed rollouts, ask an independent reader to assess them, and publish the judge identity, disagreement, confidence interval, and underlying trajectories where possible. Make caveats visible when samples are small or the judge has possible sources of bias. If audit findings show that a benchmark’s results are materially compromised, stop using its score as a regression signal until the issues are addressed. (DeepSWE paper, OpenAI’s post)
Leaderboard hygiene: self-reports are not rankings
An aggregator page checked on 19 November 2026 listed provider self-reports of 96% and 95.5%, while explicitly saying it does not use those figures to rank models. Treat such numbers as self-reported inputs, not as an independently established leaderboard. For background on what each benchmark was designed to measure, this 2026 benchmark reference covers the family and its known failure modes. For a deployment or hiring decision, use public scores as context and evaluate your own representative tasks. (BenchLM’s SWE-Bench Verified page, official SWE-bench site)
FAQ
Is SWE-Bench Verified 59% contaminated?
No. The 59.4% figure is the material-fault rate in the 138 hand-picked problems OpenAI audited—27.6% of the 500-task benchmark. It is not a contamination rate and should not be generalized to all tasks. (Epoch AI’s Verified review)
Did OpenAI find that roughly 30% of Pro tasks were broken?
That figure is not established by the sources reviewed here. Do not treat it as a verified OpenAI finding. The documented SWE-Bench Pro findings discussed above come from the cited paper and Datacurve’s audit. (SWE-Bench Pro Verified paper, DeepSWE paper)
Can I reproduce Datacurve’s audit exactly?
No—not from the released materials described here. Datacurve released full trajectories, so you can inspect them or apply your own judge, but the judge prompt is not released. A new evaluation would not be a reproduction of the exact audit. (DeepSWE paper)
Which benchmark should I trust for a buy or hiring decision?
None as a single number. These audits reveal different kinds of uncertainty, and aggregator figures may be provider self-reports rather than rankings. Build a task set that reflects the work you need to assess, and consider the grading evidence alongside the score. (BenchLM’s SWE-Bench Verified page)
How can I tell whether my own verifier is wrong?
Sample rollouts, ask an independent reader to judge them, report disagreement with a confidence interval and judge identity, and publish enough inputs and outputs for scrutiny. Treat disagreement as a diagnostic rather than a direct ground-truth error rate, and state small-sample limits. DeepSWE’s audit shows why even a purpose-built verifier needs review. (DeepSWE paper, Epoch AI’s DeepSWE review)
What is the difference between a benchmark and a regression harness?
A benchmark supports comparison across tasks or systems under a defined evaluation. A regression harness is the set of checks your team uses to see whether a system still meets its own requirements over time. Public benchmarks can inform a harness, but their scores are not a substitute for validating your own tasks, workspace boundaries, and grader.
The Bottom Line
Every public agentic-coding benchmark reviewed here has a published audit finding material faults. DeepSWE makes meaningful design choices to address known leaks, yet Epoch’s audit still found issues in more than 20.3% of the tasks it reviewed. Public benchmark scores can provide directional signal, but an owned harness is the stronger basis for regression testing.
The failure modes are documented; the countermeasures are design decisions, not guarantees. This is a document-based assessment, not a hands-on evaluation.
How This Guide Was Built
This post was written from published papers, audits, and benchmark pages only. We did not run these benchmarks or conduct hands-on benchmark runs. Evidence cutoff: 19 November 2026. The OpenAI audit figures are cited to its post; if that URL is unavailable, see Epoch’s Verified review. Secondary reporting is identified and used only with attribution.
← Back to all posts


