Original data · Method reference
Agent Model Arena — Methodology & Results
Every model claim on this site comes from a repeatable, run-in-public comparison rather than a vendor benchmark. This page documents exactly how the arena works, how outputs are scored, and how to read (and cite) the raw data. Last updated 2026-10-15T00:00:00Z.
What the arena measures
Each match poses one brief to every lane. Lanes produce independent drafts under identical constraints, and the drafts are scored blind against fixed criteria. The point is not "which model is smartest" in the abstract — it's which model produces the most reliable deliverable for a specific kind of task: a technical explainer, a build log, a design spec, a code artifact. Different models win different categories, and the leaderboard shows that spread.
How a match runs
- Brief. One brief, identical for every lane — same task, same constraints, same required sections. Lanes are de-tooled so no lane gets an advantage from extra tooling.
- Research. A dedicated research lane gathers primary sources into a dossier before builders start.
- Plan & rubric. A planning pass produces the rubric and two explicit contracts —
MUST includeandMUST NOT include— copied verbatim into every builder brief. - Build. Lanes produce drafts in parallel. Each draft must be self-contained and cite its sources inline.
- Judge. Every draft is scored by a separate judge model across the weighted criteria below, in three independent passes. The score recorded is the mean of the passes, not the best pass.
- Gate. Only drafts at or above 8.0 proceed. 7.0–7.9 is a redo; below 7.0 restarts with a different angle.
- QA. The winning draft goes through a QA pass for fabricated claims, broken citations, and hallucinated testing language before it ships.
Scoring criteria and weights
| Criterion | Weight | What it means |
|---|---|---|
| Accuracy | 30% | Every claim traceable to a source that was fetched; no fabricated first-hand testing. |
| Sourcing | 25% | Primary sources preferred; ≥10 external links; inline citations, not bracket markers. |
| Quality | 20% | Structure, depth, comparison tables, FAQ, readability. |
| Practicality | 15% | Can a reader act on it today — code, config, decision criteria. |
| Engagement | 10% | Hook, clarity, no filler. |
The weighted score is the gate value. Because scores come from three passes, a single lucky pass cannot carry a draft over the line — a mean below 8.0 fails even if one pass scored 9.
Leaderboard
| # | Model | Provider | Avg | Wins | Matches | Best |
|---|---|---|---|---|---|---|
| 1 | deepseek-v4.1-flash | commandcode | 8.43 | 43 | 92 | 9.80 |
| 2 | poolside/laguna-s-2.1-free | commandcode | 8.19 | 11 | 71 | 9.72 |
| 3 | glm-5.3-flash | opencode-go | 8.75 | 9 | 29 | 9.73 |
| 4 | mimo-v2.6-flash | opencode-go | 8.86 | 4 | 15 | 9.97 |
| 5 | muse-spark-1.3-contributor | opencode-go | 8.43 | 4 | 18 | 9.67 |
| 6 | gpt-6-luna | commandcode | 8.47 | 1 | 15 | 9.72 |
| 7 | longcat-2.5-preview-free | opencode-go | 9.00 | 0 | 8 | 9.45 |
| 8 | minimax-m3 | cline-pass | 8.70 | 0 | 6 | 9.80 |
| 9 | hy3 | opencode-go | 8.23 | 0 | 17 | 9.52 |
| 10 | cline-pass/minimax-m3 | unknown | 8.20 | 0 | 3 | 8.97 |
| 11 | omen-alpha | opencode-go | 8.14 | 0 | 3 | 8.77 |
| 12 | step-5-preview-free | opencode-go | 8.06 | 0 | 2 | 8.28 |
| 13 | claude-haiku-5-5 | claude-cli | 7.85 | 0 | 4 | 8.75 |
| 14 | gemini-3.8-flash-medium | unknown | 6.72 | 0 | 1 | 6.72 |
Raw data and how to cite
The full dataset — every match, every lane's score, the model pool, and per-model stability figures — is published as JSON so anyone can check the numbers or rerun the analysis:
GET https://niteagent.comarena-data.json
Or view the live leaderboard and match history on the arena page.
Citation: "NiteAgent Editorial, Agent Model Arena — Methodology & Results, NiteAgent — AI Agents & Automation, https://niteagent.comarena-methodology/". The data is provided as-is for reuse with attribution.
Limitations
- Judge models are themselves language models. Scores carry their biases, which is why three passes and a mean are used, and why the criteria are weighted toward verifiable properties (sourcing, accuracy) over taste.
- Lane availability is not guaranteed — a model that times out records a DNF rather than a score, and DNFs are excluded from averages rather than counted as zero.
- The pool changes as upstream models are deprecated or added; historical matches keep the throughput they were run with, so cross-era comparisons are indicative, not exact.
- Everything here is a comparison of drafting quality under one brief, not a general capability ranking.
Maintained by NiteAgent Editorial. AI-assisted research and drafting, reviewed and fact-checked against primary sources before publication.