Why do multi-agent systems fail? The MAST taxonomy, decoded

The failure rate that should scare you

If you’re building multi-agent LLM systems in production, here’s the number that matters: 41% to 86.7% failure rates across seven state-of-the-art frameworks [1]. That’s not a bug in one implementation — that’s a systemic pattern. ChatDev fails 41.4% of the time. MetaGPT fails 56.4%. Magentic-One fails 78.6%. OpenManus fails 86.7% [1].

These aren’t toy systems. These are the frameworks teams evaluate for production use. And until the MAST taxonomy paper from UC Berkeley (NeurIPS 2025 Datasets & Benchmarks Track), nobody could tell you why they fail — only that they do [2].

This post breaks down the UC Berkeley NeurIPS 2025 paper “Why Do Multi-Agent LLM Systems Fail?” (arXiv:2503.13657), decodes the 14 failure modes with real trace examples, and gives you a concrete tool to diagnose your own systems.

What MAST-Data actually is

The research team built MAST-Data, a dataset of 1,642 annotated execution traces from seven frameworks: ChatDev, MetaGPT, HyperAgent, AppWorld, AG2, Magentic-One, and OpenManus [1]. The dataset spans coding, math, and general-agent benchmarks. Each trace was annotated using Grounded Theory — a systematic qualitative method where failure categories emerge from the data itself, not from pre-conceived hypotheses.

The result: 14 failure modes organized into 3 categories, with a Cohen’s kappa of 0.88 for inter-annotator agreement across six expert annotators [2]. That’s “almost perfect” agreement in the statistical literature, making the taxonomy reliable and reproducible.

The dataset is open on Hugging Face, the full taxonomy is documented on the project site, and the codebase is on GitHub.

The MAST taxonomy, decoded

The 14 failure modes map to three execution stages: pre-execution (system design), execution (inter-agent coordination), and post-execution (verification) [2].

Category 1: System design issues (5 modes)

Failures baked into the architecture before a single token is generated.

FM-1.1: Disobey task specification (11.8%). Failure to adhere to the specified constraints or requirements of a task. In ChatDev, agents violated output format requirements, producing code that didn’t match the spec [2].

FM-1.2: Disobey role specification (1.5%). An agent behaves outside its defined responsibilities. In ChatDev, the CPO agent terminated the development process without CEO approval — the system’s termination logic was tied to a single agent’s role, not a group decision [2].

FM-1.3: Step repetition (15.7%). The most prevalent failure mode across the entire corpus. Agents unnecessarily reiterate previously completed steps. OpenManus exhibits this tendency heavily — the system loops through the same code-generation phase multiple times [2].

FM-1.4: Loss of conversation history (2.8%). Unexpected context truncation causes agents to lose recent interaction history and revert to an earlier state [2].

FM-1.5: Unaware of termination conditions (12.4%). Agents don’t recognize when a task is complete, leading to unnecessary continuation or premature exits [2].

Category 2: Inter-agent misalignment (6 modes)

Failures in agent coordination during execution.

FM-2.1: Conversation reset (2.2%). Unexpected restarting of dialogue, losing context and progress [2].

FM-2.2: Fail to ask for clarification (6.8%). Agents proceed with wrong assumptions instead of requesting additional information when faced with unclear data [2].

FM-2.3: Task derailment (7.4%). Deviation from the intended objective, resulting in irrelevant or unproductive actions [2].

FM-2.4: Information withholding (0.85%). The most insidious mode despite its low prevalence. In AppWorld, the phone agent successfully interacted with a phone UI but withheld the API username from the admin agent, who needed it for backend setup. The system deadlocked through repeated failed logins [2][3].

FM-2.5: Ignored other agent’s input (1.9%). Disregarding recommendations from other agents, leading to missed collaboration opportunities [2].

FM-2.6: Reasoning-action mismatch (13.2%). Discrepancy between what an agent reasons and what it actually does — the second most prevalent mode overall [2].

Category 3: Task verification failures (3 modes)

Failures in checking whether the task was actually completed.

FM-3.1: Premature termination (6.2%). The system stops before the task is fully done. AppWorld frequently suffers from this, potentially due to its star topology and lack of predefined workflow making termination conditions less obvious [2].

FM-3.2: No or incomplete verification (8.2%). A ChatDev-generated chess program passed superficial checks (code compilation) but contained runtime bugs because it never validated against actual game rules [2].

FM-3.3: Incorrect verification (9.1%). The system runs verification but the check itself is flawed. HyperAgent exhibits this as a dominant failure mode alongside step repetition [2].

The production hook: o1 as a failure judge

Manual annotation at κ=0.88 is great for research, but you can’t hire six PhDs to annotate every trace in production. The paper’s solution: use OpenAI o1 as an LLM-as-a-judge [2].

The pipeline:

Collect traces from your multi-agent system (agent IDs, messages, tool calls, termination conditions).
Feed the trace to o1 with a structured prompt containing MAST taxonomy definitions and few-shot examples from the human-annotated subset.
Get back a failure classification with rationale.

Results:

κ=0.77 agreement with human annotators on in-domain traces
κ=0.79 agreement on a separate human triple-annotated subset (21 traces)
The judge generalizes to unseen frameworks (OpenManus, Magentic-One) and unseen benchmarks (MMLU, GAIA) [2]

Caveat for eval builders: moderate correlations between similar-symptom modes (max 0.63) mean automated evaluators can conflate distinct root causes [2]. Human review of critical traces is still warranted for high-stakes systems.

Run MAST on your traces

The paper’s team released agentdash, a Python package on PyPI for MAST-based failure diagnosis [6].

pip install agentdash

The full toolkit on GitHub [6] and Hugging Face [4] includes:

The 1,642 annotated traces with failure-mode labels
The o1 judge prompt templates
Inter-rater reliability scripts
Framework-specific trace parsers

from datasets import load_dataset

mast_data = load_dataset("mcemri/MAST-Data")
print(mast_data["train"][0])

The practical workflow: run the judge over your production traces, get a failure-mode distribution, and prioritize by prevalence × blast radius.

Fixes are structural, not prompt-based

The paper tested targeted interventions for specific failure modes. The key result: giving ChatDev’s CEO agent final say over termination (addressing FM-1.2) yielded +9.4% task success [1].

But inter-category correlation is only 0.17 to 0.32 [2]. This means failure modes are largely independent — fixing FM-1.2 (disobey role specification) does not fix FM-2.4 (information withholding). They’re different root causes requiring different architectural changes.

This kills the “prompt engineering will save us” approach. You can’t write a better system prompt that fixes both a termination logic bug and an information-sharing policy bug. The paper’s core conclusion: most failures are system design issues, not prompt-following failures. Tactical patches are insufficient; structural strategies (explicit verification stages, confidence quantification) are required [2].

Caveats

Seven frameworks, not all. Your custom framework might have failure modes outside the taxonomy.
Task domain bias. Skews toward software development and tool use. Creative writing or data analysis might surface different modes.
Cross-model patterns. Failure distributions were analyzed across GPT-4, Claude 3, Qwen2.5-Coder, and CodeLlama-7B — a useful corrective for teams assuming “better model = fewer orchestration failures” [2].
The dataset is static. As frameworks evolve, new failure modes will emerge.

TL;DR

Multi-agent LLM systems fail at alarming rates: 41–86.7% across seven major frameworks [1].
The MAST taxonomy organizes failures into 14 modes across 3 categories: system design (5), inter-agent misalignment (6), task verification (3) [2].
Top offenders: step repetition (FM-1.3, 15.7%), reasoning-action mismatch (FM-2.6, 13.2%), unaware of termination (FM-1.5, 12.4%) [2].
o1 as a judge works: κ=0.77 agreement with humans, generalizes out-of-domain [2].
Fixes are structural: +9.4% from the CEO-consensus fix, but categories are nearly uncorrelated (0.17–0.32) — diagnose before you fix [1][2].
Get started: pip install agentdash, open dataset on Hugging Face, code on GitHub [4][6].

Stop treating multi-agent failures as “the LLM was confused.” Start treating them as structural defects with identifiable root causes. The MAST taxonomy gives you the diagnostic vocabulary. The judge pipeline gives you the tool. The dataset gives you the ground truth.

Your system is failing. Now you can find out why.