What MLPerf Client v2.0 actually is

MLPerf Client v2.0, released August 18, 2026, is a standardized on-device AI benchmark from MLCommons that adds Agentic AI and Image Generation workloads alongside updated LLM tests for AI PCs MLCommons v2.0 announcement. It is a workload specification and scoring methodology — not a leaderboard — defining how to measure agent task duration, image throughput, and LLM responsiveness on consumer hardware.

MLCommons is an open engineering consortium whose members include AMD, Intel, Microsoft, NVIDIA, Qualcomm, and leading PC OEMs, collaborating on AI benchmarking standards MLCommons Working Group. The v2.0 suite is open-source under Apache-2.0, with the reference implementation available on GitHub GitHub releases. Supported platforms include Windows 11 (x86-64 and ARM), macOS Tahoe 26, iPadOS 26, and Ubuntu 24.04 GitHub README.

The key distinction to internalize: MLPerf Client v2.0 defines how to measure, not what was measured. Vendors and OEMs self-publish their own scorecards against the spec. That means reading the spec carefully matters more than skimming a results table — because as of August 22, 2026, no results table exists yet.

Why the AI PC measurement gap existed before v2.0

MLPerf Client v1.0, released July 30, 2025, measured only single-turn LLM inference — reporting TTFT and TPS on models like Llama 2 7B and Phi 3.5 Mini — which captured token throughput but ignored the multi-turn, tool-calling reality of agent workloads MLCommons v1.0 announcement.

The v1.0 lineup was deliberately simple: Llama 2 7B Chat and Llama 3.1 8B Instruct as base models, Phi 3.5 Mini as an additional base model, and Phi 4 Reasoning 14B as an experimental option. Prompt categories covered code analysis and experimental 4K/8K summarization. Execution providers included ORT GenAI Ryzen AI, OpenVINO, DirectML, Qualcomm Genie/QAIRT, and MLX.

The problem: agents don’t run single-turn prompts. A real agent trajectory involves multiple LLM calls with growing context windows, tool invocations between calls, and constant KV-cache pressure. TPS measures only the decode phase of a single inference pass. It tells you nothing about how a model behaves when context grows from 1K to 27K tokens across ten turns. That’s a different workload entirely, and v1.0 had no vocabulary for it.

The server side recognized this gap earlier. MLCommons formalized agentic inference for datacenter MLPerf Inference in July 2026, treating agent workloads as multi-turn trajectories rather than single-shot prompts MLCommons Agentic Inference. Client v2.0 brings the same thinking to the edge.

The new Agentic AI workloads: SWE Agent and Data Analyst

The Agentic AI category benchmarks end-to-end agent task duration in seconds (lower is better), with a breakdown of LLM inference time versus tool execution time, using a SWE Agent based on Llama 3.1 8B (base tier, ~27k input / ~1k output tokens) and a Data Analyst Agent based on Qwen 3 8B (experimental tier, ~29k input / ~2k output tokens) MLCommons benchmark page.

“End-to-end duration” means total wall-clock time from task start to task completion, including all LLM inference calls and all tool executions. This is the number a user actually experiences. The LLM-vs-tool breakdown is the diagnostic gold: it tells you whether your bottleneck is model inference or tool execution. If tool execution dominates, optimizing the model won’t help — you need to cache, parallelize, or streamline your tool layer. If LLM inference dominates, you have a quantization or model-selection problem.

The token profiles matter. The SWE Agent workload pushes ~27,000 input tokens and ~1,000 output tokens across its trajectory. The Data Analyst pushes ~29,000 input and ~2,000 output. These are long-context, multi-turn scenarios — not single-shot prompts. The context growth across turns is precisely what creates the KV-cache pressure that TPS metrics ignore.

Tier status matters for score validity. Only base-tier workloads count toward official scores. The Data Analyst agent is experimental — published but not mandatory. Quality gates enforce minimum accuracy: Llama 3.1 8B must hit 62 MMLU / 39 IFEval, and Qwen 3 8B must hit 67 MMLU / 25 IFEval MLCommons benchmark page. This prevents vendors from gaming the benchmark with aggressively quantized models that produce garbage output quickly.

Note: the exact internals of the agentic tasks — the specific SWE repository, the tool set, the Data Analyst script — are not disclosed in verified public sources. Don’t speculate; treat the workloads as black-box task specifications.

Image generation and updated LLM tests

The Image Generation category uses Flux 2 Klein 4B (experimental) to produce four images per prompt across four distinct prompts at 1024×1024 resolution, scored in images per minute (higher is better), while updated LLM tests replace Phi 3.5 with Phi 4 Mini Instruct as the mandatory base model and add Qwen 3 8B as experimental MLCommons v2.0 announcement.

Flux 2 Klein 4B is experimental, and no verified performance claims exist in the source set as of August 22, 2026. Treat any vendor image-generation numbers you see with appropriate skepticism.

The LLM updates are substantial. Phi 4 Mini Instruct replaces Phi 3.5 as the mandatory base model. Qwen 3 8B joins as experimental. Phi 4 Reasoning 14B moves to the extended tier. A new base task — Intermediate Summarization with ~4K-token input — fills the gap between the existing code analysis and the 8K substantial summarization. Select summarization tasks now require structured JSON output, testing instruction-following rather than just generation quality.

All LLM models run int4-quantized. Accuracy is gated by MMLU + IFEval thresholds, which prevents the classic benchmark gaming move: quantize to oblivion, get fast tokens, fail quality. The gates force a minimum quality bar before speed counts.

Scoring explained: TTFT, TPS, and why end-to-end duration changes everything

LLM tests report TTFT (time-to-first-token in seconds, lower is better) and TPS (tokens per second, higher is better), where total response latency approximates TTFT + (TPS × output tokens), but the new agentic workloads report end-to-end task duration in seconds because agent performance depends on multi-turn context growth, tool-call latency, and KV-cache pressure — factors that TPS alone cannot capture MLCommons benchmark page.

The single-turn latency formula is straightforward: TTFT plus output tokens divided by TPS. For a 256-token response at 50 TPS with 0.5s TTFT, that’s about 5.6 seconds. Simple.

For an agent, this math breaks immediately. A single agent task might involve ten or more LLM calls, each with a different context length. The first call starts with a 1K-token prompt; by the fifth call, the context has grown to 15K tokens; by the tenth, it’s pushing 27K. Each call has its own TTFT and TPS, and those numbers degrade as context grows because the model must attend to more tokens per generation step.

The deeper issue is KV-cache pressure. As context grows across turns, memory bandwidth becomes the bottleneck — not compute TOPS. An NPU with 50 TOPS of theoretical peak throughput can be memory-starved on a 27K-token context. This is why TOPS ratings are nearly useless for predicting agent performance.

The LLM-vs-tool breakdown in v2.0 is the actionable part. If your agent spends 80% of its time in tool execution, you optimize the tool layer — caching, parallel execution, protocol efficiency. If LLM inference dominates, you look at quantization, model downsizing, or a different runtime. The breakdown tells you which lever to pull.

Quality gates enforce the minimum bar: Llama 3.1 8B at 62/39, Phi 4 Mini at 61/31, Qwen 3 8B at 67/25 on MMLU/IFEval. And only base configurations carrying the “Configuration tested by MLCommons” notice count as valid scores. Anything else is a vendor self-report.

MLPerf Client v2.0 workload map

The table below maps every v2.0 workload category, its model, approximate token counts, scoring metric, tier status, and quality gate — all sourced directly from the official MLCommons benchmark page MLCommons benchmark page.

Workload category Model (tier) Approx. input tokens Approx. output tokens Metric Direction (better =) Quality gate (MMLU / IFEval)
LLM: Content generation Llama 3.1 8B / Phi 4 Mini (base) 128 256 TTFT (s) + TPS lower / higher 62/39 (Llama) · 61/31 (Phi 4)
LLM: Creative writing Llama 3.1 8B / Phi 4 Mini (base) 512 512 TTFT (s) + TPS lower / higher 62/39 · 61/31
LLM: Structured text Llama 3.1 8B / Phi 4 Mini (base) 830 256 TTFT (s) + TPS lower / higher 62/39 · 61/31
LLM: Code analysis Llama 3.1 8B / Phi 4 Mini (base) 1,856 128 TTFT (s) + TPS lower / higher 62/39 · 61/31
LLM: Summarization intermediate (~4K) Llama 3.1 8B / Phi 4 Mini (base) 3,906 128 TTFT (s) + TPS lower / higher 62/39 · 61/31
LLM: Summarization substantial (~8K) Llama 3.1 8B / Phi 4 Mini (extended) 7,631 128 TTFT (s) + TPS lower / higher 62/39 · 61/31
Agentic: SWE Agent Llama 3.1 8B (base) ~27,000 ~1,000 End-to-end duration (s) + LLM-vs-tool breakdown lower 62/39
Agentic: Data Analyst Qwen 3 8B (experimental) ~29,000 ~2,000 End-to-end duration (s) lower 67/25
Image: Text-to-image Flux 2 Klein 4B (experimental) 4 prompts × 4 images, 1024×1024 — Images/min higher N/A

All data sourced from the MLCommons MLPerf Client benchmark page. Token counts are approximate as published. Only “base” tier workloads count toward official scores. Quality gates enforce minimum MMLU and IFEval accuracy to prevent gaming via aggressive quantization.

What v1.0-era vendor numbers tell us about 2026 hardware

No vendor had published MLPerf Client v2.0 scorecards as of August 22, 2026 — four days after release — so all available performance numbers are v0.6/v1.0-era vendor self-reports on older models and must be treated as directional, not comparable across vendors MLCommons v2.0 announcement.

Here’s what we actually know, labeled with the era and model context:

AMD. The Ryzen AI Max+ 395 achieved up to 61 TPS on Phi-3.5 with sub-0.7s TTFT, and the Ryzen AI 9 HX 375 exceeded 27 TPS on the same model. AMD’s hybrid path splits prefill (NPU, 50+ TOPS claimed) from decode (iGPU, bandwidth-limited) AMD technical article. These are v1.0-era results on Phi-3.5 — not v2.0 agentic workloads. The hybrid architecture insight, though, is durable: prefill is compute-bound, decode is memory-bandwidth-bound, and the optimal hardware split respects that.

Intel. The Core Ultra Series 2 (Ultra 9 288V) NPU hit 1.09s first-token latency and 18.55 TPS on Llama 2 7B in MLPerf Client v0.6 (released April 2025). Intel claims first and only full NPU support in that benchmark version Intel Newsroom. Again: v0.6, Llama 2 7B, single-turn.

Qualcomm. The Snapdragon X Series Hexagon NPU claims up to 80 TOPS and is positioned explicitly for “concurrent on-device AI apps and agentic workloads.” The Snapdragon X2 Elite Extreme claims “up to 2x AI performance and efficiency vs competitor GPUs and NPUs.” Both are marketing claims, not MLPerf scores Qualcomm Windows on Snapdragon AI.

Apple and NVIDIA. Apple M-series is supported via MLX and llama.cpp (Metal) execution providers. NVIDIA GeForce RTX 2000+ (8GB+ VRAM) is supported via llama.cpp (CUDA) and DirectML. No verified MLPerf TPS numbers were found in the source set for either platform.

The table below summarizes what we know, with all figures clearly labeled as vendor-claimed:

Platform NPU TOPS (vendor claim) MLPerf Client v2.0 support Verified v1.0-era perf (source, version) RAM needed Runtime path
AMD Ryzen AI Max+ 395 50+ TOPS (AMD) Yes (Ryzen AI 9/300/400) 61 TPS Phi-3.5, <0.7s TTFT (AMD, v1.0) 16GB+ (48GB GPU-only) OrtGenAI-RyzenAI hybrid
AMD Ryzen AI 9 HX 375 — Yes >27 TPS Phi-3.5 (AMD, v1.0) 16GB+ OrtGenAI-RyzenAI hybrid
Intel Core Ultra Series 2 (e.g. Ultra 9 288V) — Yes (Series 2/3 incl. Panther Lake) 18.55 TPS, 1.09s TTFT Llama 2 7B (Intel, v0.6) 16GB+ (32GB extended) OpenVINO / WindowsML
Qualcomm Snapdragon X Elite / X2 Elite up to 80 TOPS (Qualcomm) Yes (32GB+) No verified MLPerf TPS in sources [UNVERIFIED] 32GB+ NativeQNN (Win ARM)
Apple Mac M-series / iPad M2+ — Yes (16GB) No verified MLPerf TPS in sources [UNVERIFIED] 16GB+ MLX / llama.cpp-Metal
NVIDIA GeForce RTX 2000+ (discrete) — Yes (8GB+ VRAM) No verified MLPerf TPS in sources [UNVERIFIED] 8GB+ VRAM llama.cpp-CUDA / DirectML

TOPS and TPS figures are vendor self-reports on different benchmark versions and models — they are not directly comparable across vendors. No vendor had published MLPerf Client v2.0 scorecards as of August 22, 2026. “[UNVERIFIED]” indicates no MLPerf performance number was found in the verified source set for that platform.

One independent data point: ServeTheHome ran MLPerf Client 1.6.1 on a retail Bosgame M5 (Ryzen AI Max+ 395, 128GB) and validated that vendor-class results reproduce on shipping hardware ServeTheHome review. That’s the closest thing to independent validation available — but it ran v1.6.1, not v2.0.

Reading vendor claims like a benchmark auditor

Vendor self-reports are not MLCommons-validated scores — only configurations carrying the “Configuration tested by MLCommons” notice count as official MLPerf Client results, and TOPS figures represent theoretical peak throughput, not measured agent task performance MLCommons benchmark page.

Three rules for reading vendor claims:

Self-reported ≠ MLCommons-tested. Any vendor can publish any number. The benchmark spec defines the methodology; it doesn’t police the results. Only base configurations with the official “Configuration tested by MLCommons” notice carry MLCommons validation. Everything else is marketing with extra steps.

TOPS is a ceiling, not a measurement. TOPS measures theoretical peak integer/floating-point operations per second on the NPU. It does not measure end-to-end agent throughput, which depends on memory bandwidth, runtime optimization, model quantization, and tool-execution overhead. A 50-TOPS NPU can be slower than a 20-TOPS NPU on a memory-bound workload. The v2.0 agentic workloads are precisely that: memory-bound, multi-turn, tool-calling scenarios where TOPS is nearly irrelevant.

Cross-vendor comparison is invalid. Different benchmark versions (v0.6 vs v1.0), different models (Llama 2 7B vs Phi-3.5), different execution providers, different quantization schemes. The numbers you see in vendor press releases are not apples-to-apples. The only valid comparison is within a single benchmark version, on the same model, with the same quality gate — and even then, you need to check whether the configuration carried MLCommons validation.

Do not extrapolate v1.0-era TPS to v2.0 agentic workloads. The models changed (Phi 4 Mini replaces Phi 3.5), the context lengths grew (27K+ tokens), and the scoring methodology shifted from token throughput to end-to-end duration. The v1.0-era numbers are directional signals about hardware capability, nothing more. For a deeper look at how SWE-bench-style evaluations relate to the new SWE Agent workload, see our SWE-bench analysis.

What this means for shipping local agents in 2026

The v2.0 workload spec gives developers a concrete hardware floor — 16GB+ RAM, int4-quantized 8B-class models, 200GB+ disk — and the agentic duration metric reframes the optimization target from “fastest tokens” to “fastest task completion,” which means tool-execution time is now a first-class optimization surface alongside model inference MLCommons benchmark page.

Hardware decisions

The v2.0 requirements establish the minimum spec: 16GB RAM (32GB for extended tasks), 200GB free disk (400GB recommended), and a supported NPU or GPU. For agentic workloads with ~27K–29K token contexts, 32GB RAM is the practical floor — the KV-cache alone for an 8B model at 27K context can consume several GB.

The NPU vs iGPU vs hybrid question is now answerable with data. AMD’s hybrid path (NPU prefill + iGPU decode) reflects the underlying physics: prefill is compute-bound, decode is memory-bandwidth-bound. The optimal split depends on which phase dominates your workload. For short-context, high-turn agents, decode dominates and you want a high-bandwidth iGPU. For long-context, low-turn agents, prefill dominates and the NPU matters more.

Stack design

The runtime layer has matured significantly. Google’s LiteRT-LM serve command exposes an OpenAI-compatible local endpoint, which means existing agent harnesses — OpenClaw, Hermes, OpenCode, Continue, Aider — can point at a fully local model without code changes Google Developers Blog. AnythingLLM and similar frameworks provide local-vs-API routing with no token fees or rate limits AnythingLLM.

The v2.0 LLM-vs-tool breakdown tells you where to optimize. If tool execution dominates, you optimize the tool layer — caching, parallel execution, protocol efficiency. If LLM inference dominates, you consider model downsizing or quantization. The breakdown turns benchmark data into a concrete optimization roadmap. For more on tool-calling protocols, see our A2A vs MCP protocol deep dive.

When local beats API (and when it doesn’t)

Local wins when: data is privacy-sensitive, you need offline operation, per-token cost matters at scale, or rate limits are a bottleneck. API wins when: you need the quality ceiling of frontier models, you’re running many concurrent agents, or you don’t want to manage hardware.

The pragmatic 2026 pattern is OpenAI-compatible local endpoints with hybrid routing — local for routine tasks, API for hard problems. This is the same cost-optimization logic we cover in our AI agent cost optimization guide, now with the hardware floor defined by v2.0’s spec.

The agentic benchmark wave is bigger than AI PCs

MLCommons launched Agentic Inference for MLPerf Inference in July 2026, formalizing agent workloads as trajectories — multi-turn sequences with growing context, KV-cache pressure, prefix caching, and speculative decoding — using 613 trajectories (113 DeepSWE coding + 500 Workato workflow), demonstrating that the industry now treats agentic multi-turn serving as a distinct benchmark class at server scale MLCommons Agentic Inference.

The server-side shift mirrors Client v2.0 exactly: from single-turn throughput to multi-turn trajectory measurement. The 613 trajectories span 113 DeepSWE coding tasks and 500 Workato workflow automation tasks. Trajectory-based benchmarking captures KV-cache reuse, prefix caching efficiency, and context-window management — the same pressures that make agent workloads hard on AI PCs.

The implication for practitioners: the same mental model applies whether you’re deploying on a laptop or a cluster. End-to-end task duration, not TPS, is the metric that matters. Context growth and tool-call latency are first-class concerns at every scale. For a deeper look at how inference runtimes handle these pressures, see our vLLM vs SGLang vs TensorRT-LLM comparison. And for the memory-management side of agent systems, our agent memory systems guide covers KV-cache and context-growth patterns in production.

FAQ

Q1: Does MLPerf Client v2.0 publish an official leaderboard?

No central results page exists — vendors and OEMs self-publish scorecards, and only base configurations carrying the “Configuration tested by MLCommons” notice count as valid MLPerf Client scores. As of August 22, 2026, no vendor had published v2.0 results yet, four days after the benchmark’s release MLCommons benchmark page.

Q2: Why is end-to-end duration a better agent metric than TPS?

An agent run is a multi-turn trajectory with growing context and tool calls; TPS measures only LLM decode speed and ignores tool-execution time and context-reload overhead. MLPerf Client v2.0 reports total task duration plus an LLM-versus-tool time breakdown so developers can see exactly where the seconds go MLCommons benchmark page.

Q3: Can a mid-range AI PC run the v2.0 agentic workloads?

The base SWE Agent uses Llama 3.1 8B in int4 quantization (~4–5GB model class), designed to run on supported systems with 16GB or more RAM and 200GB free disk. The Data Analyst agent (Qwen 3 8B) and Flux 2 image generation are experimental and may require more resources MLCommons benchmark page.

Q4: Is local always cheaper than running agents via API?

Local inference eliminates per-token fees and rate limits while keeping data on-device, but you pay hardware cost, power consumption, and accept a model-quality ceiling. The pragmatic pattern for 2026 is OpenAI-compatible local endpoints — like Google’s LiteRT-LM serve command — with hybrid routing to cloud APIs for tasks that exceed local model capability Google Developers Blog.

Q5: Should I buy AI PC hardware based on NPU TOPS ratings?

No — TOPS is a theoretical peak operations figure and a marketing metric, not a measure of agent task throughput. Use v2.0 workload profiles (context length, memory requirements, supported execution providers) as your hardware spec floor, and treat vendor TPS claims as directional data points only, not cross-platform comparisons MLCommons benchmark page.

The Bottom Line

MLPerf Client v2.0 is the first benchmark to treat on-device agentic AI as a measurable, standardized workload — and that matters more than any single scorecard because it gives the industry a shared vocabulary for what “AI PC ready for agents” actually means. The agentic workloads measure what practitioners care about (end-to-end task completion, not just token speed), the LLM-vs-tool breakdown surfaces the real optimization levers, and the quality gates prevent gaming. But as of August 22, 2026, no vendor has published v2.0 results, so every hardware decision today is still informed by v1.0-era self-reports on different models and shorter contexts. The actionable takeaway: spec hardware to the v2.0 requirements floor (16–32GB RAM, supported NPU/GPU, 200GB+ disk), design your agent stack around OpenAI-compatible local endpoints with hybrid API routing, and wait for actual v2.0 agentic scores before committing to a platform. The benchmark wave — client and server — is real, and agent performance is now a first-class measured quantity.

How this guide was built

This analysis is based on official benchmark documentation, vendor disclosures, and published third-party reviews — we did not run the hardware hands-on. All facts are sourced from MLCommons official announcements and benchmark pages, AMD/Intel/Qualcomm/Google engineering blogs and newsrooms, the GitHub repository, ServeTheHome’s retail-hardware review, and AnythingLLM’s published materials, verified August 18–22, 2026. Vendor performance figures (AMD 61 TPS, Intel 18.55 TPS, Qualcomm 80 TOPS) are self-reported on older benchmark versions and are not cross-comparable. Last verified: August 22, 2026.

← Back to all posts