AI Agent Production Checklist: 15 Things to Verify Before Deploy
A demo proves the happy path. Production is everything else: the retry that fires twice, the enum value that arrives six months from now, the pod that gets evicted mid-run, the coroutine nobody awaited. This guide is based on official documentation and a review of our own arena store data; we did not run these products hands-on.
Why “it works in the demo” isn’t a deploy signal
Anthropic’s engineering guidance on building effective agents reports that the most successful implementations use simple, composable patterns rather than complex frameworks (the guide dates from December 2024, and its own note says much of the tooling has since changed), and warns that frameworks “often create extra layers of abstraction that can obscure the underlying prompts and responses, making them harder to debug.” The same piece notes that agentic systems often trade latency and cost for better task performance, and that you should consider when that tradeoff makes sense.
That is the shape of the problem. Frameworks can obscure prompts and responses, and agentic systems trade latency and cost for performance. The checklist below turns both observations — and a set of platform-specific behaviors from OpenAI, Kubernetes, Stripe, Python, NIST, OWASP, and OpenTelemetry — into tests you can run before merging.
The 15-point pre-deploy checklist
Architecture & tools (1–5)
1. Simple-composable test. Verify: your agent uses simple, composable patterns and you can trace any prompt/response pair end-to-end without framework internals. Test: if you use a framework, confirm you understand its underlying code by tracing one request from entrypoint to final response on paper. Source: Anthropic, Building effective agents.
2. Latency/cost tradeoff decision. Verify: you have explicitly decided when agentic latency and cost are worth the task performance. Test: write down, in the PR or design doc, which tasks are agentic multi-step runs and which are single LLM calls. Source: Anthropic, Building effective agents.
3. Tool-usage testing. Verify: the model actually uses your tools correctly. Test: run a tool-use eval suite and check that stop_reason of tool_use is handled and a correct tool_result is returned for every client tool. Sources: Anthropic, Building effective agents; Anthropic, Tool use with Claude.
4. Poka-yoke tool arguments. Verify: your tools make mistakes hard to make. Test: audit each argument for footguns and change signatures so misuse fails loudly — Anthropic’s own example is an SWE-bench agent that required absolute filepaths after relative paths broke once the agent left the root directory. Source: Anthropic, Building effective agents.
5. Client vs. server tool boundaries. Verify: you know which tools execute in your application (Claude returns tool_use, your code executes and returns tool_result) and which run on the provider’s infrastructure — web search, web fetch, code execution, tool search — plus where server-tool usage charges land on top of tokens. Test: enumerate every tool and label its execution location and billing path. Source: Anthropic, Tool use with Claude.
Throughput & limits (6–7)
6. RPM vs. TPM headroom. Verify: which synchronous limit actually binds you. Test: measure peak requests-per-minute and tokens-per-minute against each limit separately. OpenAI’s rate limit guide states the synchronous API has separate limits for requests per minute and tokens per minute, and that if RPM is the binding limit while TPM has headroom, batching multiple tasks into one request raises throughput.
7. Batch API eligibility sweep. Verify: which workloads don’t need immediate responses. Test: classify each agent task as sync-required or deferrable, and route deferrable work to the Batch API, which exists for work that doesn’t need immediate responses and does not consume synchronous rate limits. Batch has its own queue limit, counted in queued input tokens, and pending batch jobs count against it. Source: OpenAI, Rate limits.
Failure & recovery (8–10)
8. Container restart behavior. Verify: your agent container survives crashes. Test: kill the agent process in a staging pod and watch the kubelet restart it. If restarts keep failing, the container shows the CrashLoopBackOff status. The default maximum restart delay is 300 seconds; the KubeletCrashLoopBackOffMax gate (beta since v1.35, enabled by default) lets you set maxContainerRestartPeriod between 1s and 300s per node, and the alpha ReduceDefaultCrashLoopBackOffDecay gate drops the defaults to 1s and 60s. Source: Kubernetes, Pod lifecycle.
9. Non-recoverable failure handling. Verify: what happens when the cluster can’t heal a pod. Test: simulate an unrecoverable failure and confirm your Deployment or Job recreates the work. Kubernetes deletes pods that fail in a way the cluster cannot recover from and relies on other components to heal them (the recreation check is an author recommendation). Source: Kubernetes, Pod lifecycle.
10. Eviction and node-failure exposure. Verify: behavior under resource eviction and node failure. Test: drain a node and confirm no agent task is silently lost. The documentation is blunt: a pod won’t survive an eviction due to a lack of resources or node maintenance, and a pod scheduled onto a node that fails before start never starts. Source: Kubernetes, Pod lifecycle.
Correctness traps (11–14)
11. Idempotency on retries. Verify: retried agent actions don’t double-execute. Test: replay the same tool call twice and confirm the downstream effect occurs exactly once. Stripe’s idempotent-requests guide describes an idempotency key that lets the server recognise subsequent retries of the same request, and states keys are up to 255 characters long. No retry or timeout thresholds are sourced here.
12. Open-enum fallback branches. Verify: every enum you consume has a safe default branch. Test: inject an undocumented enum value in a unit test and confirm your switch or parser handles it. Stripe’s enum guidance notes that open enums can grow or change over time and warns not to assume documented values are exhaustive. Author recommendation: the specific test injection step.
13. Enum send-side validation. Verify: you only send documented enum values — open enums are still validated on requests. Test: fuzz your request builder with invalid values and confirm rejection is handled rather than surfacing as a silent agent failure. Source: Stripe, Enums.
14. Silent no-op coroutine check. Verify: every coroutine you call is actually run. Test: add a test asserting the side effect really happened, and grep for bare coroutine calls (the grep is an author recommendation, since bare calls can also be scheduled through task APIs). The Python asyncio documentation is explicit that simply calling a coroutine will not schedule it to be executed, and points to asyncio.run() as the top-level entry point.
Security, risk & observability (15)
15. Security, risk & tracing review. Verify: a security review ran against the current OWASP Top 10 for LLM Applications, that risk management follows the NIST AI Risk Management Framework — noting the page states AI RMF 1.0 is being revised as part of the White House AI Action Plan, so do not treat 1.0 as final, with the Playbook as the working companion — and that traces follow the GenAI spans in OpenTelemetry semantic conventions (OpenTelemetry is cited only as existing). Test: record a checklist sign-off naming the Top 10 review and the RMF function reviewed; requiring one GenAI span per agent turn is an author recommendation.
The verification matrix
| # | Check | What could go wrong | Verification test | Source |
|---|---|---|---|---|
| 1 | Simple-composable patterns | Opaque framework layers hide prompts | Trace one request end-to-end; confirm you understand framework code | Anthropic |
| 2 | Latency/cost tradeoff | Agentic runs billed where a single call sufficed | List agentic vs. single-call tasks in the PR | Anthropic |
| 3 | Tool-usage testing | Wrong tool, missing tool_result |
Tool-use eval suite; assert stop_reason handling |
Anthropic tool use |
| 4 | Poka-yoke arguments | Relative paths break outside the root | Audit args; require absolute filepaths | Anthropic |
| 5 | Client vs. server tools | Unexpected usage charges; wrong execution path | Label each tool’s execution location and billing | Anthropic tool use |
| 6 | RPM vs. TPM | Throttled while tokens sit idle | Measure peak RPM and TPM separately | OpenAI |
| 7 | Batch eligibility | Deferrable work burns sync quota | Classify tasks; route deferrable ones to Batch | OpenAI |
| 8 | Container restart | Container stuck in CrashLoopBackOff | Kill the process in a staging pod; watch restart | Kubernetes |
| 9 | Unrecoverable failure | Work deleted with the pod | Simulate failure; confirm higher-level recreation | Kubernetes |
| 10 | Eviction / node failure | Task silently lost mid-run | Drain a node; confirm no task disappears | Kubernetes |
| 11 | Idempotency | Retried action executes twice | Replay a tool call; assert one downstream effect | Stripe |
| 12 | Open-enum fallback | New enum value crashes the parser | Inject an undocumented value in a unit test | Stripe enums |
| 13 | Send-side enums | Validated request rejected at the API | Fuzz the request builder with invalid values | Stripe enums |
| 14 | Awaited coroutines | Silent no-op side effects | Grep bare calls; assert the side effect ran | Python asyncio |
| 15 | Security & risk | Known LLM vulnerabilities unreviewed | Sign-off against OWASP list, NIST RMF, GenAI spans | OWASP, NIST, OpenTelemetry |
What our own arena data says about failed runs
Our own arena store (src/data/niteagent-arena.json, last updated 2026-10-15) records 95 matches, 413 lane-runs across 27 models, and 43 failed lane-runs (10.4%). A lane-run counts as failed when its score is null or 0.0 — failure here means a zero or missing score, not a diagnosed cause, and the store is a sample of one blog.
| Model (12 of 27 shown: 40 of 43 failed runs) | Failed / runs |
|---|---|
| poolside/laguna-s-2.1-free | 11/71 |
| space-bunny-free | 6/10 |
| glm-5.3-flash | 4/30 |
| longcat-2.5-preview-free | 4/10 |
| ling-3.0-flash-fin-free | 3/4 |
| ling-3.0-flash-sante | 3/4 |
| pixel-canary | 3/3 |
| deepseek-v4 | 2/2 |
| deepseek-v4.1-flash | 1/100 |
| mimo-v2.5 | 1/71 |
| mimo-v2.6-flash | 1/16 |
| minimax-m3 | 1/6 |
The sample is one blog and too small to generalise. The practical tie-back is item 3: run tool-use evals and track their failures per model rather than trusting one pooled average, because the failures are uneven (author recommendation). (The three failed runs outside the table are not itemised.)
Comparison: sync agent vs. batch-routed agent
| Dimension | Sync agent | Batch-routed agent |
|---|---|---|
| Latency needs | Immediate responses required | Immediate responses not needed |
| Rate-limit exposure | Consumes synchronous RPM and TPM separately | Does not consume synchronous rate limits; has its own queue limit in queued input tokens |
| Throughput behavior | Packing multiple tasks into one request raises throughput when RPM binds and TPM has headroom | Suited to deferrable work submitted as a batch |
| Best-fit tasks (author examples) | Interactive turns, tool loops, user-facing replies | Classification, backfills, overnight evaluation sweeps |
Qualitative only — this table restates OpenAI’s rate limit guidance; no cost figures were fetched for this piece.
What this checklist can’t tell you
Two gaps, stated plainly. First, no measured cost or latency numbers were fetched for this guide, so nothing here tells you how much an agentic loop costs per task or how many milliseconds it adds. Second, no independent source was found for specific thresholds — “set your budget at X,” “p95 under Y ms.” Any numeric threshold you see in a checklist like this one is an author’s recommendation, not a sourced figure. Items marked Author recommendation above carry no sourced number by design.
FAQ
How do I know if my framework is hiding too much? If you cannot trace a single prompt and response pair end-to-end without reading framework internals, it is hiding too much for debugging purposes. Anthropic’s guidance recommends starting with LLM APIs directly, and if you do use a framework, ensuring you understand the underlying code.
Do server tools cost more than client tools? Client tools run in your application; server tools run on the provider’s infrastructure. Some server tools add usage-based charges on top of tokens, so check the linked pricing sections in the tool use documentation before committing to them. This guide fetched no price figures.
What’s the fastest check with the highest payoff? Author recommendation: items 11 and 14. A single replayed tool call and a single grep for un-awaited coroutines are quick to run, and each guards against a class of silent, duplicated, or missing side effects.
Is NIST AI RMF 1.0 final? No. The NIST page states AI RMF 1.0 is being revised as part of the White House AI Action Plan. Treat it as under revision rather than final, and the Playbook is a companion resource.
The Bottom Line
Treat the checklist as a release gate. Anthropic’s guidance favours simple, composable patterns, and Kubernetes says an evicted or node-failed pod does not survive, so the failure paths need their own tests before deploy. Items marked author recommendation are judgement calls to argue about in review, not sourced rules.
How This Guide Was Built
Desk research from official documentation — Anthropic, OpenAI, Kubernetes, Stripe, NIST, OWASP, Python, and OpenTelemetry — fetched 2026-10-20, plus a review of our own arena store data. Item 15 bundles three security, risk and observability references, so treat it as one combined item rather than three separate checks. We did not run these products hands-on; no benchmarks were executed. The Stripe idempotent-requests page was fetched in full, and the Stripe enum page was used for the open-enum claims. OpenTelemetry’s GenAI semantic conventions are cited only as existing; no attribute names are quoted. All numeric thresholds in this piece are author recommendations unless explicitly attributed to a source above.
← Back to all posts
