AI Agents for Infrastructure: Automate Your DevOps

03:14 and the page fires: checkout errors are climbing. The responder opens metrics, then logs, then the deploy timeline, then Slack, then the runbook — and an hour later the cause turns out to be an IAM misconfiguration on the instance hosting a GitHub runner, or a Lambda configuration nobody has touched in months. Both outcomes are real, relayed by vendors from this year’s launches. The job being sold is the boring part: gather evidence, form a hypothesis, test it against telemetry and code. What separates the products is not the model underneath but the structure of the investigation, the gates on what the agent may touch, and whether anyone can prove it is improving.

What an infrastructure agent is — and the four shapes it ships in

incident.io’s docs draw the line that matters: “An investigation isn’t a single prompt to a model. It’s a structured process that gathers evidence, forms a hypothesis, then tests that hypothesis against your code and telemetry” (incident.io Investigations). AWS describes its offering as a remote server supporting “MCP, ACP, and A2A”, organised around Agent Spaces and a dual console — Management Console for spaces, integrations and access controls, web app for investigations (AWS DevOps Agent user guide).

Four market shapes are documented well enough to compare: a cloud vendor’s GA agent (AWS DevOps Agent, generally available 31 March 2026 — AWS Cloud Operations Blog — six regions, Azure and on-premises support via MCP, on-demand SRE tasks, code indexing); an incident platform’s agent (PagerDuty’s Paige); an incident-channel engine (incident.io Investigations); and an observability vendor’s agent (Datadog’s Bits Investigation). The differentiator across all four is structure, gating and evaluation — not the model.

Product What it is Where it runs the investigation Knobs the operator has What it hands back to a human
AWS DevOps Agent GA 31 March 2026; remote server over MCP, ACP, A2A; dual console AWS-hosted, six regions; AWS, Azure and on-prem workloads via MCP topology Agent Spaces, access controls, nine pre-GA integrations plus four added at GA, skills per agent type, code indexing, Triage LINKED dedupe Mitigation plans with “agent-ready instructions” for another agent (e.g. Kiro); deterministic permission and blast-radius proofs in release review
PagerDuty Paige Supervisor + stateless sub-agents in one process; Virtual Responder Early Access Console, incident page, Slack, MS Teams (Early Access) Per-service memory plus a Memory API, nine nudge actions, priority queue Recommended diagnostic and remediation steps; logs from Grafana, Datadog, New Relic, CloudWatch; runbooks from Confluence and GitHub
incident.io Investigations Structured investigation inside the incident Incident channel; sandboxed ephemeral code workspaces Holds off on thin information; desktop app pulls a live investigation into Claude Code, Codex or Cursor via MCP Findings graded on a five-rung conviction ladder; discounted hypotheses kept in the audit trail
Datadog Bits Investigation Hypothesis-forming agent over your telemetry Monitors with Auto-Investigate enabled bits.md rules, monitor wording, service scoping, feedback memories Root-cause pinpointing by data-based reasoning; reusable feedback memories; confidence grading: not stated

No source supports ranking these platforms; the table compares design choices, not performance.

Attribution matters. AWS reports preview-period customer and partner figures (AWS GA announcement) of up to 75% lower MTTR, 80% faster investigations, 94% root cause accuracy, and 3–5x faster incident resolution — with no methodology or sample sizes published. Two anecdotes: Western Governors University (>191,000 students, a Dynatrace user) saw one investigation fall from an estimated two hours to 28 minutes, a 77% MTTR improvement, root cause in a Lambda configuration; Zenchef wrapped one in 20–30 minutes against 1–2 hours, root cause an IAM misconfiguration on the EC2 instance hosting GitHub. All of it is vendor-relayed, not verified.

An investigation is breadth first, depth second

incident.io documents three phases (how investigations work):

  1. Fast diagnosis — establish what is broken before deep-diving anywhere.
  2. Gather context in parallel — fan out across Slack, similar past incidents, runbooks and reference docs, deploys/feature-flag/config change events, code changes, telemetry, and third-party outage status “all at once, so the slow searches never hold up the rest”.
  3. Analyze and refine — form a hypothesis, then test it against your code and telemetry.

The number that should reshape how you judge these tools: “Roughly half of an investigation’s work goes not into forming a hypothesis but into challenging and proving it.” Most of the engineering is in the second half — and most of the marketing is not.

One big agent breaks in four documented ways

PagerDuty’s engineers built version one of their SRE agent as a single agent with a large context window and tools to query logs, check metrics, and look at recent deploys (PagerDuty engineering write-up). It failed in four ways they named verbatim:

  1. Context Rot — “More data, worse decisions”, citing Liu et al., 2023.
  2. Instruction Overload — research suggests an inverse relationship between instruction volume and output quality (Jaroslawicz et al., 2025).
  3. Sequential Synchronous execution — a moderately complex incident with three or four candidate causes “could take 10+ minutes to diagnose”.
  4. No interactivity during execution — on-call engineers could not inject what they knew, such as “the service had been deployed 10 minutes before the alert”, so “the agent was operating without information the human already had”.

Those are not model-quality failures. They are architecture: one context, one prompt, one locked execution thread.

Supervisor plus sub-agents: pick a fan-out model, respect the BSP trap

PagerDuty names three ways to run sub-agents:

  1. Sequential — simple, but latencies compound; the 10+ minute diagnosis above is the cost.
  2. Parallel, wait for all — faster, but the orchestrator blocks until every tool in the batch resolves.
  3. Parallel fan-out and concurrent fan-in — what they shipped: the supervisor processes each result as it arrives, user input is a first-class event alongside sub-agent results, and new work is injectable mid-run. The costs are interrupt-and-resume execution, a buffer for concurrent arrivals, and prioritising user events over sub-agent results.

The trap is framework lock-in. LangGraph executes with a Bulk Synchronous Parallel (BSP) model in supersteps: a parallel tool call is one superstep, and control returns to the orchestrator only when every tool in the batch resolves. Consequences, verbatim: the orchestrator cannot react to sub-agent 1’s result at t=3min while others run; “No external event — including user input — can reach the graph while it’s blocked inside a parallel tool call”; “The tool set is fixed at the point of dispatch.” One hung sub-agent blocks synthesis indefinitely.

PagerDuty’s shipped architecture: supervisor and sub-agents in one process (the work is IO-bound, not compute-bound); the supervisor durable with checkpoints, sub-agents stateless and re-spawned rather than resumed; an in-process mailbox (asyncio.Queue) instead of webhooks, PubSub, or a durable event store; a task_id UUID convention where the LangGraph thread_id equals the agent’s task_id; and a priority queue where user input is priority 0 and sub-agent results priority 1. Their caution: verify polling cannot meet your latency needs before building a custom reactive loop — sub-agents run for minutes, so a few-second polling cycle is invisible.

Conviction grading beats confidence theatre

incident.io grades findings on a five-rung ladder (how investigations work):

Rung Label
1 Speculation (too little to anchor on)
2 Pattern match
3 Supported by context
4 Validated by system state
5 Validated, alternatives ruled out

Two overrides sit on top: contradicted, when newer evidence disproves a hypothesis, and flagged, when “an alternative is just as likely”.

The mechanism is the point. The agent reports top-of-ladder confidence only with compelling, verifiable evidence; otherwise it lowers confidence rather than dressing a guess up as a conclusion. A theory “that has only ever been supported, and never tested, isn’t one it trusts.” Discounted hypotheses stay in the audit trail rather than being dropped, and with too little information an investigation holds off rather than guessing — anti-hallucination by refusal.

Gating: skills, sandboxes, memory, and noise

Procedure gating. AWS ships custom skills targetable to agent types — verbatim “(On-demand, Incident Triage, Incident RCA, Incident Mitigation, Evaluation)” — plus learned skills. This is the fix for instruction overload: a scoped, loadable procedure instead of a monolithic prompt.

Noise gating. AWS’s Triage agent assesses severity and links duplicates with LINKED status — “Linked tasks won’t start automatically” — so one fleet-wide incident does not spawn a fleet of investigations.

Compute containment. incident.io runs all code analysis in isolated, sandboxed containers; repositories are cloned into ephemeral workspaces and deleted after use.

Identity, data, and network containment. AWS documents customer managed keys, IdP integration with Okta and Microsoft Entra ID, and private connections to VPC or internal-network services (MCP servers, self-hosted Grafana or Splunk, source control). PagerDuty scopes memory to a service and exposes a Memory API to view, update, and redact “at human speed, not machine speed” (PagerDuty SRE Agent docs); its nine nudge actions — “Upload Runbook”, “Update Runbook”, “Analyze Past Incidents”, “Analyze Related Incidents”, “Generate a Playbook”, “Check Change Events”, “Search Logs”, “Search Code”, “Update Memory” — are human-triggered edits, not autonomous writes.

The honest gap: no source describes approval gates, dry-run modes, blast-radius limits, or rollback for automated write actions. AWS documents mitigation plans with agent-ready instructions; PagerDuty documents recommended steps. These agents plan and recommend; the sources stop there, and so should your expectations. Mid-run steering and cancellation exist — the supervisor pattern takes user input at priority 0 and its lifecycle includes canceled — but autonomy over changes does not.

Agent-to-agent handoff is the seam — and the audit boundary

AWS’s mitigation plans include “agent-ready instructions that can be implemented by another frontier agent, such as code improvements that can be implemented by Kiro” (AWS docs). Because the service is a remote server over MCP, ACP, and A2A, another agent can invoke an investigation or a release-readiness check like an API call. Release readiness uses “deterministic proofs” that infrastructure changes do not expand permissions beyond what the application requires, reasons about blast radius from the service topology, and autonomous release testing generates change-specific test plans “rather than a static regression suite” — the same instinct as reviewing a Terraform plan before apply, enforced continuously.

incident.io’s desktop app pulls a live investigation into a local coding agent (Claude Code, Codex, Cursor) over MCP, and findings sent back are attributed in the incident channel. AWS’s integration list tells the same story: pre-GA it named CloudWatch, Datadog, Dynatrace, New Relic, Splunk, GitHub, GitLab, ServiceNow and Slack; GA added PagerDuty, Grafana (via a built-in Grafana MCP server), Azure DevOps, and Amazon EventBridge.

One agent investigates, another acts; the protocol is both the seam and the audit boundary.

Accuracy comes from tuned knowledge, not bigger prompts

Datadog’s docs name the four knowledge sources Bits reads during an investigation, verbatim: “bits.md, monitor messages and runbooks, and past feedback” (Bits Investigation docs). The tuning rules:

  • Rules, not descriptions. “A description repeats what Bits can already infer from telemetry, while a rule resolves something it can’t infer on its own, like a naming mismatch across tools.” Their example: alerts tag a service billing-svc while APM and logs use billing_service, so a rule treats them as one.
  • bits.md priorities, verbatim: “Cross-system name mapping”; “Known noise”; “Standing scope rules”.
  • Monitor hygiene. Put the dashboard, log query or notebook you would check first in the monitor message, plus downstream services; scope the query by service, because without that tag “Bits falls back on weaker signals like the monitor name”.
  • Prune ruthlessly. “A stale runbook link is worse than no link, since it points Bits at the wrong dashboard or a decommissioned service.”
  • Feedback quality. Corrections must name the root cause, the services and metrics, and link the proving telemetry — “‘That’s wrong’ gives Bits nothing to change”.
  • Signal dilution. Auto-Investigate on every monitor “spreads investigations across noisy, low-priority alerts and dilutes the signal you actually care about”; enable it on critical monitors only, and re-run a known investigation after any bits.md edit, because “Chat and investigations don’t always draw on knowledge the same way” (improve accuracy docs).

Build the eval harness you actually need

Datadog’s engineering blog describes how they learned to trust Bits (Bits AI eval platform). Every evaluation label has two parts: a ground truth (the root cause) and a world snapshot capturing the signals available when the issue occurred — for example root cause “Kubernetes pod OOM killed”, with the snapshot preserving the telemetry queries for memory metrics, container logs, and deployment events, “rather than raw data”. “The agent never sees the root cause directly. It only has access to the signals that existed when the issue occurred.”

A reusable label schema, graded on the five-rung ladder above (with contradicted and alternative-equally-likely forcing a downgrade or flag):

label_id: INC-2026-0142
ground_truth:
  root_cause: "Kubernetes pod OOM killed"
  component: payments-worker
  mechanism: "memory limit below observed peak after config change"
  verifiable_assertion: "container memory crosses limit at 14:03 UTC, before the error-rate spike"
world_snapshot:
  archived_signal_queries:            # pointers to queries, not raw data
    - {source: metrics, query: "container.memory.used{service:payments-worker}"}
    - {source: logs, query: "service:payments-worker status:oom"}
    - {source: deploy_events, query: "deployments{service:payments-worker}"}
  change_events: ["PR #4811 merged 13:47 UTC"]
  incident_channel_excerpt: "slack://payments-incident-2026-0142 (archived)"
confidence_grade: "Validated, alternatives ruled out"
disconfirming_checks:
  - "Replay with memory under the limit; if errors persist, downgrade to Pattern match"
  - "Check Kafka lag for an alternative consumer bottleneck"
owner: "oncall-payments"  # team accountable for keeping the label current

Three cheap rules made Datadog’s harness work:

  1. Labels grow from real feedback. “Every user interaction becomes a potential evaluation label”; embedding label creation in the product “increased our label creation rate by an order of magnitude”.
  2. The agent proposes ground truth before human review. Agentic pre-review cut “validation time per label… by more than 95% in a single week”.
  3. The dataset stays diverse and unclean. “From Kubernetes pod failures to Kafka lag, and from simple bad-code deployments to complex multi-service business logic side effects…” because “a narrow or overly clean dataset would inflate performance and hide weaknesses”.

Why off-the-shelf evals fail here, in Datadog’s words: “Off-the-shelf eval frameworks assume clean inputs and static test sets, which breaks down when your agent reasons across live production telemetry.” Tool-level tests missed interaction bugs — “the agent might retrieve valid signals from multiple tools but combine them incorrectly, leading it to attribute an issue to the wrong component”. Their scale: “Every week, we run Bits against tens of thousands of scenarios drawn from real incidents.”

Caveats. These figures describe Datadog’s private harness and publish no scores; no public benchmark exists in these sources for any of the four products.

What breaks first in production

  1. Context rot and instruction overload (PagerDuty’s v1) — fixed by a supervisor with sub-agents and skills scoped to agent types.
  2. Fan-out lock-in — a framework that blocks until every parallel tool resolves also blocks user input and synthesis.
  3. Stale documentation poisoning the agent — fixed by a runbook review cadence.
  4. Signal dilution — Auto-Investigate everywhere spreads investigations across noisy alerts.
  5. Useless feedback and memory rot — fixed by structured corrections plus periodic memory review.
  6. Thin-brief guessing and duplicate-ticket load — fixed by hold-off behavior, disconfirmation as half the work, and LINKED duplicate triage.

A rollout sequence that survives contact

  1. Bound the surface. One Agent Space, one service, or one set of critical monitors. Datadog’s dilution warning applies from day one.
  2. Wire knowledge and scoping. Connect telemetry, logs, runbooks and source control; group monitors by service; write rules, not descriptions; prune stale runbooks first.
  3. Replay a known incident. Take a closed postmortem — Google’s SRE Book on postmortem culture is the right mindset — and grade the agent’s path against your conviction ladder.
  4. Stand up the harness. Turn that replay and every later interaction into labels with archived world snapshots, per the schema above.
  5. Switch on the gates, then expand on evidence. Skills per agent type, sandboxed code analysis, memory review, LINKED triage, mid-run steering; weekly harness runs decide what widens.

FAQ

Do these agents fix things automatically? No source documents automated write actions. AWS produces mitigation plans containing “agent-ready instructions” another agent can implement; Paige recommends diagnostic and remediation steps. No approval gate, dry-run mode, blast-radius limit, or rollback mechanic is described — that gap is stated, not filled.

Can I self-host any of them? Nothing in these sources establishes self-hosted deployment of the agents themselves. AWS’s on-premises support reaches on-prem workloads through MCP — workload coverage, not agent self-hosting.

What do they cost? AWS states the model — per-second billing for time on operational tasks, no upfront commitments, and monthly credits for AWS Support customers as a percentage of their Support charge, varying by plan — but publishes no rates. The other three sources state no pricing at all.

The Bottom Line

Adopt the architecture pattern before you adopt any product: a supervisor with stateless sub-agents and concurrent fan-in, conviction grading with explicit overrides, curated rules-not-descriptions knowledge, sandboxed code analysis, memory governance at human speed, and a private eval harness fed by real incidents. As investigators, these agents are credible — the sources document structure, containment, and honest refusal. As actors, every source we read stops at plans, recommendations, and human-speed edits; in our reading, that is where a careful operator should stop in 2026. One buyer’s test: if a vendor cannot show you their conviction ladder or their evaluation labels, treat their accuracy claims as marketing.

How This Guide Was Built

This is a documentation-and-engineering-writing review, not a hands-on evaluation. Primary sources: the AWS Cloud Operations Blog GA announcement and the AWS DevOps Agent user guide; PagerDuty’s engineering write-up on the SRE agent and its SRE Agent documentation; incident.io’s Investigations documentation; Datadog’s Bits Investigation documentation and its Bits AI evaluation-platform engineering post; with Google’s SRE Book, Kubernetes probe documentation, and the Terraform CLI plan reference for context. Reviewed 2026-11-03. No product was run hands-on for this article. Where the sources are silent — pricing rates, write-action permissioning, public benchmark scores — this article says “not stated” rather than filling the gap.

📖 Related Reads

  • CodeIntel Log — code quality, debugging, and software engineering benchmarks
  • ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
  • Hermes Tutorials — Hermes Agent setup, configuration, and advanced workflows
  • NoCode Insider — AI workflow automation with no-code tools, agents, and APIs

Cross-links automatically generated from NiteAgent.

← Back to all posts