Checkpointing is not durable execution: keeping long-running AI agents alive

Checkpointing is not durable execution — and your agent will crash

Checkpointing snapshots graph or run state at step boundaries so a crashed agent can resume from the last saved point, but durable-execution engines journal every step’s result and replay completed steps from the log, preventing duplicate side effects and re-paid LLM calls — the difference determines whether your production agent survives a crash or silently breaks your customer’s data.

The naive-retry failure taxonomy

Every agent that talks to the outside world eventually hits a wall: a payment API that charges twice, an LLM call that re-runs and doubles your inference bill, an email that sends before the agent remembers it already sent it. These aren’t edge cases — they’re the default when you rely on naive retry loops around long-running, stateful agent workflows.

The failure taxonomy is short and brutal:

Duplicate side effects. An agent calls a payment API, then an LLM to draft a confirmation email, then crashes before sending. On resume, the payment call re-executes. The customer is charged twice.
Lost tool-call state. An agent sends an email but the crash happens before the result is recorded. On resume, the agent doesn’t know the email was sent. It sends again.
Orphaned external operations. An agent triggers a Kubernetes pod creation, then crashes. The pod runs, but the agent’s state says it never happened. Nobody cleans it up.
Re-paid LLM tokens. The same 12K-token prompt re-sends on retry. Your inference bill grows every time the agent crashes.
Non-deterministic retries. The LLM picks a different tool the second time. The agent’s behavior diverges from what you tested.

These are the failure modes that turn a promising agent demo into a production incident.

Why agents are “distributed systems on steroids”

Agents make roughly an order of magnitude more remote calls than a traditional service, and every one of those calls is failure-prone. That’s why resilience can’t be an afterthought — it has to be built into the execution model itself.

Temporal puts it bluntly: agents are “distributed systems on steroids,” and durable execution is the only way to survive the blast radius of that complexity. Source: Temporal AI blog

The cost of re-running LLM calls on retry

Inngest illustrates the compounding failure math with a simple example: if each of 5 steps has a 99% reliability rate, the overall workflow succeeds only 95% of the time. With 10 steps, it drops to 90%. And unlike stateless services, every retry in an agent workflow re-executes expensive LLM calls — meaning you pay for each call multiple times unless your execution model guarantees you don’t.

“Durable execution’s caching behavior means you pay for each LLM call exactly once.” — Inngest blog

How Framework Checkpointers Work Under the Hood

Framework checkpointers save snapshots of graph or run state at step boundaries — keyed by thread or session ID — so a crashed agent can resume from the last committed checkpoint, but they do not journal individual step results, meaning side effects can duplicate on replay unless you build idempotency yourself.

LangGraph — super-step snapshots and pending writes

LangGraph’s checkpointer model saves a snapshot of the full graph state at each super-step, organized by thread_id. It also persists per-node task writes, so if a super-step partially fails, the successful nodes in that step don’t re-run on resume. This is what enables HITL interrupts, time-travel debugging, and fault tolerance.

But there’s a catch: InMemorySaver loses everything on restart. Production deployments need PostgresSaver or SqliteSaver, which persist to a durable backend. Source: LangGraph checkpointers docs | persistence docs

CrewAI — event-driven checkpointing

CrewAI’s checkpointing (as of v1.15) is event-driven: snapshots are taken by default on task_completed events. You can store them via JsonProvider or SqliteProvider, and resume a run with crew.kickoff(from_checkpoint=...). You can also fork a run into a branch for experimentation. The CLI includes a TUI for inspecting and resuming checkpoints.

Source: CrewAI checkpointing docs

OpenAI Agents SDK — sessions and sandbox rehydration

The OpenAI Agents SDK provides two durability mechanisms. First, the Sessions layer persists conversation and run items across backends including SQLite, Redis, SQLAlchemy, MongoDB, Dapr, and OpenAI Conversations. Interrupted runs resume by reusing the same session. Second, sandbox agents provide persistent workspaces with snapshot and rehydration support — if a sandbox container dies, the agent’s state is restored in a fresh container from the last checkpoint.

As the April 2026 announcement states: “When the agent’s state is externalized, losing a sandbox container does not mean losing the run.” Source: Sessions docs | sandbox docs | announcement

How Durable-Execution Engines Work Under the Hood

Durable-execution engines persist every step’s input and result in an append-only event history or journal, so on crash the engine replays completed steps from the log — returning cached results instead of re-executing them — and resumes from the first incomplete step, guaranteeing side effects are never duplicated.

Temporal — event sourcing and Activity replay

Temporal’s model is event sourcing: every step in a workflow is recorded as an event in the workflow’s Event History, which is durably persisted by the Temporal service. Workers don’t store state — they rebuild it by replaying the history. When a worker crashes, the Temporal service schedules a new worker that replays the history and resumes from the point of failure.

LLM calls, tool invocations, and MCP client calls are wrapped in Activities, which have built-in retry policies. If an Activity fails, Temporal retries it according to the policy, and the result is cached in the event history — no re-execution on replay.

Source: Temporal AI docs | event docs | AI blog

DBOS Transact — Postgres-backed durability without a server

DBOS Transact is a library, not a server. It decorates Python, TypeScript, Go, Java, and Kotlin functions with @DBOS.workflow, @DBOS.step, and @DBOS.transaction to make them durable. Workflow and step state is persisted directly in your existing Postgres-compatible database — there’s no separate orchestrator to manage.

On restart, DBOS recovers pending workflows and resumes from the last completed step. Each workflow is recovered only once to prevent duplicate work. In distributed setups, Conductor failover handles recovery with a 60-second default executor timeout.

Source: DBOS Transact | workflow recovery docs | Python repo

Restate — journal, Virtual Objects, and idempotency keys

Restate wraps agent handlers in durable execution. Every step’s operation and result is recorded in a journal stored in the Restate Server (a Rust binary). On crash, the server replays the journal — skipping completed steps (LLM responses are not re-fetched, tool side effects are not duplicated) and resuming at the first incomplete step.

Restate also provides Virtual Objects — entities with single-writer guarantees powered by an embedded KV store. Idempotency keys in request headers deduplicate incoming requests, and handlers can suspend on FaaS while waiting for external events.

Note: Restate uses the Business Source License 1.1 (BSL 1.1), which is not an OSI-approved open-source license. Source: Restate repo LICENSE

Inngest and Hatchet — checkpointed steps and zero-compute suspend

Inngest splits code into checkpointed steps. Each completed step creates a checkpoint in a durable event log. On worker crash, the workflow replays from the last checkpoint without re-running prior steps. The step.waitForEvent function suspends the workflow with zero compute while waiting — for example, a 7-day approval timeout for HITL.

Hatchet follows a similar model: each completed chunk of a durable task creates a checkpoint in a durable event log. On worker crash, the task replays from that checkpoint without re-running prior steps. Hatchet explicitly targets agentic HITL loops and non-idempotent work.

Source: Inngest blog | Hatchet durable execution

The 5-Engine Comparison Table

The five engines differ fundamentally in how they persist state and guarantee side-effect safety — Temporal and DBOS journal every step for replay, Restate uses a per-entity journal with Virtual Objects, LangGraph snapshots graph state at super-step boundaries, and OpenAI Agents SDK persists run items in sessions with sandbox rehydration.

Engine	Persistence model	Resumability granularity	Side-effect guarantees	SDK language support	Pricing / open-source	Best for
Temporal	Append-only Event History per workflow, durably persisted by the Temporal service (event sourcing); state lives in the service, decoupled from workers — event docs	Step/Activity level — completed Activities replay from history without re-execution; workers rebuild state by replaying history; whole workflow survives crashes, deploys, multi-day HITL waits — AI docs	Exactly-once semantics for Activities via recorded results; automatic retries with policies; Saga pattern with idempotent compensations; Side Effects execute once — saga pattern	Python, Go, Java, TypeScript, .NET, Ruby, PHP — AI blog	MIT (server + SDKs); managed Temporal Cloud; $1,000 free credits promo — GitHub	Long-running stateful agent loops, HITL approvals, internal agent platforms, pipelines needing replay + audit trail
DBOS (Transact)	Workflow + step state persisted in your existing Postgres(-compatible) database via decorators; no separate orchestrator server — DBOS Transact	Step level — on restart, recovers pending workflows and resumes from last completed step; steps checkpointed so they don’t re-run once recorded; executor-ID-based recovery in distributed setups — recovery docs	Exactly-once via cached step outputs; `@DBOS.transaction` runs steps inside a Postgres transaction; recovery-once per workflow; duplicate work prevented — DBOS repo	Python, TypeScript, Go, Java, Kotlin — DBOS Transact	MIT library (runs anywhere) + DBOS Cloud/Conductor (optional ops layer) — DBOS Transact	Teams wanting durability inside their app + existing Postgres; LangGraph agents needing durable tools; serverless deployments
Restate	Journal of every step’s operation + result recorded in Restate Server (Rust, stream-processing, embedded KV store for Virtual Object/Workflow state) — key concepts	Step level — replays journal, skips completed steps (LLM responses not re-fetched), resumes at first incomplete step; handlers suspend on FaaS while waiting — durable agents	Exactly-once invocations; no duplicate tool side effects; idempotency keys in request headers deduplicate requests; single-writer Virtual Objects prevent races — key concepts	TypeScript, Java, Kotlin, Python, Go, Rust — key concepts	Source-available (BSL 1.1 — NOT OSI open source); Restate Cloud (managed) — key concepts	Stateful agent services, entity-per-user state (Virtual Objects), low-latency durable RPC
LangGraph checkpointer	Snapshot of graph state per super-step, keyed by thread_id, in a pluggable checkpointer (InMemory, Sqlite, Postgres, Redis, etc.); per-node task writes also persisted — checkpointers docs	Super-step (node) level — resume from last committed checkpoint; successful nodes in a failed super-step don’t re-run (pending-writes recovery); time-travel to any checkpoint — checkpointers docs	Framework-level: no engine-level exactly-once for arbitrary side effects — tools re-run on replay unless you add idempotency yourself; HITL via interrupts — persistence docs	Python and JavaScript/TypeScript — GitHub	MIT (LangGraph); checkpointers are OSS libs; LangSmith/Agent Server optional hosted layers — GitHub	Agent graphs inside LangChain ecosystem; conversational memory, HITL, time travel; when you control tool idempotency
OpenAI Agents SDK	Sessions = persisted conversation/run items (SQLite, Redis, SQLAlchemy, MongoDB, Dapr, and OpenAI Conversations backends); sandbox agents add workspace snapshots for state rehydration — Sessions docs	Run/turn level via sessions (resume an interrupted run with same session); sandbox-level: snapshot/rehydrate into a fresh container to continue from last checkpoint — sandbox docs	Sessions preserve context; sandbox snapshotting keeps filesystem/tool state; no engine-level exactly-once for external side effects — retries are app-level — Sessions docs	Python (sandbox/harness first; TypeScript planned) — GitHub	MIT (openai-agents-python); API usage billed via OpenAI (standard token/tool pricing) — GitHub	OpenAI-model agent loops, multi-turn chat with memory, coding/document agents in resumable sandboxes, quick production durability without new infra

Verdict: Temporal is the most mature for long-running, multi-step agent workflows with non-idempotent side effects. DBOS is the lightest lift for teams already on Postgres. Restate offers the strongest per-entity guarantees via Virtual Objects but uses BSL 1.1. LangGraph checkpointers are sufficient for graph-based agents where you control tool idempotency. OpenAI Agents SDK sessions are the fastest path for OpenAI-model agents needing basic resumability.

Code sketch: Temporal Activity wrapping an LLM call

from temporalio import workflow
from temporalio.workflow import retry

@retry(
    initial_interval=2.0,
    maximum_interval=10.0,
    maximum_attempts=5,
    non_retryable_error_types=[ValueError],
)
@workflow.defn
class LLMActivity:
    @workflow.run
    async def summarize(self, text: str) -> str:
        # This Activity is retried automatically by Temporal.
        # The result is cached in the Event History — on replay,
        # Temporal returns the cached result instead of calling the LLM again.
        import openai

        response = await openai.AsyncOpenAI().chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": f"Summarize: {text}"}],
            max_tokens=150,
        )
        return response.choices[0].message.content

Production Patterns That Make It Real

The five patterns that turn durable execution from theory into production reality are: checkpoint at tool-call boundaries, saga compensations for multi-step side effects, idempotency keys for external requests, timeouts and heartbeats for stuck operations, and workflow versioning with Continue-As-New to avoid event-history limits.

Checkpoint at tool-call boundaries

Every external call — whether to a payment API, an LLM, or a database — should be wrapped in a durable step. This ensures that if the agent crashes, the engine knows exactly which calls succeeded and which didn’t, and can replay from the right point without re-executing completed work.

This pattern is especially important for agents built with LangGraph + DBOS, where @DBOS.workflow-decorated tools combine with LangGraph’s PostgresSaver checkpointer to provide both graph-level snapshots and step-level durability. Source: DBOS blog

Saga compensations for multi-step side effects

The saga pattern handles multi-step workflows where each step has a side effect that must be undone if a later step fails. The rule is simple: register a compensation function before each step, and if any step fails, run the compensations in reverse order. Compensations must be idempotent — running them twice must have the same effect as running them once.

Temporal’s official saga pattern documentation covers this in detail. Source: Temporal saga pattern

Idempotency keys for external requests

When an agent calls an external API, the request should include an idempotency key — a unique identifier that the server uses to deduplicate requests. If the agent crashes and retries, the server returns the cached result from the first call instead of executing the side effect again.

Restate uses idempotency keys in request headers to deduplicate incoming requests at the framework level. Source: Restate key concepts

Timeouts, heartbeats, and HITL suspend/resume

Long-running operations — especially human-in-the-loop approvals — need suspend/resume primitives. The engine must be able to pause execution with zero compute while waiting for a human response, then resume when the response arrives.

Inngest’s step.waitForEvent suspends the workflow while waiting, and Temporal’s workflow-level timeouts allow a workflow to wait for days without consuming resources. Source: Inngest HITL docs

Continue-As-New and workflow versioning

Temporal workflows have hard limits on Event History size. When a workflow approaches those limits, Continue-As-New creates a fresh workflow execution with a clean history, passing the current state as the starting input. This is the standard pattern for long-running workflows.

Workflow versioning allows you to update a running workflow without breaking in-flight executions. Source: Temporal event docs

Decision Framework — When Is a Framework Checkpointer Enough?

A framework checkpointer is sufficient when your agent is short-lived, your tools are idempotent, and your LLM costs are low — but you need a durable-execution engine when your agent has non-idempotent side effects, long HITL wait times, expensive LLM re-runs, or audit and replay requirements.

The four decision axes are: (1) cost of re-run — cheap tool calls vs. expensive LLM inference; (2) side-effect risk — idempotent reads vs. non-idempotent writes like payments and emails; (3) HITL wait times — seconds vs. hours or days, which require durable suspend/resume; and (4) event-history limits — Temporal’s 51,200-event cap and Lambda’s 1-year max.

The four decision axes

Axis	Framework checkpointer threshold	Durable-execution engine threshold
Cost of re-run	Cheap tool calls, low LLM cost	Expensive LLM calls, high re-run cost
Side-effect risk	Idempotent tools only	Non-idempotent writes (payments, emails, provisioning)
HITL wait times	Seconds to minutes	Hours to days (requires suspend/resume)
Audit/replay	Not required	Required for compliance or debugging

Framework checkpointer sweet spot

If your agent runs for minutes, not hours; if your tools are idempotent or you can tolerate occasional duplicates; and if your LLM costs are low enough that re-runs don’t matter — a framework checkpointer is simpler and sufficient. LangGraph, CrewAI, and OpenAI Agents SDK all provide this level of durability.

Durable-execution engine sweet spot

If your agent has non-idempotent side effects (charging a customer, sending an email, provisioning infrastructure), if it waits for human approval for hours or days, if your LLM calls are expensive and you can’t afford to re-run them, or if you need audit trails and deterministic replay — you need a durable-execution engine. Temporal, DBOS, Restate, Inngest, and Hatchet all provide this level of durability.

For teams already deep in the LangChain ecosystem, our agent SDK comparison walks through the tradeoffs between framework checkpointers and engine-level durability.

The 2026 State of Play

2026 has been a pivotal year for durable execution in AI agents, with OpenAI adding snapshot/rehydrate to the Agents SDK, AWS shipping Lambda durable functions with custom SDKs, Temporal launching Serverless Workers on Lambda, and CrewAI shipping event-driven checkpointing.

OpenAI Agents SDK — snapshot/rehydrate and sandbox durability

In April 2026, OpenAI announced built-in snapshotting and rehydration for the Agents SDK. The key capability: “When the agent’s state is externalized, losing a sandbox container does not mean losing the run. With built-in snapshotting and rehydration, the Agents SDK can restore the agent’s state in a fresh container and continue from the last checkpoint.”

This decouples compute (the sandbox container) from state (the persisted session and workspace), so a container crash doesn’t lose the agent’s progress. Source: OpenAI announcement

AWS Lambda durable functions — custom SDKs and 1-year max

AWS Lambda durable functions shipped with built-in checkpointing and automatic retries, supporting executions that run for up to one year. In July 2026, AWS released custom Durable Execution SDKs for Node.js, Python, Java, and C#/.NET, along with open-source conformance tests.

This makes Lambda a viable platform for long-running agent workflows without managing separate orchestration infrastructure. Source: AWS announcement

Temporal Serverless Workers on Lambda and agent-framework integrations

In August 2026, Temporal announced Serverless Workers on Lambda, along with a Strands agent integration. The LENNY loan-underwriting demo demonstrates a Temporal workflow that survives a Worker crash mid-review — the workflow pauses, the Worker dies, a new Worker picks up the workflow from the Event History, and it resumes without losing state.

Temporal also claims a consumer-packaged-goods customer running Temporal on AWS Bedrock AgentCore cut costs by 66% while processing millions of events per month. Source: Temporal AWS blog

The broader ecosystem

CrewAI’s 2026 checkpointing is event-driven with resume and fork support. Microsoft’s Agent Framework combines AutoGen and Semantic Kernel with session-based state management for long-running and HITL scenarios. Inngest reports that durable execution “crossed the chasm into the early majority in late 2025,” with AWS Durable Functions, Cloudflare Workflows GA, and Vercel Workflow DevKit as evidence.

For teams evaluating the Microsoft stack, our Microsoft Agent Framework production guide covers session-based state management in depth.

Real Limits You Must Know Before You Ship

Every durable-execution engine has hard limits that will terminate your workflow if exceeded — Temporal’s event-history caps, AWS Lambda’s 1-year max, DBOS’s 60-second recovery timeout, and LangGraph’s in-memory data loss on restart are the ones that bite in production.

Temporal event-history limits

Temporal self-hosted logs a warning after 10,240 events in a workflow’s history. The workflow execution is terminated when the Event History exceeds 51,200 events, contains more than 2,000 Updates, or more than 10,000 Signals. There is no time limit on workflow duration — a workflow can run indefinitely as long as it stays under the event count.

Temporal Cloud adds a size constraint: the Event History is capped at 51,200 events or 50 MB, and any single Event History transaction is limited to 4 MB.

Continue-As-New is the standard workaround — it creates a fresh workflow execution with a clean history. Source: Temporal event docs | Cloud limits

AWS Lambda 1-year max and DBOS recovery semantics

AWS Lambda durable functions allow executions that run for up to one year. This is the longest-running durable execution available in a serverless context.

DBOS recovers interrupted workflows on restart (single server) or via Conductor failover (distributed setups). The default executor timeout is 60 seconds — if a workflow doesn’t resume within that window, it may be picked up by another executor. Each workflow is recovered only once to prevent duplicate work. Source: AWS announcement | DBOS recovery docs

LangGraph and OpenAI Agents SDK — what “durable” actually means for checkpointers

LangGraph’s InMemorySaver loses all data on restart. Production deployments must use PostgresSaver or SqliteSaver — the in-memory option is only for development.

The OpenAI Agents SDK’s Sessions layer provides resumability via SQLite, Redis, SQLAlchemy, MongoDB, Dapr, or OpenAI Conversations backends. Interrupted runs resume by reusing the same session. Sandbox agents add workspace snapshots for state rehydration into fresh containers.

But neither LangGraph nor the OpenAI Agents SDK provides engine-level exactly-once semantics for arbitrary tool side effects — you must add idempotency yourself. Source: LangGraph persistence | Sessions docs

The Bottom Line

Checkpointing saves state at boundaries so you can resume; durable execution journals every step so you can replay. For production agents with non-idempotent side effects, long HITL waits, expensive LLM calls, or audit requirements, a durable-execution engine is not optional — it’s infrastructure.

For short-lived agents with idempotent tools and low side-effect risk, a framework checkpointer is sufficient and simpler. The choice isn’t about features — it’s about the cost of being wrong. If a duplicate payment, a lost email, or a re-paid LLM call would be a business incident, you need the guarantees that come from event-sourced replay and journaled step results.

Choose your durability layer accordingly.

FAQ

Q: What is the difference between checkpointing and durable execution for AI agents?

Q: Can LangGraph checkpointers provide the same guarantees as Temporal?

LangGraph checkpointers save super-step snapshots and persist per-node task writes, but they do not provide engine-level exactly-once for arbitrary tool side effects — you must add idempotency yourself, whereas Temporal journals Activity results for automatic replay without re-execution. Source: LangGraph checkpointers | Temporal AI docs

Q: What are the hard limits of Temporal’s event history?

Temporal self-hosted warns at 10,240 events and terminates at 51,200 events, more than 2,000 Updates, or more than 10,000 Signals; Temporal Cloud adds a 50 MB cap and 4 MB per transaction; Continue-As-New avoids these limits, and there is no time limit on workflow duration. Source: Temporal event docs | Cloud limits

Q: How does AWS Lambda durable functions compare to Temporal for agent workflows?

AWS Lambda durable functions provide built-in checkpointing and retries with executions up to 1 year, while Temporal offers event-sourced replay, Saga patterns, and multi-language SDKs with no time limit on workflow duration — Lambda is serverless-first, Temporal is self-hosted or Cloud-managed. Source: AWS announcement | Temporal event docs

Q: When should I use a durable-execution engine instead of a framework checkpointer?

Use a durable-execution engine when your agent has non-idempotent side effects like payments or emails, long HITL wait times of hours or days that require suspend/resume, expensive LLM re-runs that double your inference bill, or audit and replay requirements for compliance — framework checkpointers suffice only for short-lived agents with idempotent tools and low side-effect risk. Source: Inngest HITL docs | Temporal AI docs

Q: Is Restate open source?

No — Restate uses the Business Source License 1.1 (BSL 1.1), which is not an OSI-approved open-source license; the source code is available for inspection and modification, but production use may require a commercial license depending on your use case, and Restate Cloud is the managed offering. Source: Restate repo LICENSE

How This Guide Was Built

This architecture deep dive is based on official documentation from Temporal, DBOS, Restate, LangGraph, OpenAI Agents SDK, AWS, Inngest, Hatchet, and CrewAI, supplemented by vendor engineering blogs. We did not run these engines hands-on in a benchmark. All performance numbers and scale claims are attributed to their sources and are vendor-published, not independently audited. The comparison table reflects documented capabilities as of August 2026.

ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
CodeIntel Log — code quality, debugging, and software engineering benchmarks
Hermes Tutorials — Hermes Agent setup, configuration, and advanced workflows

Cross-links automatically generated from NiteAgent.

← Back to all posts