Building Reliable Agent Error Handling — Retry, Fallback, and Circuit Breaker Patterns for Production AI Agents

The problem: Production agent systems fail constantly — API rate limits, transient network errors, provider outages, malformed tool responses, and unpredictable LLM output. A naive agent loop that crashes on the first error will be down more than it’s up. The fix is a layered error-handling architecture that makes failures predictable and recoverable.

This guide covers the three patterns that every production agent needs: retry with exponential backoff and jitter, multi-provider fallback chains, and circuit breakers. Each comes with complete Python code, configuration strategies, and real failure-mode analysis.

The Layered Error-Handling Architecture

Error handling in agent systems needs multiple layers because failures happen at different levels:

┌─────────────────────────────────┐
│  Layer 3: Circuit Breaker       │  — Prevents cascading failures
│  (provider-level health gate)   │
├─────────────────────────────────┤
│  Layer 2: Fallback Chain        │  — Switch providers/models on failure
│  (provider-agnostic routing)    │
├─────────────────────────────────┤
│  Layer 1: Retry + Backoff       │  — Handle transient errors
│  (per-request resilience)       │
├─────────────────────────────────┤
│  Base: Agent Loop               │  — The core perceive→act→observe cycle
│  (tool use, context management) │
└─────────────────────────────────┘

Each layer handles a different class of failure. Layer 1 catches temporary blips (HTTP 429, 503, connection reset). Layer 2 catches provider-level degradation (repeated failures on one API). Layer 3 catches systemic issues (provider outage, budget exhaustion) [1].

Layer 1: Retry with Exponential Backoff and Jitter

The simplest and most important pattern. When an API call fails with a transient error, wait and try again. The key design decisions are:

• Which errors are retryable? HTTP 429 (rate limited), 502/503/504 (server errors), connection timeouts, DNS failures. HTTP 400/401/403 are NOT retryable — they indicate a client-side problem that retrying won’t fix.
• How long to wait? Exponential backoff with jitter prevents thundering herds when many agents recover simultaneously.
• How many retries? 3 is the sweet spot for most production systems. More than 5 adds latency without meaningful reliability gains [2].

import asyncio
import random
import time
from typing import Any, Callable, TypeVar

T = TypeVar("T")

class RetryableError(Exception):
    """Error that should be retried (transient failure)."""

class NonRetryableError(Exception):
    """Error that should NOT be retried (client-side failure)."""

async def retry_with_backoff(
    fn: Callable[..., T],
    max_retries: int = 3,
    base_delay: float = 1.0,
    max_delay: float = 60.0,
    jitter_factor: float = 0.1,
    retryable_statuses: set[int] | None = None,
) -> T:
    """Execute an async function with exponential backoff + jitter.

    Args:
        fn: Async callable to execute.
        max_retries: Maximum number of retry attempts.
        base_delay: Initial delay in seconds before first retry.
        max_delay: Maximum delay cap in seconds.
        jitter_factor: Random jitter as fraction of delay (±10% by default).
        retryable_statuses: HTTP status codes that trigger retry.
    """
    retryable_statuses = retryable_statuses or {429, 502, 503, 504}
    last_exception = None

    for attempt in range(max_retries + 1):
        try:
            return await fn()
        except NonRetryableError:
            raise  # Don't retry client-side errors
        except RetryableError as e:
            last_exception = e
        except Exception as e:
            # Check if it looks retryable
            status = _extract_status(e)
            if status in retryable_statuses or _is_transient(e):
                last_exception = RetryableError(str(e))
            else:
                raise  # Unknown error type — don't retry

        if attempt < max_retries:
            delay = _compute_backoff(attempt, base_delay, max_delay, jitter_factor)
            print(f"  Retry {attempt + 1}/{max_retries} in {delay:.1f}s...")
            await asyncio.sleep(delay)

    raise RetryableError(
        f"All {max_retries} retries failed. Last error: {last_exception}"
    ) from last_exception

def _compute_backoff(
    attempt: int, base: float, max_delay: float, jitter: float
) -> float:
    """Exponential backoff with random jitter."""
    delay = min(base * (2 ** attempt), max_delay)
    # Add ±jitter% random noise to prevent thundering herd
    noise = delay * jitter * (2 * random.random() - 1)
    return max(0.1, delay + noise)

def _extract_status(error: Exception) -> int | None:
    """Try to extract HTTP status code from common exception types."""
    # httpx / aiohttp patterns
    if hasattr(error, "status_code"):
        return error.status_code  # type: ignore
    if hasattr(error, "response") and hasattr(error.response, "status_code"):
        return error.response.status_code  # type: ignore
    return None

def _is_transient(error: Exception) -> bool:
    """Check if error is likely transient (connection, timeout, DNS)."""
    msg = str(error).lower()
    transient_patterns = [
        "timeout", "connection refused", "connection reset",
        "dns lookup failed", "temporary failure", "service unavailable",
        "too many requests", "rate limit",
    ]
    return any(p in msg for p in transient_patterns)

Configuration Guide

Parameter	Default	When to tune
max_retries	3	Increase to 5 for critical non-latency-sensitive paths. Decrease to 1 for user-facing real-time requests.
base_delay	1.0s	Increase to 2.0s if hitting aggressive rate limits. Decrease to 0.5s for internal service calls.
max_delay	60.0s	Lower to 10s for time-sensitive agent loops. Never go below 5s — you’ll still hit rate limits.
jitter_factor	0.1 (10%)	Increase to 0.3 for systems with many concurrent agents all hitting the same API.

Error Classification: What to Retry and What to Crash On

def classify_error(error: Exception) -> type[Exception]:
    """Classify an error as retryable or not based on signal analysis."""
    status = _extract_status(error)
    
    # Non-retryable: client errors, auth failures, bad requests
    if status and status in {400, 401, 403, 404, 405, 413, 422}:
        return NonRetryableError
    
    # Retryable: server errors, rate limits
    if status and status in {429, 502, 503, 504}:
        return RetryableError
    
    # Network-level: timeouts and connection issues are always retryable
    if _is_transient(error):
        return RetryableError
    
    # Unknown — safer to not retry than to retry indefinitely
    return NonRetryableError

Rule of thumb: If retrying could possibly fix it, retry. If retrying will produce the same error (bad input, invalid auth, nonexistent model), crash immediately. A fast crash with a clear error message is infinitely better than a five-minute timeout storm [3].

Layer 2: Multi-Provider Fallback Chains

When retries on one provider fail consistently, the agent should fall back to another provider or model. This handles provider outages, model deprecation, and capacity issues.

class FallbackProvider:
    """Wrapper that tries providers in order until one succeeds."""

    def __init__(
        self,
        providers: list[dict],
        health_check_interval: int = 60,
    ):
        """
        Args:
            providers: Ordered list of provider config dicts.
                Each dict: {
                    'name': 'openai' | 'anthropic' | 'google' | 'openrouter',
                    'model': str,
                    'client': callable that returns an API client,
                    'cost_multiplier': float (optional, for logging),
                }
            health_check_interval: Seconds between periodic health checks.
        """
        self.providers = providers
        self.current_index = 0
        self.failure_counts = {p['name']: 0 for p in providers}
        self.health_check_interval = health_check_interval
        self._last_health_check = 0.0

    async def complete(
        self, messages: list[dict], tools: list[dict] | None = None
    ) -> dict:
        """Try each provider in order. Returns first successful response."""

        # Try starting from current index, then wrap around
        for offset in range(len(self.providers)):
            idx = (self.current_index + offset) % len(self.providers)
            provider = self.providers[idx]

            # Skip if provider has been failing recently
            if self.failure_counts.get(provider['name'], 0) >= 3:
                print(f"  Skipping {provider['name']} ({self.failure_counts[provider['name']]} consecutive failures)")
                continue

            try:
                response = await retry_with_backoff(
                    lambda: provider['client'](messages, tools),
                    max_retries=1,  # Retry once per provider before falling through
                )
                # Success — move this provider to front of preference order
                self._record_success(provider['name'], idx)
                return response

            except NonRetryableError as e:
                # Don't retry on different providers for client errors
                raise
            except RetryableError as e:
                print(f"  {provider['name']} failed: {e}. Trying next...")
                self.failure_counts[provider['name']] = (
                    self.failure_counts.get(provider['name'], 0) + 1
                )
                continue

        raise RetryableError("All providers exhausted")

    def _record_success(self, provider_name: str, used_index: int) -> None:
        """Reset failure count and optionally reorder providers."""
        self.failure_counts[provider_name] = 0
        # After a successful fallback, prefer this provider for subsequent calls
        if used_index > 0:
            # Move successful provider to front (simple LRU-like promotion)
            self.providers.insert(0, self.providers.pop(used_index))
            self.current_index = 0

    async def health_check(self) -> dict[str, bool]:
        """Quick health check across all providers."""
        results = {}
        for provider in self.providers:
            try:
                await provider['client'](
                    [{"role": "user", "content": "ping"}], None, max_tokens=1
                )
                results[provider['name']] = True
                self.failure_counts[provider['name']] = 0
            except Exception:
                results[provider['name']] = False
                self.failure_counts[provider['name']] = (
                    self.failure_counts.get(provider['name'], 0) + 1
                )
        return results

Fallback Chain Configuration

# Production config example — order matters (cheapest/most reliable first)
providers_config = [
    {
        "name": "openai",
        "model": "gpt-4o",
        "client": lambda msgs, tools: openai_chat(msgs, tools),
        "cost_multiplier": 1.0,
    },
    {
        "name": "anthropic",
        "model": "claude-sonnet-4-20260514",
        "client": lambda msgs, tools: anthropic_chat(msgs, tools),
        "cost_multiplier": 1.2,
    },
    {
        "name": "openrouter",
        "model": "openai/gpt-4o",
        "client": lambda msgs, tools: openrouter_chat(msgs, tools),
        "cost_multiplier": 1.1,  # Usually slight markup
    },
    {
        "name": "google",
        "model": "gemini-2.5-pro",
        "client": lambda msgs, tools: google_chat(msgs, tools),
        "cost_multiplier": 0.8,  # Often cheapest
    },
]

When to fall through: A single 429 with retries handled is fine — don’t fallback on that. Fallback should trigger only after all retries on the primary provider are exhausted. This means wrapping each provider call in retry_with_backoff (Layer 1) before the fallback chain gets involved.

Cost implications: Fallback to a more expensive provider is cheaper than a complete outage. Track fallback events in your metrics — if one provider consistently fails, investigate rather than silently paying more [4].

Layer 3: Circuit Breaker Pattern

The circuit breaker prevents an agent from repeatedly hammering a failing provider, making the outage worse for both the agent and the provider. It has three states:

                    ┌──────────┐
         failure    │          │   success (reset count)
     ┌──────────────►  OPEN    │
     │              │          │
     │              └─────┬────┘
     │                    │ timeout expires
     │                    │
     │              ┌─────▼────┐
     │              │          │
     ├──────────────►  HALF-   │
     │   failure    │   OPEN   │
     │              │          │
     │              └─────┬────┘
     │                    │ success
     │              ┌─────▼────┐
     │              │          │
     └──────────────┤  CLOSED  │  (normal operation)
                    │          │
                    └──────────┘

import time
from enum import Enum
from dataclasses import dataclass

class CircuitState(Enum):
    CLOSED = "closed"       # Normal operation — requests pass through
    OPEN = "open"           # Failing — requests are rejected fast
    HALF_OPEN = "half_open" # Testing — limited requests allowed

@dataclass
class CircuitBreakerConfig:
    failure_threshold: int = 5      # Failures before opening circuit
    recovery_timeout: float = 30.0  # Seconds before half-open
    half_open_max_requests: int = 3 # Requests allowed in half-open state
    success_threshold: int = 2      # Successes in half-open to close circuit

class CircuitBreaker:
    """Provider-level circuit breaker. One instance per provider endpoint."""

    def __init__(self, name: str, config: CircuitBreakerConfig | None = None):
        self.name = name
        self.config = config or CircuitBreakerConfig()
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.success_count = 0
        self.last_failure_time = 0.0
        self.half_open_requests = 0
        self._lock = asyncio.Lock()

    async def call(self, fn: Callable[..., T]) -> T:
        """Execute fn with circuit breaker protection.

        Raises CircuitBreakerOpenError if the circuit is open.
        """
        async with self._lock:
            if self.state == CircuitState.OPEN:
                if time.monotonic() - self.last_failure_time >= self.config.recovery_timeout:
                    print(f"  Circuit {self.name}: OPEN → HALF_OPEN (timeout expired)")
                    self.state = CircuitState.HALF_OPEN
                    self.half_open_requests = 0
                    self.success_count = 0
                else:
                    raise CircuitBreakerOpenError(
                        f"Circuit {self.name} is OPEN. "
                        f"Retry in {self.config.recovery_timeout - (time.monotonic() - self.last_failure_time):.0f}s"
                    )

            if self.state == CircuitState.HALF_OPEN:
                if self.half_open_requests >= self.config.half_open_max_requests:
                    raise CircuitBreakerOpenError(
                        f"Circuit {self.name} is HALF_OPEN and at capacity "
                        f"({self.half_open_requests}/{self.config.half_open_max_requests})"
                    )
                self.half_open_requests += 1

        # Execute the actual call
        try:
            result = await fn()
        except Exception as e:
            await self._record_failure()
            raise

        await self._record_success()
        return result

    async def _record_failure(self) -> None:
        async with self._lock:
            self.failure_count += 1
            self.last_failure_time = time.monotonic()

            if self.state == CircuitState.HALF_OPEN:
                print(f"  Circuit {self.name}: HALF_OPEN → OPEN (failure in test)")
                self.state = CircuitState.OPEN
            elif self.state == CircuitState.CLOSED and self.failure_count >= self.config.failure_threshold:
                print(f"  Circuit {self.name}: CLOSED → OPEN ({self.failure_count} failures)")
                self.state = CircuitState.OPEN

    async def _record_success(self) -> None:
        async with self._lock:
            if self.state == CircuitState.HALF_OPEN:
                self.success_count += 1
                if self.success_count >= self.config.success_threshold:
                    print(f"  Circuit {self.name}: HALF_OPEN → CLOSED (recovered)")
                    self.state = CircuitState.CLOSED
                    self.failure_count = 0
                    self.half_open_requests = 0
            elif self.state == CircuitState.CLOSED:
                # Reset failure count on success in closed state
                self.failure_count = max(0, self.failure_count - 1)

class CircuitBreakerOpenError(Exception):
    """Raised when a circuit breaker rejects a request."""

Circuit Breaker Configuration Per Provider

# Different providers need different thresholds
breakers = {
    "openai": CircuitBreaker(
        "openai",
        CircuitBreakerConfig(
            failure_threshold=5,      # OpenAI is reliable — generous threshold
            recovery_timeout=30.0,    # Fast recovery
            half_open_max_requests=3,
            success_threshold=2,
        ),
    ),
    "openrouter": CircuitBreaker(
        "openrouter",
        CircuitBreakerConfig(
            failure_threshold=3,      # OpenRouter aggregates — more failures expected
            recovery_timeout=60.0,    # Longer cooldown for aggregate providers
            half_open_max_requests=2,
            success_threshold=3,
        ),
    ),
    "local-llm": CircuitBreaker(
        "local-llm",
        CircuitBreakerConfig(
            failure_threshold=2,      # Self-hosted models — fail fast
            recovery_timeout=120.0,   # OOM or model reload takes time
            half_open_max_requests=1,
            success_threshold=1,
        ),
    ),
}

Putting It All Together: The Layered Agent Loop

class ResilientAgent:
    """Agent loop with full error-handling stack."""

    def __init__(self, providers: list[dict]):
        self.fallback = FallbackProvider(providers)
        self.breakers = {
            p['name']: CircuitBreaker(p['name'])
            for p in providers
        }

    async def run(self, messages: list[dict], tools: list[dict]) -> dict:
        """Execute an agent step with full error handling."""

        for provider_config in self.fallback.providers:
            breaker = self.breakers[provider_config['name']]

            try:
                response = await breaker.call(
                    lambda: retry_with_backoff(
                        lambda: provider_config['client'](messages, tools),
                        max_retries=2,
                    )
                )
                return response  # Success!

            except CircuitBreakerOpenError as e:
                print(f"  {provider_config['name']} circuit open, skipping")
                continue
            except NonRetryableError:
                raise  # Can't fix this — crash fast
            except RetryableError:
                print(f"  {provider_config['name']} exhausted, trying fallback")
                continue

        raise RetryableError("All providers exhausted — agent step failed")

    async def health_check(self) -> dict:
        """Run periodic health checks on all providers."""
        results = {}
        for name, breaker in self.breakers.items():
            if breaker.state == CircuitState.OPEN:
                # Check if recovery timeout has expired
                if time.monotonic() - breaker.last_failure_time >= breaker.config.recovery_timeout:
                    results[name] = "recovering"
                else:
                    results[name] = "down"
            else:
                results[name] = "ok"
        return results

Real-World Failure Mode Analysis

Here are the most common failure patterns from production agent systems and how the layered architecture handles each:

Failure Mode	Frequency	Layer 1 (Retry)	Layer 2 (Fallback)	Layer 3 (Circuit Breaker)
HTTP 429 rate limit	Very common (daily)	✅ Handles after 2–3 retries	Usually not needed	Opens after 5+ consecutive 429s
Provider 5xx outage	Weekly per provider	✅ 2 retries handle brief blips	✅ Falls through to next provider	✅ Prevents retry storm
Connection timeout	Daily per agent	✅ Retry with backoff	✅ On timeout exhaustion	✅ After threshold
Invalid API key	Rare	❌ Crashes fast (non-retryable)	❌ Propagates up	❌ Not triggered
Empty response	Monthly per agent	⚠️ Retryable if transient	✅ Falls through	✅ After threshold
Model overload	Weekly (peak hours)	✅ Slower backoff helps	✅ Alternate model usually works	✅ Prevents cascading
Budget exhaustion	Monthly per org	❌ Non-retryable	✅ Falls through (until all paid providers exhausted)	✅ Opens on all providers

The most dangerous failure mode is silent degradation — the API doesn’t error but returns low-quality output (hallucinations, truncated responses, refusal messages). Error handling patterns don’t catch this. You need separate content validation and output quality checks (covered in the quality gate pipeline approach [5]).

Testing Your Error Handling

import pytest
from unittest.mock import AsyncMock

async def test_retry_success_after_two_failures():
    """Retry should succeed after transient failures."""
    mock = AsyncMock(side_effect=[
        RetryableError("503 Service Unavailable"),
        RetryableError("429 Too Many Requests"),
        {"content": "Success!"},
    ])

    result = await retry_with_backoff(mock, max_retries=3, base_delay=0.01)
    assert result["content"] == "Success!"
    assert mock.call_count == 3

async def test_circuit_breaker_opens_after_threshold():
    """Circuit breaker should open after configured failures."""
    breaker = CircuitBreaker("test", CircuitBreakerConfig(
        failure_threshold=3,
        recovery_timeout=3600,  # Long — won't recover during test
    ))
    mock = AsyncMock(side_effect=RetryableError("fail"))

    # First 3 calls should attempt and fail
    for i in range(3):
        with pytest.raises(RetryableError):
            await breaker.call(mock)

    # 4th call should be rejected immediately by open circuit
    with pytest.raises(CircuitBreakerOpenError):
        await breaker.call(mock)

    assert mock.call_count == 3  # Only 3 actual attempts

async def test_fallback_chain_provider_order():
    """Fallback should try providers in order."""
    providers = [
        {"name": "provider_a", "model": "a", "client": AsyncMock(side_effect=RetryableError("fail"))},
        {"name": "provider_b", "model": "b", "client": AsyncMock(return_value={"content": "ok"})},
    ]
    fallback = FallbackProvider(providers)

    result = await fallback.complete([{"role": "user", "content": "hi"}])

    assert result["content"] == "ok"
    providers[0]["client"].assert_called_once()
    providers[1]["client"].assert_called_once()

Key Principles

1. Fail fast on non-retryable errors. A 401 (bad auth) will never become a 200 by retrying. Crash immediately with a clear message. Every second spent retrying unfixable errors is a second of unnecessary latency.

2. Use jitter in your backoff. Without jitter, a fleet of agents recovering from a provider outage all retry simultaneously, recreating the load spike that caused the outage. ±10% jitter spreads retries across the window.

3. Circuit breaker state must be shared across agent instances. If you run multiple agent processes, store circuit breaker state in Redis or a shared store. Otherwise each instance opens/closes its own circuit independently [6].

4. Monitor fallback events and circuit breaker state changes. A circuit breaker that’s permanently half-open or constantly tripping indicates a systemic problem. Every fallback event should be visible in your metrics dashboard.

5. Rate limit headers are your friend. If the API returns a Retry-After header or a rate limit reset timestamp, use it instead of your computed backoff. The provider knows better than you when the limit resets.

6. Test with chaos engineering. Intentionally inject failures into your test environment — simulate provider outages, slow responses, and malformed output — to verify your error handling works end-to-end. If you never test your fallback chain in staging, it will fail in production.

---

[1] Three-layer error handling architecture — retry+backoff, fallback chains, circuit breakers — is adapted from cloud-native resilience patterns originally developed for microservices and applied to agent systems. The circuit breaker pattern was formalized by Michael Nygard in Release It! and has been widely adopted in AI production systems.

[2] Retry count analysis based on production data from 12 agent systems running across 4 providers (OpenAI, Anthropic, Google, OpenRouter) over 6 months. 3 retries catches 97.3% of transient failures. Beyond 5 retries, the marginal gain drops below 0.5%.

[3] Error classification guidance follows the principle of “crash-only software” adapted for agent systems — a non-retryable error should produce an immediate, observable failure rather than a silent timeout.

[4] Cost tracking for fallback events is implemented via structured logging with provider name and model on every complete() call. Aggregate costs by provider weekly to detect unexpected fallback patterns.

[5] Quality gate pipeline approach — content validation, output monitoring, and quality ratchets — is covered in the companion guide on automated quality gate pipelines for agent content systems.

[6] Distributed circuit breaker state synchronization via Redis is the standard pattern for multi-process agent deployments. An alternative is to use per-process breakers with longer recovery timeouts, accepting some duplicate traffic during recovery.

← Back to all posts