Agent Context Window Management: Sliding Windows, Summarization, and Priority Eviction for Production Systems

The core problem: Every multi-turn agent session drifts toward context window overflow. Tool call results pile up, retrieved documents stack, conversation history accumulates — and the LLM starts dropping information silently. A production agent needs a structured context management strategy that decides what to keep, what to compress, and what to evict, all while staying within the model’s context budget.

This guide covers three battle-tested patterns for agent context window management — sliding windows, rolling summarization, and priority-based eviction — with Python implementations, token budgeting strategies, and configuration guidelines for production deployments.

Why Agent Context Management Is Different

Chatbot context management is simple: keep recent messages, drop old ones. Agent context management is harder because the content is heterogeneous — system prompts, conversation turns, tool call results (potentially large JSON blobs), retrieved documents, and intermediate reasoning steps all compete for the same limited window 1.

A typical agent turn might involve:

• System prompt: 500–2,000 tokens
• User message: 100–500 tokens
• Retrieved documents: 1,000–8,000 tokens
• Tool call + result: 200–4,000 tokens (results can be enormous)
• Assistant response: 200–1,000 tokens

After 5–10 turns, even a 200K context window fills up. The LLM doesn’t warn you — it just starts producing worse outputs as relevant context gets pushed out.

Strategy 1: Token-Aware Sliding Window

The simplest production pattern is a sliding window that maintains a fixed token budget and drops the oldest messages when the budget is exceeded.

Implementation

import tiktoken

class TokenSlidingWindow:
    """Fixed-budget sliding window for agent conversation history."""

    def __init__(self, max_tokens: int = 32_000, model: str = "gpt-4o"):
        self.max_tokens = max_tokens
        self.encoder = tiktoken.encoding_for_model(model)
        self.messages: list[dict] = []
        self._token_count = 0

    def count_tokens(self, text: str) -> int:
        return len(self.encoder.encode(text))

    def add(self, role: str, content: str, metadata: dict | None = None) -> None:
        """Add a message and evict oldest if over budget."""
        tokens = self.count_tokens(content)
        self.messages.append({
            "role": role,
            "content": content,
            "tokens": tokens,
            "metadata": metadata or {},
        })
        self._token_count += tokens
        self._evict()

    def _evict(self) -> None:
        """Drop oldest messages until under budget."""
        # Always keep the system prompt (assumed first message)
        while self._token_count > self.max_tokens and len(self.messages) > 2:
            removed = self.messages.pop(1)  # skip system prompt at index 0
            self._token_count -= removed["tokens"]

    def get_context(self) -> list[dict]:
        """Return messages formatted for the LLM API."""
        return [{"role": m["role"], "content": m["content"]} for m in self.messages]

Key configuration tips:

• Set max_tokens to 70–80% of the model’s rated limit to leave room for generation output 2
• Always pin the system prompt — it’s the most critical message and should never be evicted
• Track token counts accurately: tiktoken per-model encoders are more reliable than character-based estimates

Strategy 2: Rolling Summarization

When the sliding window drops too much useful context, rolling summarization compresses older messages instead of discarding them entirely.

Architecture

[Turn 1] [Turn 2] [Turn 3] ... [Turn N-5]
    ↓           ↓                  ↓
  Summary ← Summarizer LLM ←  Older turns
    ↓
[Summary] [Turn N-4] [Turn N-3] ... [Turn N]

The summarizer runs when the token budget is exceeded. It compresses all messages older than a threshold into a single summary message that preserves key facts, decisions, and action outcomes 3.

class RollingSummarizer:
    """Compresses older context into summaries instead of dropping it."""

    def __init__(
        self,
        summarizer_fn: callable,
        max_tokens: int = 48_000,
        summary_threshold: float = 0.75,
        model: str = "gpt-4o",
    ):
        self.max_tokens = max_tokens
        self.summary_threshold = int(max_tokens * summary_threshold)
        self.summarizer = summarizer_fn
        self.encoder = tiktoken.encoding_for_model(model)
        self.messages: list[dict] = []
        self.summary: str | None = None
        self._token_count = 0

    def add(self, role: str, content: str) -> None:
        tokens = len(self.encoder.encode(content))
        self.messages.append({"role": role, "content": content, "tokens": tokens})
        self._token_count += tokens

        if self._token_count > self.summary_threshold:
            self._compress()

    def _compress(self) -> None:
        """Summarize older messages, keep recent ones intact."""
        # Keep last 3 turns intact, summarize everything before that
        keep_count = 6  # 3 turns × 2 messages each (user + assistant)
        if len(self.messages) <= keep_count:
            return

        to_summarize = self.messages[:-keep_count]
        recent = self.messages[-keep_count:]

        summary_text = self._generate_summary(to_summarize)
        old_tokens = sum(m["tokens"] for m in to_summarize)
        new_tokens = len(self.encoder.encode(summary_text))
        saved = old_tokens - new_tokens

        self.summary = summary_text
        self.messages = [
            {"role": "system", "content": f"[Previous context summary: {summary_text}]", "tokens": new_tokens}
        ] + recent
        self._token_count = new_tokens + sum(m["tokens"] for m in recent)

    def _generate_summary(self, messages: list[dict]) -> str:
        """Call the summarizer LLM to compress message history."""
        text = "\n".join(f"{m['role']}: {m['content'][:500]}" for m in messages)
        prompt = (
            "Summarize the key facts, decisions, completed actions, "
            "and user preferences from this agent conversation history. "
            "Be concise. Include specific data points and outcomes:\n\n" + text
        )
        return self.summarizer(prompt)

Best practices:

• Run summarization when context reaches 75% of the budget, not at 100% — compression takes a turn and you need headroom 3
• Keep the last 2–3 interaction turns intact for immediate conversational coherence
• Include extracted facts, decisions, and action outcomes in the summary — not vague “the user discussed X” statements
• Benchmark summary fidelity periodically: sample compressed conversations and verify the LLM can still answer questions about the summarized portion

Strategy 3: Priority-Based Eviction

Not all context is equal. Tool call results from CRUD operations (which return large data dumps) are less valuable than the user’s stated preferences or a confirmed decision. Priority-based eviction scores each message and drops the lowest-value content first.

Priority scoring function

def score_message(msg: dict) -> float:
    """Score a message by its retention priority (higher = keep longer)."""
    role = msg.get("role", "")
    content = msg.get("content", "")

    # System prompt: highest priority
    if role == "system":
        return 100.0

    # User messages with explicit instructions
    if role == "user":
        priority = 50.0
        # Boost messages that look like instructions or preferences
        if any(word in content.lower() for word in ["remember", "always", "prefer", "never", "my name", "set"]):
            priority += 30.0
        # Boost short, dense user instructions
        if 50 < len(content) < 500:
            priority += 10.0
        return priority

    # Tool results: lowest priority unless marked critical
    if role == "tool":
        base = 10.0
        # Error results deserve retention (they're rare and important)
        if "error" in content.lower() or "exception" in content.lower():
            base += 40.0
        # Large results are costly to keep
        if len(content) > 2000:
            base -= 20.0
        return base

    # Assistant messages
    if role == "assistant":
        priority = 30.0
        # Boost messages with tool calls
        if "tool_calls" in msg or "</tool>" in content:
            priority += 15.0
        return priority

    return 20.0

class PriorityContextManager:
    """Evicts lowest-priority content when budget is exceeded."""

    def __init__(self, max_tokens: int = 48_000, model: str = "gpt-4o"):
        self.max_tokens = max_tokens
        self.encoder = tiktoken.encoding_for_model(model)
        self.messages: list[dict] = []

    def add(self, role: str, content: str) -> None:
        tokens = len(self.encoder.encode(content))
        msg = {"role": role, "content": content, "tokens": tokens}
        msg["priority"] = score_message(msg)
        self.messages.append(msg)
        self._evict()

    def _evict(self) -> None:
        """Evict lowest-priority messages until under budget."""
        total = sum(m["tokens"] for m in self.messages)
        if total <= self.max_tokens:
            return

        # Sort by priority (ascending), evict from bottom
        # Never evict system prompt (index 0)
        candidates = list(enumerate(self.messages[1:], start=1))
        candidates.sort(key=lambda x: (x[1]["priority"], -x[1]["tokens"]))

        to_remove = set()
        for idx, msg in candidates:
            if total <= self.max_tokens:
                break
            total -= msg["tokens"]
            to_remove.add(idx)

        self.messages = [
            m for i, m in enumerate(self.messages) if i not in to_remove
        ]

When to use priority eviction: Systems where context diversity is high — agents that switch between retrieval, code generation, and database queries in a single session. The priority system naturally retains user instructions and error results while discarding large but low-signal tool outputs 4.

Combining Strategies in Production

A production agent rarely uses just one strategy. The recommended architecture layers all three:

TokenSlidingWindow (budget enforcement)
  → RollingSummarizer (compression when budget tight)
    → PriorityEviction (fine-grained selection of what to drop)
      → LLM API call

class ProductionContextManager:
    """Layered context management for production agents."""

    def __init__(
        self,
        max_tokens: int = 48_000,
        summarizer_fn: callable = None,
        model: str = "gpt-4o",
    ):
        self.max_tokens = max_tokens
        self.sliding = TokenSlidingWindow(max_tokens, model)
        self.summarizer = RollingSummarizer(
            summarizer_fn or self._default_summarize,
            max_tokens,
            0.70,
            model,
        )
        self.priority = PriorityContextManager(max_tokens, model)
        self.encoder = tiktoken.encoding_for_model(model)

    def _default_summarize(self, text: str) -> str:
        """Minimal fallback summarizer — replace with an LLM call in production."""
        lines = text.split("\n")
        return "\n".join(lines[:5]) if len(lines) > 10 else text

    def add_turn(self, user_msg: str, assistant_msg: str, tool_results: list[dict] | None = None) -> None:
        """Add a full interaction turn through all three layers."""

        # Layer 1: Sliding window for basic budget enforcement
        self.sliding.add("user", user_msg)

        if tool_results:
            for tr in tool_results:
                self.sliding.add("tool", tr.get("content", str(tr)))

        self.sliding.add("assistant", assistant_msg)

        # Layer 2: Summarization at threshold
        if self.sliding._token_count > int(self.sliding.max_tokens * 0.7):
            # Feed sliding window messages into summarizer
            for msg in self.sliding.messages[1:]:  # skip system
                self.summarizer.add(msg["role"], msg["content"])

            # Layer 3: Priority eviction if still over budget
            if self.summarizer._token_count > self.summarizer.summary_threshold:
                for msg in self.summarizer.messages:
                    self.priority.add(msg["role"], msg["content"])

    def get_context(self) -> list[dict]:
        """Return the best available context."""
        if self.priority.messages:
            return [{"role": m["role"], "content": m["content"]} for m in self.priority.messages]
        if self.summarizer.messages:
            return [{"role": m["role"], "content": m["content"]} for m in self.summarizer.messages]
        return self.sliding.get_context()

Token Budget Planning

Set per-component budgets before your agent’s first turn:

Component	Budget (% of limit)	Notes
System prompt	5%	Keep under 2K tokens
Conversation history	20–30%	Sliding window allocation
Tool call results	15–20%	Most variable component
Retrieved documents	15–20%	Apply relevance filtering before insertion
Generation output	15–25%	Leave room for the model to write
Reserved headroom	10%	Buffer for unexpected growth

A 128K context window on GPT-4o or Claude 4 Opus should reserve ~100K for content and leave 28K for generation output 5.

Monitoring and Alerting

Add observability to your context management layer:

def log_context_stats(manager, turn_count: int) -> dict:
    """Log context window utilization for monitoring."""
    total = manager.sliding._token_count
    limit = manager.sliding.max_tokens
    return {
        "turn_count": turn_count,
        "tokens_used": total,
        "pct_used": round(total / limit * 100, 1),
        "message_count": len(manager.sliding.messages),
        "has_summary": manager.summarizer.summary is not None,
        "eviction_count": sum(
            1 for m in manager.priority.messages
            if m.get("priority", 0) < 20
        ),
    }

Alert when pct_used exceeds 90% consistently — it signals that your budget or eviction strategy needs tuning. Track token utilization per session in your metrics system (Prometheus, OpenTelemetry) to identify conversation patterns that overload the window 1.

Summary

The three patterns form a progression in sophistication and cost:

Strategy	Token overhead	Context fidelity	Implementation cost
Sliding window	None	Low (drops context)	Minimal
Rolling summarization	~200 tokens per summary	High (compresses)	Medium (LLM call per compression)
Priority eviction	None	Medium (drops low-value)	Medium (scoring logic)

Start with a token-aware sliding window on day one. Add rolling summarization when you observe context-dependent quality regression. Add priority-based eviction when you need fine-grained control over heterogeneous context.

---

1 Anthropic. “The Claude Context Window.” November 2024. 2 OpenAI. “Managing Context Windows with System Messages.” 2025. 3 LangChain. “How to manage conversation history with LLMs.” 2026. 4 Pinecone. “Context Management for AI Agents: Best Practices.” May 2026. 5 Zylos Research. “LLM Context Optimization: Budget Allocation Patterns for Production Agent Systems.” March 2026.

← Back to all posts