Multi-Tenant AI Agent Platforms: Isolation Patterns for the LLM Era
The Database Is Also an LLM — Why Multi-Tenancy Changes for Agent Platforms
The AWS SaaS Lens defines three multi-tenancy patterns — silo, pool, and bridge — driven by a service’s regulatory profile and noisy-neighbor attributes, but that framing assumes the shared resource is a database row store, not an LLM endpoint that generates per-token cost and consumes tenant-scoped credentials. In an agent platform the isolation surface expands to four coupled layers that must be isolated together: data, the shared model endpoint, tool credentials, and observability/cost. This post leads with a decision framework (pool vs silo vs bridge), then drills into each layer’s mechanics, so you can pick a pattern today without rebuilding tomorrow.
The Four Isolation Layers (Data, Shared Model Endpoint, Tool Credentials, Observability/Cost)
The four layers that must be isolated together are: (1) data — conversations, sessions, memory, vector stores, workflow state, governed by RLS, schema-per-tenant, or database-per-tenant; (2) the shared model endpoint — all tenants call the same provider APIs, so isolation moves to per-tenant keys, token budgets, and rate limits at a gateway or provider workspace; (3) tool credentials — agents call MCP servers and third-party APIs on behalf of tenants, where a leaked or confused credential is a cross-tenant breach; and (4) observability and cost — traces, token usage, and spend must be tenant-attributable or billing and debugging both break. The MCP Authorization spec (revision 2025-06-18) makes clear that credential isolation is not optional in the MCP era, and LiteLLM team budgets show how the gateway tier enforces token budgets.
Classic SaaS multi-tenancy partitions data rows, schemas, or databases across tenants, with the shared resource being storage and compute that you own and operate. In an agent platform, the shared resource is also an LLM endpoint that you do not own — it is cost-generating, credential-bearing, and rate-limited by a third party. As Cloudflare’s performance isolation post documents, pooled Postgres clusters suffer noisy-neighbor effects when one tenant’s burst of transactions starves neighbors of CPU, disk I/O, and connections — the same problem now applies to token budgets at an LLM endpoint. The isolation problem is structurally the same, but the currency and attack surface have expanded beyond your database into the provider’s API layer.
Pool, Silo, and Bridge — Choosing Your Tenancy Pattern
The AWS SaaS Lens defines three patterns: silo, where each tenant has a fully independent infrastructure stack or at minimum a separate database; pool, where tenants rely on shared, scalable infrastructure to achieve economies of scale; and bridge, a mixed mode where some microservices are siloed and some are pooled. The two drivers AWS names for pushing a service toward silo are its regulatory profile and its noisy-neighbor attributes, while agility, access patterns, and cost profile push toward pool. The decision is not binary — it is per-service, and it is the first architecture decision you make.
Silo Pattern — Dedicated Stacks, When Isolation Trumps Cost
In the silo pattern, each tenant gets a fully independent infrastructure stack — or at minimum a separate database — with identity, onboarding, and operations still shared across tenants. The AWS SaaS Lens treats silo as the isolation extreme, and AWS Tenant Isolation confirms that a dedicated database is the strongest physical separation. The cost is fleet sprawl, per-tenant migrations, and per-tenant capacity ceilings. Slack’s Vitess migration post (Dec 2020) is the canonical caution: a single large tenant could outgrow the largest shard, hot spots left the rest of the fleet underutilized, and a shard outage took every tenant on it down. Silo is reserved for regulated tenants, not the default.
Pool Pattern — Shared Infrastructure, Economies of Scale
In the pool pattern, tenants share scalable infrastructure to achieve economies of scale — the more classic notion of multi-tenancy. The AWS SaaS Lens frames pool as the cost-efficiency extreme, and AWS Data Partitioning treats shared constructs as the default for high tenant counts. The risk is the noisy neighbor: as Cloudflare’s performance isolation post documents, one tenant’s burst of transactions starved neighbors of CPU, disk I/O, and PgBouncer connections. Pool is the default for agent platforms because conversations, memory, and traces are naturally per-row scopable — but it requires enforcement at the gateway tier to prevent token-budget exhaustion.
Bridge Pattern — Letting Regulatory Profile and Noisy-Neighbor Behavior Decide per Service
The bridge pattern is mixed mode: some microservices siloed, some pooled, with the AWS SaaS Lens naming the two drivers as regulatory profile and noisy-neighbor attributes. The bridge is where most mature SaaS platforms land: pool the core data and the shared model endpoint, silo the card-data path or the tenant whose burst behavior destabilizes the fleet. The decision table later in this post reproduces this as a six-row comparison; the verdict column recommends bridge as the production sweet spot.
Data Partitioning Spectrum (Shared Construct → Separate DB)
The AWS SaaS Lens Data Partitioning doc frames the core choice as either a separate database per tenant or comingled data in a shared construct. That spectrum maps directly to the three data-isolation tiers: pooled shared schema with Row-Level Security (RLS) at the row level, schema-per-tenant in the middle, and database-per-tenant at the silo extreme. Each step up the spectrum trades cost and operational complexity for stronger isolation.
Data-Layer Isolation — RLS vs Schema-per-Tenant vs Database-per-Tenant
Row-Level Security makes policies act as an implicit WHERE clause on every query, the Postgres docs confirm, and the Supabase RLS guide adds the operational essentials: grants decide if a role can touch the table, policies decide which rows, and views bypass RLS by default so they must be audited separately. The Neon RLS guide shows the tenant pattern directly with a policy scoped to the JWT-derived tenant id injected per request, and Neon’s Data API requires RLS on all tables. Schema-per-tenant is the middle option — shared database, separate schema per tenant — and database-per-tenant is the strongest isolation tier, though as Slack’s Vitess migration post (Dec 2020) documents, it brings per-tenant capacity ceilings and hot spots.
-- Tenant-scoped RLS policy, attributed to Neon RLS guide
ALTER TABLE conversations ENABLE ROW LEVEL SECURITY;
CREATE POLICY "tenant_isolation"
ON conversations
FOR ALL
USING (tenant_id = current_setting('app.tenant_id')::text)
WITH CHECK (tenant_id = current_setting('app.tenant_id')::text);Row-Level Security — Per-Row Enforcement in Shared Tables
RLS is the default starting pattern for agent platforms because conversations, memory, and traces are naturally per-row scopable. The Postgres docs define the mechanism: policies act as an implicit WHERE clause on every query. The Supabase RLS guide adds the operational essentials — grants and policies are separate checks, enable RLS per table, write a policy per operation (USING for select/delete, WITH CHECK for insert/update), and watch the gotcha that views bypass RLS by default. RLS is defense in depth because it protects data reached through third-party tooling that does not enforce your application-level scoping.
Schema-per-Tenant — the Middle Ground
Schema-per-tenant is the middle option on the data-partitioning spectrum: a shared database with a separate schema per tenant, offering stronger isolation than shared tables with RLS but cheaper than a database per tenant. The tradeoff is migration and search-index complexity: N schemas to migrate, and cross-tenant search or analytics get harder because each tenant’s data lives in a different namespace. It is the right choice for mid-tier tenants needing cleaner separation than RLS without the cost of a silo, but it does not solve the shared model endpoint problem — that is the gateway tier’s job.
Database-per-Tenant — Strongest Isolation, Highest Operational Cost
Database-per-tenant is the strongest isolation tier, giving per-tenant restore/backup and capacity, but as Slack’s Vitess migration post (Dec 2020) documents, it brings per-tenant capacity ceilings and hot spots. The AWS Tenant Isolation doc confirms that a dedicated database is the strongest physical separation. Database-per-tenant is for enterprise or regulated tenants whose data profile or noisy-neighbor behavior justifies dedicated resources, not the default for a high-tenant-count agent platform.
The Shared Model Endpoint — Per-Tenant Keys, Token Budgets, and Rate Limits
All tenants call the same provider APIs, so isolation moves to a gateway or provider workspace layer where per-tenant keys, token budgets, and rate limits are enforced. The OpenAI projects & access guide defines Projects as the boundary for an application’s API usage, service accounts, rate limits, spend alerts, and project settings, with hard spend limits returning 429 with organization_spend_limit_exceeded or project_spend_limit_exceeded as the OpenAI spend limits doc confirms. The Anthropic rate limits doc and Anthropic Admin API show the equivalent: workspaces carry custom spend and rate limits, rate limiting uses the token bucket algorithm, and responses include anthropic-ratelimit-* headers plus an anthropic-workspace-id header showing which workspace the key resolved to.
The Gateway Pattern — LiteLLM Team Budgets as the Reference Implementation
The gateway pattern is the self-hosted complement to provider-side primitives, and it is mandatory for any multi-tenant agent platform regardless of data-layer pattern. The LiteLLM proxy implements exactly the per-tenant quota model: create a team with max_budget and budget_duration, mint keys bound to that team, set per-model TPM/RPM for teams, and export litellm_remaining_team_budget_metric to Prometheus. The layered model is: provider limits → gateway budgets → your own meter → Stripe metered billing invoices.
Stripe’s Four-Tier Rate Limiter as a Design Reference
The Stripe rate limiter post is the canonical production design for tenant protection at scale. It defines four tiers: a request rate limiter (N req/s per user), a concurrent-requests limiter (protects CPU-heavy endpoints), a fleet-usage load shedder (reserves capacity for critical methods, rejects with 503), and a worker-utilization load shedder (last line of defense). Implementation notes: token bucket over Redis, per-user buckets, fail-open middleware, kill switches, and dark-launch each limiter before enforcing. This is the design pattern to adapt for an LLM gateway: per-tenant token buckets at the gateway tier, with a fleet-usage shedder that reserves headroom for high-priority tenants.
Tool Credentials in the MCP Era — Avoiding the Confused Deputy
Agents call MCP servers and third-party APIs on behalf of tenants, and a leaked or confused credential is a cross-tenant breach. The MCP Authorization spec (revision 2025-06-18) mandates two hard rules for per-tenant credential safety: MCP servers must validate the token’s audience (reject tokens issued for other resources), and they must not pass through the client token to upstream APIs — pass-through creates the confused deputy problem. Each upstream call needs its own token, bound to the MCP server’s audience via resource indicators (RFC 8707). Per-tenant credential safety therefore means per-tenant OAuth client registrations plus token exchange at the gateway, never tenant A’s token forwarded to tenant B’s upstream service.
MCP Authorization Spec — OAuth 2.1, Audience Binding, No Passthrough
The MCP Authorization spec (revision 2025-06-18) defines the authorization flow as OAuth 2.1 with dynamic client registration (RFC 7591), authorization-server metadata (RFC 8414), and protected-resource metadata (RFC 9728). Clients must use resource indicators (RFC 8707) so access tokens are bound to the specific MCP server, and MCP servers must validate the token’s audience — rejecting tokens issued for other resources. For the full mechanics of how agents call tools safely, see our agent-ready API design guide.
Per-Tenant Credential Lifecycle and Secrets Management
Per-tenant tool credentials require a lifecycle that starts with OAuth client registration, moves through token exchange at the gateway, and ends with periodic rotation. The Anthropic Admin API shows the provider-side pattern: list keys per workspace, read expires_at, and rotate periodically. Secrets storage, such as HashiCorp Vault, is the ops complement, with rotation and lease-management primitives. One transport-level nuance from the MCP spec: HTTP transports should follow OAuth 2.1, but STDIO transports should not — instead, retrieve credentials from the environment, which means per-tenant credential injection via environment variables set per request. For the full server-side deployment picture, see our MCP server production deployment patterns guide.
Tenant-Aware Observability and Cost Allocation
Traces, token usage, and spend must be tenant-attributable, or billing and debugging both break. The OpenTelemetry GenAI semantic conventions (now maintained at the semantic-conventions-genai repo) define span attributes and metrics for LLM calls, agents, and MCP — the standard place to attach tenant_id plus token usage. Langfuse is the practical platform: it tracks usage and cost per LLM call with usage types (input, output, cached_tokens), supports ingested or inferred cost, and exposes a Metrics API for analytics, billing, and rate-limiting. The cost allocation pattern is: tenant_id on every span + usage from provider responses + a price table = per-tenant token cost, fed into Stripe metered billing. For the broader tracing setup, see our AI agent observability guide.
Temporal Namespaces for Workflow Isolation (with Multi-Tenant Caveat)
Temporal namespaces are a unit of isolation — workflow-ID uniqueness per namespace, and the docs state that heavy traffic from one namespace will not impact other namespaces, with per-namespace retention and archival config. But the docs caveat that a single namespace is still multi-tenant, so teams must coordinate workflow-ID and task-queue naming to avoid collisions. For agent platforms, the pattern is to map namespace-per-tenant or namespace-per-tier, and to store tenant_id as a workflow search attribute regardless. The blast-radius benefit is real — namespace isolation contains noisy workflows — but the multi-tenant caveat means you still need application-level scoping within a namespace, which is where RLS and the gateway tier do their work.
GDPR, PCI, and Compliance-Driven Isolation Decisions
GDPR cares about controls, not topology — DPA commitments, data residency, retention, and access governance can all hold in pooled designs. The OpenAI DPA (effective Jan 1, 2026) establishes OpenAI as a data processor with SCCs for EEA/UK transfers, sub-processor authorization, data-subject-request assistance, and return/deletion obligations. The OpenAI data controls doc is the practical layer: API data is not used to train OpenAI models since March 1, 2023 (unless opted in), default 30-day abuse-monitoring retention, Zero Data Retention configurable per organization and per project, and BYOK encryption. The Anthropic data residency doc provides the equivalent: per-request inference_geo, workspace-level allowed_inference_geos/default_inference_geo policies, and a workspace geo for data at rest. PCI commentary stays at the AWS SaaS Lens level — the regulatory profile of data steers a service toward the silo model — and readers should confirm scope with their QSA; for the security tooling layer, see our LLM security toolkit comparison.
Cloudflare’s Governed Data Platform as an Operational Governance Example
Cloudflare’s Town Lake / Skipper post (May 2026) shows tenant/account-aware governance at platform scale: default-closed tables (unqueryable until reviewed), automated PII detection, dynamic RBAC policies rendered for the query engine, time-bounded permission grants, auditable access, and an AI agent (Skipper) that answers questions with those controls enforced. The pattern maps to agent platforms: scope every query to tenant_id, render RBAC policies dynamically, and audit every access — because isolation is as much an access-control problem as a topology problem.
Case Studies — What Cloudflare and Slack Learned the Hard Way
Two real-world cases illustrate the noisy-neighbor and blast-radius problems that now apply to token budgets at an LLM endpoint. Cloudflare’s performance isolation post (Aug 2022) documents pooled Postgres clusters where one tenant’s burst of transactions starved neighbors of CPU, disk I/O, and PgBouncer connections. Slack’s Vitess migration post (Dec 2020) documents the workspace-sharded model’s capacity ceilings and blast radius. Both are the same problems, new currency — tokens instead of CPU.
Lessons for Agent Platforms (Tokens as the New CPU)
Cloudflare’s noisy-neighbor escalation — per-user connection limits, a PgBouncer fork with runtime per-user/per-pool throttling, and an experimental TCP-Vegas-inspired adaptive per-tenant concurrency controller — maps directly to an LLM gateway tier with per-tenant token budgets and per-model TPM/RPM caps. Slack’s tenant-sharded silo hitting capacity ceilings maps to the danger of database-per-tenant for agent platforms: a single large tenant’s conversation history can outgrow the largest shard. The bridge model resolves both: pool the core data with RLS, run an LLM gateway tier for per-tenant token budgets and key management, and escalate to database-per-tenant only where regulatory profile or noisy-neighbor behavior demands it. For the blast-radius implications, see our agent sandboxing field guide.
Decision Table — Choosing Your Isolation Pattern
| Pattern | Data isolation | Cost | Latency / Performance | Operational complexity | When to choose | Verdict |
|---|---|---|---|---|---|---|
| Pooled: shared schema + RLS | Row-level policies as implicit WHERE clause; policy bugs = cross-tenant read risk (Postgres RLS, Supabase) | Lowest — one fleet, shared capacity (AWS Data Partitioning) | Shared capacity; noisy-neighbor risk tamed with quotas/throttling (Cloudflare) | Low-to-moderate; one migration path; policies tested per operation; audit grants/views (Supabase) | Default for most agent platforms; high tenant counts; per-row scoping of conversations, memory, traces | ✅ Default starting pattern |
| Pooled: schema-per-tenant | Schema boundary per tenant; stronger than RLS, weaker than a DB (AWS Data Partitioning) | Low-to-moderate; shared instance | Shared instance; limited per-schema isolation | Moderate; N schemas to migrate/back up; cross-schema search harder | Mid-tier tenants needing cleaner separation than RLS without silo cost | — Niche middle ground |
| Silo: database-per-tenant | Strongest — dedicated DB, backups, capacity (AWS Tenant Isolation) | Highest — fleet cost grows per tenant; idle capacity (Slack) | Predictable per-tenant performance; no noisy neighbor | High; per-tenant migrations, monitoring, restore drills (Slack) | Enterprise/regulated tenants; per-tenant backup/restore SLAs; compliance-driven | 🔒 Escape hatch for regulated tenants |
| Bridge: hybrid pool + silo | Tiered — pooled core + silo islands for sensitive data/workloads (AWS Bridge) | Moderate; pay for silos only where needed | Mixed; must route per-tenant traffic correctly | High; two operating models, routing/data-flow complexity | Most mature SaaS; regulatory profile and noisy-neighbor attributes decide per service | ✅ Recommended for production |
| LLM gateway tier | Credential + quota isolation at proxy; does NOT isolate your DB (LiteLLM) | Direct pass-through of token cost; low gateway overhead | Central bottleneck if not scaled; enables per-tenant throttling (LiteLLM) | Moderate; run/scale a gateway, wire budget alerts + Prometheus metrics | Any multi-tenant agent platform — mandatory layer for token budgets | ✅ Mandatory complement |
| Tenant-keyed sharding / namespaces | Logical shard/namespace per tenant; physical sharing within shard (Temporal) | Moderate; good fleet utilization, hot spots possible (Slack) | Good at scale; per-shard blast radius (Slack) | High; shard rebalancing, metadata routing, namespace governance (Temporal) | Very large tenant counts and/or heavy workflow state | — Scale-stage pattern |
Recommended Hybrid: Pooled Data + LLM Gateway + Silo Escape Hatch
Start with pooled shared schema + RLS as the default, add an LLM gateway tier as the mandatory complement for per-tenant token budgets and key management, and escalate to database-per-tenant only where regulatory profile or noisy-neighbor behavior demands it — that is the bridge model most agent platforms should land on. For the trust-boundary implications of this hybrid, see our production prompt injection defense guide.
Frequently Asked Questions
Q1. Should I start with pooled RLS or database-per-tenant for an agent platform?
Start pooled with Row-Level Security unless a tenant’s regulatory or contractual profile forces dedicated infrastructure. RLS provides per-row enforcement inside Postgres with one fleet to operate. Add database-per-tenant as a bridge escape hatch for enterprise tenants whose data profile or noisy-neighbor behavior justifies dedicated resources. (AWS SaaS Lens, Postgres RLS)
Q2. How do I enforce per-tenant token budgets when all tenants share one model endpoint?
Route every model call through an LLM gateway that owns per-tenant keys. Assign each tenant a team with a max_budget and budget_duration, plus per-model RPM/TPM caps, then surface remaining budget as Prometheus metrics. Providers also expose limits: OpenAI enforces project-level token and spend limits; Anthropic lets workspaces carry custom spend and rate limits. (LiteLLM team budgets, OpenAI spend limits, Anthropic rate limits)
Q3. How should per-tenant tool credentials work for MCP-based agents?
Never let agents hold long-lived upstream tokens. Follow the MCP authorization spec: OAuth 2.1 flows with dynamic client registration, tokens bound to the MCP server’s audience via resource indicators, and no token passthrough to upstream APIs. Store provider API keys in a secrets manager, scope per tenant, rotate on expiry, and use STDIO transports only with environment-injected credentials. (MCP Authorization spec)
Q4. How do I attribute LLM spend to individual tenants?
Tag every trace with tenant_id and capture token usage from provider responses; both providers return usage plus rate-limit headers. Observability platforms like Langfuse store usage types (input, output, cached) with ingested or inferred prices. Aggregate via metrics APIs to feed billing, alerts, and rate-limiting decisions. (Langfuse cost tracking, OpenTelemetry GenAI semconv)
Q5. Can OpenAI Projects or Anthropic Workspaces serve as my tenant isolation layer?
Partially, and only for provider-side concerns. Projects and workspaces give per-tenant rate limits, spend caps, scoped keys, and residency controls. They do not isolate your own database, traces, or tool credentials. Treat them as the provider-facing tier of a bridge model, not the whole isolation story. (OpenAI projects & access, Anthropic Admin API)
Q6. Does GDPR force me into database-per-tenant?
No. GDPR cares about controls, not topology: DPA commitments, data residency, retention, and access governance can all hold in pooled designs. Providers offer region controls and zero-data-retention options at project or workspace level. Escalate to silo when a tenant’s regulatory profile — or your auditor — demands physical separation, following the bridge pattern. (OpenAI DPA, OpenAI data controls, Anthropic data residency)
The Bottom Line
Start pooled with RLS for data, add an LLM gateway tier for per-tenant token budgets and key management, adopt MCP OAuth for tool credentials, tag every trace with tenant_id for cost allocation, and escalate to silo only where regulatory profile or noisy-neighbor behavior demands it — that is the bridge model, and it is where most agent platforms should land. The gateway tier is non-negotiable regardless of data-layer pattern, because the shared model endpoint is the cost-generating surface that RLS cannot protect.
← Back to all postsHow This Guide Was Built This guide is based on official documentation and primary sources (AWS SaaS Lens, OpenAI and Anthropic platform docs, Stripe engineering, Cloudflare and Slack engineering blogs, the MCP specification) — all 30 source URLs were verified HTTP 200 on August 19, 2026. We did not run a live multi-tenant agent platform hands-on. All rate-limit values, pricing, and model-specific figures are volatile and intentionally omitted; consult the linked source docs for current numbers.
