Every voice agent has exactly one moment the caller experiences: the gap between the end of their sentence and the first audible sound of the reply. The AWS 800ms budget and the DestiLabs fleet report bookend that gap at the same two events — the clock starts when the user stops speaking and ends when the agent starts speaking. Everything between those two events is a ledger of stages, and a vendor claim is a fourth evidence class — labelled every time it appears.
This post audits that ledger using three distinct classes of evidence, plus vendor claims — labelled every time they appear:
- Design targets — budgets someone designed against, self-labelled as targets, not measurements. The AWS budget is the primary example.
- Measured fleet telemetry — p50/p95 numbers reported from live deployments. DestiLabs reports this for its own fleet; practitioner-measured, not peer-reviewed.
- Independently measured model benchmarks — third-party timings applied uniformly across vendors. Coval’s time-to-first-audio board is the example.
If you do not have a working agent yet, start with How to Build a Voice Agent From Scratch; the loop design and barge-in companion to this post is Production Voice Agents: Architecture and Barge-In.
The five-line budget, as published
The AWS 800ms rule states plainly that its numbers are “the targets I design against, not measurements from one deployment.” Reproduced as published:
| Stage | Target (ms) | Class of number |
|---|---|---|
| Endpointing (VAD) — deciding the user finished | 150–300 | Design target |
| ASR final transcript — streaming flush to final | 100–200 | Design target |
| LLM time to first token — prompt processing + first token | 300–600 | Design target |
| TTS time to first byte — synthesis streaming out | 100–250 | Design target |
| Network + telephony — region hop, SIP leg, jitter buffer | 50–150 | Design target |
| Sum of the rows | 700–1,500 | Arithmetic |
| Stated total in the same article | “roughly 800ms to 1.5 seconds” | Design target |
The arithmetic: 150 + 100 + 300 + 100 + 50 = 700 ms at the low end; 300 + 200 + 600 + 250 + 150 = 1,500 ms at the high end. The high end matches the stated 1.5-second ceiling exactly. The low end does not — the row-wise minima sum to 700 ms while the stated floor is 800 ms, leaving 100 ms of the quoted floor unaccounted for by the rows themselves. The same article separately says a stitched pipeline is “typically 800ms or more,” which agrees with the stated total but not with the row minima. We flag that rather than reconcile it.
For context, the 800 ms line sits above the human baseline: conversational turn-taking research puts the average human response gap around 200 ms with a natural range up to 500 ms (WebRTC Ventures), and humans “start to feel a gap at roughly 500ms of silence” (AWS).
Stage 1: endpointing costs more than the model does
Endpointing gets the smallest headline number and the largest hidden one. The target is 150–300 ms (design target, AWS), but the configuration range that produces it is wide. A 700 ms silence threshold “feels safe in testing and feels awful on a real call” (AWS, design-target article). The workable pattern it prescribes is an aggressive threshold plus a cancel path — not a conservative threshold.
Two configuration facts change this stage by more than any model swap:
- Local VAD beats remote VAD by 150–200 ms. Pipecat’s speech-input guide states “Use local VAD: 150-200ms faster than remote VAD services” — a framework-published figure, not a third-party benchmark. The same source gives Silero VAD as processing “30+ms audio chunks in less than 1 ms,” also framework-published.
- The stop threshold is a config key, not a constant. Pipecat’s
stop_secsdefault of 0.2 s exists because “a short value (0.2s) allows STT services to finalize sooner” (Pipecat). LiveKit’s turn-detection docs exposemin_delayandmax_delay, plus mode choices:Turn detector model(default),Realtime models,VAD only“when you need minimal latency,” andSTT endpointingfor vendors like AssemblyAI or Deepgram Flux. The same doc records the Realtime API’s own turn-detection fields —threshold,prefix_padding_ms,silence_duration_ms,eagerness, andinterrupt_response— and its telephony example setsthreshold=0.7,prefix_padding_ms=300, andsilence_duration_ms=400for noisy phone audio.
The verdict: endpointing is the one stage where a single integer, chosen once and never instrumented, can consume a quarter of your entire turn.
Stage 2: transcription, and the end-of-turn shortcut
The target for an ASR final transcript is 100–200 ms (design target, AWS) — that is the cost of flushing the last partial into a final, not of streaming recognition itself. Vendor numbers for streaming transcription sit higher: Deepgram’s latency docs claim models “optimized to deliver transcription latency in 300 milliseconds or less for streaming workloads,” and Nova-3 delivers “sub-300 ms streaming latency.” Both are vendor claims, unverified here. There is no per-provider STT leaderboard in this dossier — AssemblyAI’s streaming docs describe the event model (open / turn / close, "end_of_turn": true, "end_of_turn_confidence") with no latency figures at all.
The interesting lever is not transcript speed — it is end-of-turn detection. Deepgram’s Flux claim: “Flux STT can reduce agent response latency by 200–600 ms compared to traditional STT+VAD approaches by detecting turn endings earlier and more accurately.” That is a vendor claim with no independent measurement in any source here, but WebRTC Ventures independently endorses the architectural idea as “a genuine architectural win, not just faster hardware,” and Deepgram’s EagerEndOfTurn lets you “start preparing LLM responses before the turn definitively ends” (qualitative).
The measurement advice that follows: measure end-of-turn latency, not just transcript latency. Deepgram’s own doc concedes end-of-turn latency “is difficult to measure precisely without ground truth timestamps” and approximates it with VAD. A dashboard that only reports transcript finalisation measures the easy half of stage 2.
Stage 3: the LLM’s first token
The target band is 300–600 ms for time to first token (design target, AWS), described as “prompt processing plus the first token.” WebRTC Ventures rates this stage “Large — the other biggest single contributor; varies enormously by model size and hosting,” and notes that Groq and Cerebras compress TTFT with custom inference silicon.
TTFT matters because the listener does not wait for a complete answer — they wait for the first sound of one, and streaming tokens into TTS is the only way stages 3 and 4 overlap. Total generation time is a cost metric; first token is a latency metric.
This dossier does not establish a TTFT figure for any named model. What is established is the lever: “Long conversation history inflates time to first token on every single turn. Summarise past turns instead of replaying them” (AWS) — the context-trimming lever, covered in depth in Agent Memory Systems: A Production Guide.
Stage 4: TTS, where the vendor clock and the listener’s clock diverge
This is the stage where three number classes collide, so keep them separate.
Design target. TTS time to first byte: 100–250 ms (AWS).
Independently measured. Coval’s TTFA board times 32 models over the 30 days to 28 October 2026, “from the moment we send the text to the first sound a listener could hear.” Four models averaged under 100 ms; the fastest, vui (Fluxions), hit 43 ms; the median model took 245 ms; the slowest, Fish Audio S2.1 Pro Free, took 826 ms — a 19.2x spread across the board.
Vendor claim. Cartesia Sonic claims “first audio in under 90ms” and “sub-90ms model latency,” in 44 languages from one model.
Coval measures Sonic 3.5 at 291 ms and Sonic 3.6 at 292 ms, ranking them 20th and 21st of 32 — roughly 3.2x the vendor figure. Both can be true. Coval states the problem directly: “TTS vendors publish latency numbers for it, but they don’t all start and stop the clock in the same place.” Cartesia says model latency; Coval times text-in to first audible sound, which includes network transit and leading silence. This is a definitional mismatch layered on top of a magnitude gap — so do not average 90 and 291, and do not declare either wrong.
The leading-silence problem makes the case concrete. Coval’s split table puts Google’s Chirp 3 HD at 185 ms to first byte and 402 ms of leading silence, a 588 ms TTFA that a first-byte stopwatch would report as 185 ms — while Cartesia’s Sonic 3.5 splits 160 ms first byte against 131 ms of leading silence.
Stage 5: transport and playout
The AWS target for “Network and telephony — region hop, SIP leg, jitter buffer” is 50–150 ms (design target). WebRTC Ventures puts the physical floor at “roughly 20-30ms one-way for a well-routed session” — below that “there is nothing left to optimize” — and notes that a vendor’s advertised low “WebRTC latency” usually means transport plus jitter buffering, “a small fraction of the total budget.” Jitter buffer and playout are rated “Small,” tunable against glitching, with no millisecond figure published.
Two costs are real but bounded: a far-region model “can quietly cost 100ms per turn” (AWS), and telephony’s narrowband leg adds its own framing. Twilio Media Streams contributes no latency figure — only the fork-budget constraint: “Plan your fork budget across all active features on the call.”
This is why the network is the wrong place to optimise first. Co-locate STT, LLM, and TTS with your media servers, collect the 100 ms region hop, and stage 5 becomes a rounding error next to stages 1 and 4.
What the arithmetic shows
The constants in the script below are quoted from the cited sources fetched on 2026-11-13, and the script only adds, divides and compares them — it is arithmetic over published figures, not a measurement of any pipeline. Nothing here was run against a live voice system.
#!/usr/bin/env python3
"""Latency-budget arithmetic and reconciliation for the NiteAgent voice-latency post.
Every input constant below is quoted from a source fetched on 2026-11-13; the
source id is in the label. Nothing here is measured by us: the script only adds,
divides and compares published numbers.
aws = https://builder.aws.com/content/3JDFAfXBiuwPP5MPzgf4RUWSAIp/the-800ms-rule-budgeting-latency-for-a-real-time-voice-agent-on-aws
dg = https://developers.deepgram.com/docs/measuring-streaming-latency
coval = https://www.coval.ai/blog/tts-latency-benchmark-2026/
desti = https://www.destilabs.com/blog/ai-voice-agent-benchmark-2026
"""
BUDGET_TARGET_MS = [
("Endpointing (VAD)", 150, 300, "aws"),
("ASR final transcript", 100, 200, "aws"),
("LLM time to first token", 300, 600, "aws"),
("TTS time to first byte", 100, 250, "aws"),
("Network + telephony", 50, 150, "aws"),
]
DEEPGRAM_COMPONENTS_MS = [
("Network transit time", 20, 200, "dg"),
("Transcription latency", 150, 300, "dg"),
]
DEEPGRAM_STATED_TOTAL_MS = (200, 500, "dg")
# Coval 30-day averages, fastest / board median / slowest
TTFA_FASTEST_MS = (43, "vui (Fluxions)", "coval")
TTFA_MEDIAN_MS = (245, "board median of 32 models", "coval")
TTFA_SLOWEST_MS = (826, "Fish Audio S2.1 Pro Free", "coval")
CEILING_MS = (800, "aws")
# DestiLabs per-project p50 latencies as listed in the article's fleet table
FLEET_P50_MS = [540, 720, 690, 610, 560, 740, 660, 700, 580, 810, 630, 760]
FLEET_STATED = {"p50": 680, "p95": 1180}
STATED_BANDS = [("under 600", 30), ("600-800", 40), ("800-1000", 20), ("over 1000", 10)]
def band_of(ms):
if ms < 600:
return "under 600"
if ms < 800:
return "600-800"
if ms < 1000:
return "800-1000"
return "over 1000"
def main():
print("== 1. AWS per-stage budget targets (published as targets, not measurements) ==")
for name, lo, hi, src in BUDGET_TARGET_MS:
print(f" {name:<26} {lo:>4}-{hi:<4} ms [{src}]")
lo = sum(r[1] for r in BUDGET_TARGET_MS)
hi = sum(r[2] for r in BUDGET_TARGET_MS)
print(f" {'SUM':<26} {lo:>4}-{hi:<4} ms article states 'roughly 800ms to 1.5 seconds'")
print(f" low-end gap vs the quoted 800 ms floor: {800 - lo} ms of the stated floor is not covered")
print(f" high end matches the quoted 1.5 s ceiling exactly: {hi == 1500}")
print("\n== 2. Deepgram client-side streaming components ==")
for name, dlo, dhi, src in DEEPGRAM_COMPONENTS_MS:
print(f" {name:<26} {dlo:>4}-{dhi:<4} ms [{src}]")
dlo = sum(r[1] for r in DEEPGRAM_COMPONENTS_MS)
dhi = sum(r[2] for r in DEEPGRAM_COMPONENTS_MS)
print(f" {'SUM (no LLM, no TTS)':<26} {dlo:>4}-{dhi:<4} ms")
print(f" doc's own stated total transcript latency: {DEEPGRAM_STATED_TOTAL_MS[0]}-{DEEPGRAM_STATED_TOTAL_MS[1]} ms"
f" -> component sum agrees: {dlo >= DEEPGRAM_STATED_TOTAL_MS[0] - 30 and dhi <= DEEPGRAM_STATED_TOTAL_MS[1] + 30}")
print("\n== 3. TTS share of a single 800 ms turn (measured TTFA, Coval) ==")
for label, (ms, name, src) in (
("fastest model", TTFA_FASTEST_MS),
("board median", TTFA_MEDIAN_MS),
("slowest model", TTFA_SLOWEST_MS),
):
share = 100.0 * ms / CEILING_MS[0]
print(f" {label:<14} {name:<28} {ms:>4} ms = {share:5.1f}% of the {CEILING_MS[0]} ms budget")
spread = TTFA_SLOWEST_MS[0] - TTFA_FASTEST_MS[0]
print(f" fastest-to-slowest spread on the measured board: {spread} ms ({TTFA_SLOWEST_MS[0]/TTFA_FASTEST_MS[0]:.1f}x)")
print("\n== 4. Fleet tail ratio (DestiLabs telemetry) ==")
print(f" p50 {FLEET_STATED['p50']} ms, p95 {FLEET_STATED['p95']} ms -> p95 is {FLEET_STATED['p95']/FLEET_STATED['p50']:.2f}x the median turn")
print(f" headroom to the 800 ms line at p50: {800 - FLEET_STATED['p50']} ms; p95 exceeds it by {FLEET_STATED['p95'] - 800} ms")
print("\n== 5. Reconciliation: stated latency bands vs the 12 listed fleet rows ==")
counts = {b: 0 for b, _ in STATED_BANDS}
for ms in FLEET_P50_MS:
counts[band_of(ms)] += 1
print(f" rows in the article table: {len(FLEET_P50_MS)} p50 values: {sorted(FLEET_P50_MS)}")
print(f" {'band':<12}{'computed share':>16}{'stated share':>14}")
for band, stated in STATED_BANDS:
computed = 100.0 * counts[band] / len(FLEET_P50_MS)
print(f" {band:<12}{computed:>15.0f}%{stated:>13}%")
computed_median = sorted(FLEET_P50_MS)[len(FLEET_P50_MS) // 2 - 1: len(FLEET_P50_MS) // 2 + 1]
print(f" median of the listed rows (mean of middle pair {computed_median}): "
f"{sum(computed_median)/2:.0f} ms vs stated fleet p50 {FLEET_STATED['p50']} ms")
print(" -> the row-level shares do not reproduce the stated shares; the table is a")
print(" sample of a larger fleet, so treat the stated shares as fleet-level and")
print(" the rows as illustration. Do not present both as one distribution.")
print("\n== 6. What one misconfigured value costs ==")
for threshold in (200, 700):
print(f" endpointing silence threshold {threshold} ms = {100.0*threshold/FLEET_STATED['p50']:.0f}% of the {FLEET_STATED['p50']} ms fleet median turn")
print(" the 700 ms value is the one the AWS article names as feeling 'awful on a real call'")
if __name__ == "__main__":
main()
Captured stdout, verbatim:
== 1. AWS per-stage budget targets (published as targets, not measurements) ==
Endpointing (VAD) 150-300 ms [aws]
ASR final transcript 100-200 ms [aws]
LLM time to first token 300-600 ms [aws]
TTS time to first byte 100-250 ms [aws]
Network + telephony 50-150 ms [aws]
SUM 700-1500 ms article states 'roughly 800ms to 1.5 seconds'
low-end gap vs the quoted 800 ms floor: 100 ms of the stated floor is not covered
high end matches the quoted 1.5 s ceiling exactly: True
== 2. Deepgram client-side streaming components ==
Network transit time 20-200 ms [dg]
Transcription latency 150-300 ms [dg]
SUM (no LLM, no TTS) 170-500 ms
doc's own stated total transcript latency: 200-500 ms -> component sum agrees: True
== 3. TTS share of a single 800 ms turn (measured TTFA, Coval) ==
fastest model vui (Fluxions) 43 ms = 5.4% of the 800 ms budget
board median board median of 32 models 245 ms = 30.6% of the 800 ms budget
slowest model Fish Audio S2.1 Pro Free 826 ms = 103.2% of the 800 ms budget
fastest-to-slowest spread on the measured board: 783 ms (19.2x)
== 4. Fleet tail ratio (DestiLabs telemetry) ==
p50 680 ms, p95 1180 ms -> p95 is 1.74x the median turn
headroom to the 800 ms line at p50: 120 ms; p95 exceeds it by 380 ms
== 5. Reconciliation: stated latency bands vs the 12 listed fleet rows ==
rows in the article table: 12 p50 values: [540, 560, 580, 610, 630, 660, 690, 700, 720, 740, 760, 810]
band computed share stated share
under 600 25% 30%
600-800 67% 40%
800-1000 8% 20%
over 1000 0% 10%
median of the listed rows (mean of middle pair [660, 690]): 675 ms vs stated fleet p50 680 ms
-> the row-level shares do not reproduce the stated shares; the table is a
sample of a larger fleet, so treat the stated shares as fleet-level and
the rows as illustration. Do not present both as one distribution.
== 6. What one misconfigured value costs ==
endpointing silence threshold 200 ms = 29% of the 680 ms fleet median turn
endpointing silence threshold 700 ms = 103% of the 680 ms fleet median turn
the 700 ms value is the one the AWS article names as feeling 'awful on a real call'
What each of the six sections means:
- The AWS budget rows. Design targets summed by hand. Discrepancy one: the minima total 700 ms while the same article quotes an 800 ms floor — 100 ms of the stated floor is not covered by the rows, while the 1,500 ms ceiling matches exactly. The rows under-sum the stated total; both figures are reproduced rather than reconciled.
- Deepgram’s client-side components. Network transit plus transcription, summed against the doc’s own stated 200–500 ms transcript latency — the component sums agree within tolerance.
- TTS share of one turn. Measured Coval TTFA expressed as a percentage of the 800 ms line: the fastest model spends 5.4%, the median 30.6%, the slowest 103.2% — one model’s first audio alone exceeds the entire budget.
- The fleet tail. DestiLabs’ measured p95 is 1.74x its p50, with 120 ms of headroom at the median and a 380 ms overshoot at the tail.
- Band reconciliation. Discrepancy two: the 12 listed fleet rows yield shares of 25/67/8/0% against stated shares of 30/40/20/10%. The rows do not reproduce the stated bands; the likely cause is that the listed rows sample a “10+ live deployments” fleet, which the excerpt cannot confirm — so the two distributions are reported separately and never merged.
- Cost of one misconfigured value. A 200 ms endpointing threshold is 29% of the 680 ms fleet median turn; the 700 ms threshold AWS calls awful is 103% of it.
Cascaded or speech-to-speech?
Published character and speed come from the AWS 800ms rule; every measured cell below comes from the DestiLabs fleet table, whose p50s are measured end-to-end and whose cost figures are all-in per connected minute.
| Dimension | Cascaded (STT→LLM→TTS) | Speech-to-speech |
|---|---|---|
| What it is | Three models in series with two serialisation boundaries removed only by streaming | A single model mapping speech to speech |
| Published latency character | “Sum of four stages, typically 800ms or more” (target framing) | “A single model, noticeably lower” (target framing) |
| Measured median p50 | 610–810 ms across listed cascaded projects | 540–580 ms across listed speech-to-speech projects |
| Measured cost per minute | $0.07–$0.13 all-in | $0.18–$0.21 all-in |
| Barge-in | “You implement it” | “Native” |
| Prosody / emotion | Discards tone, hesitation, annoyance | Keeps that signal |
| Debuggability | “Cheaper at scale and easier to control”; each stage is a separate artefact (structural, not measured) | One opaque end-to-end call (structural, not measured) |
| Language coverage | Models swappable per language; no per-language figures here | Example blocker: “Amazon Nova Sonic does not support Vietnamese yet… that is a hard stop” |
The measured rows agree directionally with the published character: speech-to-speech projects in this fleet are the three fastest (540, 560, 580 ms) and the three most expensive. Containment runs 72–86% on those rows versus 62–88% on cascaded rows — no architectural conclusion is available from that overlap.
Seven things to fix before you buy a faster model
- Move VAD local. Framework-published at 150–200 ms faster than remote VAD services (Pipecat).
- Tighten the endpointing threshold and add a cancel path. Stated effect: a 700 ms threshold “feels awful on a real call”; the script above puts it at 103% of the fleet median turn (AWS, design target plus arithmetic).
- Integrate end-of-turn detection. Vendor-claimed 200–600 ms reduction for Deepgram Flux (Deepgram) — not independently verified; the architectural principle is separately endorsed by WebRTC Ventures.
- Stream every boundary. “If any stage waits for the previous stage to finish, you have already lost” (AWS) — qualitative; no millisecond figure is published for streaming itself.
- Mask tool calls with filler speech. Stated effect: “buys back a full second of real budget” (AWS) — stated, not measured. Pair it with a hard per-turn budget on the tool itself.
- Trim context every turn. Directional, no millisecond figure: summarise past turns rather than replaying them (AWS); see Agent Memory Systems.
- Use perceived-latency features before changing hardware. Pre-connect audio capture, preemptive generation, and
preemptive_tts“reducing perceived latency… at the cost of higher wasted compute” (LiveKit audio docs) — qualitative, no effect size published.
If you only do one, do 1 or 2 — endpointing is the stage teams leave at defaults.
FAQ
Is 800 ms a hard failure threshold? No. It is explicitly a set of design targets — “the targets I design against, not measurements” (AWS). Measured practice is looser: the fleet p50 is 680 ms and p95 is 1,180 ms, with “p95 under ~1,400 ms” treated as the production bar (DestiLabs).
Should I switch to speech-to-speech? Only if speed and native barge-in are worth the cost and coverage constraints. It measured 540–580 ms p50 against 610–810 ms cascaded, at $0.18–$0.21/min against $0.07–$0.13/min, and it can hit hard language stops (AWS’s Nova Sonic/Vietnamese example) (AWS).
What single change moves the number most? The best-documented single levers are endpointing-side: local VAD at 150–200 ms (framework-published) and integrated end-of-turn detection at a vendor-claimed 200–600 ms. Neither is independently measured here; both are larger than any plausible model swap inside the 300–600 ms TTFT band.
My TTS vendor claims sub-90 ms — is my TTS stage 90 ms? Not necessarily. Cartesia claims sub-90 ms model latency while Coval’s independent board measures Sonic 3.5/3.6 at 291/292 ms from text sent to first audible sound. The clocks start and stop in different places, and leading silence can be most of the gap (Chirp 3 HD: 402 of 588 ms). Do not average them; measure your own first audible byte.
How much is my VAD/endpointing configuration costing? Arithmetic over published figures: a 700 ms silence threshold is 103% of the 680 ms fleet median turn; a 200 ms threshold is 29%. The 700 ms value is the one AWS names as feeling awful on a real call.
My p50 is fine but my p95 is terrible — why? Because the tail is 1.74x the median in the measured fleet (680 → 1,180 ms). Tail sources named in the dossier: region hops (up to 100 ms per turn), jitter-buffer sizing (qualitative), synchronous tool calls with no hard budget, and TTS models whose measured TTFA spans 43–826 ms. No per-stage p95 figures exist in these sources.
The Bottom Line
The 800 ms figure is a five-stage ledger of design targets that sums to 700–1,500 ms, beside a measured fleet already running 680 ms at p50 and 1,180 ms at p95. The biggest untuned line items are endpointing configuration and TTS model choice — not the LLM — and both are rarely instrumented. Vendor clocks are not comparable across vendors, so a sub-90 ms claim and a 291 ms measurement can both be true of the same model family. Fix the threshold, move VAD local, and put a stopwatch on first audible byte before buying a faster model.
How This Guide Was Built
Twelve sources were fetched on 2026-11-13: WebRTC Ventures, AWS, Coval, DestiLabs, LiveKit turns, AssemblyAI, Deepgram, Pipecat, OpenAI Realtime, Twilio Media Streams, LiveKit audio, and Cartesia Sonic. Published totals, quotes, and the fleet table were checked against the excerpts; per-model LLM TTFT, per-provider STT latency, and independent verification of vendor claims were not available and are marked as such. The arithmetic script is the only computed artifact in this post; no pipeline was run.
← Back to all posts


