How to Build a Voice AI Agent from Scratch

This guide is based on official documentation and public repositories; it was not run hands-on in this pipeline.

1. What a voice AI agent actually is (the 30-second mental model)

At its core, a voice AI agent is a real-time system that listens, thinks, and speaks.

  • Listening, thinking, and speaking: Deepgram frames the full speech pipeline as (listening, thinking, and speaking).
  • STT-LLM-TTS: LiveKit frames it as a streaming STT-LLM-TTS pipeline that must handle the core challenges of real-time voice AI.
  • Client/server split: As noted by Pipecat, a typical application has a client (browser, mobile app, or phone) and a server that runs the pipeline to process audio, run LLMs, and generate speech in real-time.

The flow is straightforward in principle: audio comes in from a user, it is transcribed (STT), reasoned over by an LLM, and turned back into speech (TTS) to be returned to the user over a real-time transport.

2. Choose your build path: three frameworks, three philosophies

Not every builder needs the same amount of control. Based on the official docs, there are three documented paths to consider:

Path Philosophy License Where to learn more
LiveKit Agents Treat your agent as a full real-time participant in a LiveKit room. Open, extensible, and built to mix and match providers. Apache 2.0 (LiveKit Agents GitHub) LiveKit Agents docs
Pipecat A provider-agnostic Python framework. Orchestrate services through pipelines, processors, and transports with the goal of working with “any AI provider and any hosting environment”. BSD-2 (Pipecat GitHub) Pipecat docs
Deepgram Voice Agent API A fully managed speech pipeline. It “handles the full speech pipeline (listening, thinking, and speaking)” so you can focus on what the agent does. Managed API (Deepgram Voice Agent docs) Deepgram Voice Agent docs

Note: Pipecat states it is “built and maintained by Daily”. No other ownership claims are established by these sources.

3. Anatomy of the pipeline: STT, LLM, TTS

Each framework names the same three stages, but exposes them differently:

Stage What the official docs establish Source
STT (Speech-to-Text) Part of the STT-LLM-TTS pipeline. LiveKit states its ecosystem allows you to “mix and match the right STT, LLM, TTS, and Realtime API”. Deepgram lists “speech-to-text models… and audio formats” as configuration surfaces. docs.livekit.io/agents/, github.com/livekit/agents, developers.deepgram.com/docs/voice-agent
LLM (Large Language Model) LiveKit names “LLM orchestration” as a core SDK capability. Deepgram states LLM providers are configurable for its agent. Pipecat states the server “runs a Pipecat pipeline that processes audio, runs LLMs, and generates speech in real-time.” docs.livekit.io/agents/, developers.deepgram.com/docs/voice-agent, docs.pipecat.ai/
TTS (Text-to-Speech) The third stage of the STT-LLM-TTS pipeline. Deepgram lists “TTS voices” as a configuration surface, with the end goal of generating speech in real-time as noted by Pipecat. docs.livekit.io/agents/, developers.deepgram.com/docs/voice-agent, docs.pipecat.ai/

The official docs do not establish specific model identifiers, API schemas, or audio format values here. Those remain unconfirmed.

4. Turn detection and interruptions: how the agent knows when to speak

For a natural back-and-forth, an agent must know when a user has finished speaking.

  • LiveKit Agents calls out “reliable turn detection” and “handling interruptions” as components of the SDK. Its GitHub README specifically notes “Semantic turn detection: Uses a transformer model to detect when a user is done with their turn, helps to reduce interruptions.”
  • Deepgram Voice Agent API lists endpointing as an agent setting to configure.

This section stays at capability level. It is intentionally not a deep dive into barge-in mechanics — check your chosen framework’s docs for those internals.

5. Transport: getting audio to and from the user

Real-time voice requires a low-latency transport between client and agent, with the agent often communicating to its backend over standard web protocols.

Framework Transport surfaces established Source
LiveKit LiveKit WebRTC is used between the frontend and the agent. The agent communicates with your backend using HTTP and WebSockets. docs.livekit.io/agents/
Pipecat Names pipelines, processors, and transports as its three core concepts. Provides Pipecat Clients for JavaScript, React, React Native, iOS, Android, and C++ to connect users via browser, mobile app, or phone. docs.pipecat.ai/
Deepgram Offers server-side SDKs and WebSocket API (Python, JavaScript, C#, Go), the Browser Agent SDK, and Telephony as transport surfaces. developers.deepgram.com/docs/voice-agent

Telephony: joining from a phone

Both LiveKit and Deepgram document telephony as an option: LiveKit states it “includes full support for telephony, so the user can join the call from a phone instead of a frontend app”, and Deepgram lists Telephony for inbound and outbound calls.

6. How your agent comes alive: the LiveKit server/job lifecycle

For builders looking at LiveKit specifically, the official docs describe a concrete lifecycle:

  1. Register. When your agent code starts, it first registers with a LiveKit server (either self-hosted or LiveKit Cloud) to run as an “agent server” process. (docs.livekit.io/agents/)
  2. Wait for dispatch. The agent server waits until it receives a dispatch request. (docs.livekit.io/agents/)
  3. Boot a job. To fulfill that request, the agent server boots a “job” subprocess which joins the room. (docs.livekit.io/agents/)
  4. Default behavior. By default, agent servers are dispatched to each new room created in your LiveKit Cloud project or self-hosted server. (docs.livekit.io/agents/)

LiveKit also notes production-readiness features such as built-in agent server orchestration, load balancing, and Kubernetes compatibility. The docs do not provide concrete orchestration manifests or deployment steps.

7. Installation and project setup (what the sources actually show)

Stick to only what is verifiably sourced.

Item Established Source
Verified pip install pip install "livekit-agents[openai,deepgram,cartesia]" github.com/livekit/agents
Pipecat package The PyPI project is pipecat-ai. The repository lists env.example, pyproject.toml, and examples/, but their contents were not captured in this run. github.com/pipecat-ai/pipecat
Build-time estimates (not latency) LiveKit states a voice assistant can be built “in less than 10 minutes”. Pipecat states you can “create a voice AI bot in 5 minutes”. Treat these strictly as build-time estimates, not voice latency. docs.livekit.io/agents/, docs.pipecat.ai/ 
# Illustrative, not executed in this pipeline.
# The only install command verifiable from the sources:
pip install "livekit-agents[openai,deepgram,cartesia]"
# Pipecat's PyPI project (from its repo): pipecat-ai
# (A concrete install form beyond the project name is not verifiably sourced here.)

8. Tools, multi-agent handoff, and going further

Capabilities named in the sources:

The docs do not capture concrete function schemas, MCP configuration, or test examples.

Official next steps to follow yourself (unfetched content): LiveKit Voice AI quickstart, Pipecat Quickstart and Learning Pipecat, and Deepgram’s “Build a Voice Agent” guide plus the Agent API Playground. These are references only; this guide does not summarize their steps.

9. Your build worksheet and next steps

Use this worksheet to make explicit decisions as you go. Leave blanks and verify against official docs for anything marked unconfirmed.

# Stage What the official docs establish Your choice (fill in) Verified how? (fill in)
1 Build path LiveKit Agents / Pipecat / Deepgram Voice Agent API ___________ ___________
2 Language/runtime Python or Node.js (LiveKit); Python (Pipecat); Python, JavaScript, C#, or Go (Deepgram SDKs) ___________ ___________
3 Install command pip install "livekit-agents[openai,deepgram,cartesia]" (LiveKit) / pipecat-ai (PyPI) ___________ ___________
4 STT provider STT is a mix-and-match pipeline stage; Deepgram exposes STT models + audio formats in config (config details unconfirmed here) ___________ ___________
5 LLM provider LLM orchestration built in; LLM providers configurable ___________ ___________
6 TTS voice TTS is the third pipeline stage; TTS voices configurable ___________ ___________
7 Turn detection LiveKit semantic turn detection (transformer model); Deepgram endpointing setting (no thresholds/values established) ___________ ___________
8 Transport WebRTC (frontend↔agent) + HTTP/WebSockets (agent↔backend); or Pipecat transports; or Deepgram WebSocket API ___________ ___________
9 Client surface Browser / mobile SDK / phone (telephony) ___________ ___________
10 Tools & handoff Tool use, multi-agent handoff, MCP support, function calling (schemas/payloads unconfirmed) ___________ ___________
11 Hosting LiveKit Cloud or self-hosted; Pipecat Cloud or own VPC; Deepgram managed ___________ ___________
12 First-build time (your measurement) Docs claim “less than 10 minutes” (LiveKit) / “5 minutes” (Pipecat) — build time only, not voice latency. Voice latency is unconfirmed. ___________ ___________

FAQ

1. Q: Do I need to manage STT, LLM, and TTS myself, or can someone do it for me?
A: Both paths are documented. LiveKit Agents and Pipecat give you a pipeline you assemble and orchestrate. Deepgram’s Voice Agent API “handles the full speech pipeline (listening, thinking, and speaking)” so you can focus on agent behavior.

2. Q: Can users call my agent from a regular phone?
A: Yes, per documentation. LiveKit states “full support for telephony, so the user can join the call from a phone instead of a frontend app”. Deepgram lists Telephony for inbound and outbound calls.

3. Q: What latency can I expect end-to-end?
A: Unconfirmed. The sources contain no latency numbers of any kind. “Ultra-low latency” from Pipecat is unquantified marketing language. “Less than 10 minutes” and “5 minutes” are build-time estimates only. Measure your own stack — use row 12 of the worksheet for this exact purpose.

4. Q: How do I configure OpenAI’s Realtime API as my speech backend?
A: Unconfirmed. The fetched OpenAI Realtime page contains only sidebar navigation with no guide content captured. No events, session structure, audio format, or related details are established here. Treat this as requiring a fresh fetch from the official docs, and do not infer configuration from this source base.

The Bottom Line

  • Pick your control level: Choose LiveKit for room-based extensibility, Pipecat for a provider-agnostic Python pipeline, or Deepgram Voice Agent API for a managed, full-pipeline approach.
  • Know the anatomy: Plan around STT-LLM-TTS with awareness of turn detection (LiveKit semantic turn detection, Deepgram endpointing) and a suitable transport (WebRTC + HTTP/WebSockets, or the transports offered by your chosen path).
  • Start small and verify: Use the worksheet above, rely only on the sourced pieces (notably pip install "livekit-agents[openai,deepgram,cartesia]" from LiveKit), and explicitly flag anything the official docs do not confirm as unconfirmed.

How This Guide Was Built

← Back to all posts