The first thing callers notice about a voice agent is not what it says. It is how long it takes to say it. A pause of a second and a half on a phone line does not read as the agent thinking. It reads as the line dropping, and people fill it with "hello?", which the agent then has to deal with as well.
Real-time systems are my day job: at Acefone, calls, WhatsApp and IVR run through one inbox, where a delay is something a customer hears rather than something a dashboard shows. Voice agents inherit every one of those constraints and add a language model in the middle. This post breaks one agent turn into its stages, puts a number on each, and ends with a calculator for your own stack.
How fast people actually answer
The best measurement of human turn-taking is still a 2009 study by Tanya Stivers and colleagues, who timed turn transitions in ten languages around the world. Their finding, in the paper's own words: "an overall mode of 0 ms" between one speaker stopping and the next starting, a median of about 100 ms and a mean of about 208 ms. Every language sat within a few hundred milliseconds of that mean, from Japanese at 7 ms to Danish at 469 ms.
That is faster than anyone could hear a question, understand it and plan an answer. As Levinson and Torreira point out, planning even a short reply takes more than 600 ms. People manage it by predicting where the other person's sentence is going and preparing their answer while it is still being spoken.
A voice agent does not predict. It waits for the caller to finish, works out what they said, decides what to say, and then starts speaking. So the realistic targets are well above human speed. Daily's guidance is to "aim for 800ms median voice-to-voice latency". Twilio's latency guide sets a target of 1,115 ms mouth-to-ear, with 1,400 ms as the upper limit.
Where the milliseconds go
A turn begins when the caller stops talking and ends when the caller hears the agent start. In between, most pipelines do four things in order.
| Stage | What happens | Twilio's figure |
|---|---|---|
| Speech-to-text | Deciding the caller has finished, then producing the final transcript | 350 ms |
| Model | Time until the first token of the reply comes back | 375 ms |
| Voice | Time until the first audio of that reply is ready | 100 ms |
| Network and telephony | Audio crossing the phone network and reaching your servers, both ways | the remaining 290 ms |
| Total | 1,115 ms |
The first stage is the one people underestimate. Knowing that a caller has finished speaking, rather than paused mid-sentence, is a guess. Wait too briefly and the agent interrupts people who were only taking a breath. Wait too long and every turn starts with dead air. Most of the tuning in a real deployment happens here.
Work out your own budget
Change the stages to match your stack, and add the lookups your agent actually does mid-conversation.
calculator · milliseconds from the caller's last word to the agent's first
Where your voice agent's response time goes
the waterfall, with the targets people quote
Defaults are the per-stage figures from Twilio's guide to voice agent latency (speech-to-text 350 ms, model first token 375 ms, speech first byte 100 ms, 1,115 ms in all). The 800 ms target is Daily's; the ~200 ms human gap is from Stivers et al., 2009. Stages are treated as sequential, which is how most pipelines run them.
quick check
Your agent is at 1,115 ms per turn. You add a CRM lookup that takes 300 ms and a retrieval step that takes 200 ms before the model runs. What does the caller hear?
In a sequential pipeline every stage adds to the wait. 1,115 + 300 + 200 is 1,615 ms, past Twilio's 1,400 ms upper limit. The usual fix is to acknowledge first ("let me check that") and do the lookup while that plays.
What blows the budget
Most agents that feel slow are not slow in one place. They are slow in several small ones that add up.
- Tool calls in the critical path. Order status, appointment slots, account lookups. Each one is a network round trip to a system that was never built to answer in 50 ms.
- Retrieval before every reply. Embedding the question, searching a vector store and reranking the results can easily take a few hundred milliseconds, and many agents do it even when the answer is already in the conversation.
- Waiting for the whole reply. If the voice only starts once the model has finished generating, you pay for every token of the answer before the caller hears the first word.
- Long prompts. Time to first token grows with how much the model has to read. A system prompt that has quietly grown to several thousand tokens costs latency on every turn, not just money.
- Servers in the wrong place. If the phone carrier's media lands in one region and your model and voice providers answer from another, every turn pays for the distance twice.
How to win the time back
In roughly the order I would try them:
- Stream every stage. Stream audio into speech-to-text, stream tokens out of the model, and start synthesising speech from the first sentence rather than the whole reply. This alone is the difference between agents that feel fine and agents that do not.
- Say something first. A short acknowledgement ("sure, one moment") buys a second of cover for a lookup and sounds like a person. Keep a few recorded so they cost nothing and play instantly.
- Take lookups off the path. Fetch the caller's account while the phone is still ringing. Prefetch what the next step of the flow is likely to need.
- Keep the prompt short and cached. Move rarely used instructions into tools the model can call when it needs them.
- Co-locate. Put your orchestration in the same region as your telephony media and your model and speech providers.
- Consider the channel. An agent inside your own app or website can use WebRTC and skip the phone network entirely, which removes the telephony leg from every turn. It is a product decision as much as an engineering one, but it is worth asking whether the caller needs a phone number at all.
Speed and cost pull against each other in a few places. The fastest voices are not always the cheapest, and a bigger model can be slower to its first token. The cost side of the same stack is in what an AI phone agent costs per minute.
questions people ask
What is good latency for a voice AI agent?
Around 800 ms from the caller's last word to the agent's first is a common target. Twilio's guide treats 1,115 ms as its target and 1,400 ms as the upper limit. Beyond that, callers start talking over the agent.
How fast do people respond in conversation?
Very fast. Across ten languages, Stivers and colleagues found a mean gap of about 208 ms between turns, and the most common gap was zero, because people plan their reply while the other person is still talking.
What causes latency in voice agents?
Four sequential stages: network and telephony, speech-to-text deciding the caller has finished, the model producing its first token, and the voice producing its first audio. Tool calls and retrieval add to the same line.
Does WebRTC reduce voice agent latency?
It removes the phone network from the path when the caller is in your app or browser, which cuts the telephony share of each turn. For callers on a phone line, the telephony leg is unavoidable.
How do I make my voice agent respond faster?
Stream every stage, start speaking from the first sentence of the reply, acknowledge before slow lookups, keep the prompt short and cached, and run everything in one region.
The short version
People answer in about 200 ms. A good voice agent answers in about 800, and a slow one in well over a second. The time goes to four stages in a line, and anything you add mid-turn joins the queue.
Stream every stage, speak from the first sentence, acknowledge before you look anything up, and keep slow work off the critical path. Measure each stage separately, because the one you are sure is slow is often not the one that is.