A caller asks "are you open on Sunday?" and hears an answer 900 milliseconds later. In that gap four separate systems ran in sequence. Understanding which one is slow — and which one you are paying for — is most of what you need to evaluate a voice agent properly.
This covers the pipeline, why latency is the genuinely hard part, what drives cost per minute, and the situations where a voice agent is the wrong tool.
The short answer
An AI voice agent is four components in a chain:
- Speech-to-text (ASR) — turns the caller's audio into words
- A language model — decides what to say and what to do
- Tools — checks a calendar, looks up an order, writes a CRM record
- Text-to-speech (TTS) — turns the reply back into audio
Every stage adds delay, and the delays are cumulative. That is the whole engineering problem: a chatbot can take two seconds to reply and nobody minds. On a phone call, two seconds of silence is when people say "hello? are you there?"
Where the time goes
| StageTypical contributionWhat makes it worse | ||
| Speech-to-text | Small to moderate | Waiting for the caller to definitely stop talking |
| Language model | Usually the largest | Long prompts, large models, tool calls needing a second round trip |
| Tool calls | Varies wildly | A slow CRM or calendar API blocks the whole turn |
| Text-to-speech | Small | Generating a whole sentence before playing any of it |
| Network | Small, but repeated | Every hop between separate vendors adds another |
A note on the numbers you will be quoted. Independent benchmarks of voice-agent latency vary widely and often do not disclose their methodology. Vendor figures in the 200–400 ms range typically measure model inference alone, while end-to-end tests that include speech recognition, network hops and speech synthesis report medians closer to 700–1,300 ms. Treat any single number with suspicion and ask what it measures.
What is not disputed is the perceptual threshold. Below roughly 800 ms a caller experiences the exchange as normal conversation. Past about 1.5 seconds, the pause is obvious and people realise they are talking to a machine. That is the bar — and it is why "which model is smartest" is often the wrong question. A slightly less capable model that answers in half the time produces a better call.
Test on a real phone line, not a browser demo. Browser-to-browser demos skip the telephony leg entirely, which is exactly the part that adds delay and degrades audio. A demo that sounds flawless on a laptop can be noticeably worse on an actual call.
The parts nobody demos
Turn-taking
Knowing when the caller has finished speaking is harder than transcribing what they said. End the turn too early and you interrupt someone mid-sentence; too late and you add dead air to every exchange. People pause mid-thought, say "um", and trail off. This is where cheap voice agents feel wrong even when the words are right.
Barge-in
Can the caller interrupt? If the agent is halfway through a long sentence and the caller says "no, wait", a system without barge-in support keeps talking over them. This single behaviour does more for perceived quality than almost anything else.
Accents, names and noise
Speech recognition accuracy is not uniform. It degrades with background noise, regional accents, and — critically for a business — proper nouns. A system that transcribes ordinary sentences perfectly may still mangle a street name, a product code, or a caller's surname. Test with your vocabulary, not generic phrases.
Numbers
Phone numbers, order references and amounts are where errors are least forgivable and most common. Any serious deployment reads them back for confirmation. Ask a vendor how they handle this; a shrug is an answer.
What it costs, and what drives the number
Voice agents are usually priced per minute, and that figure is assembled from parts:
| Cost componentNotes | |
| Telephony | Carrier charges for the call itself; varies enormously by country and by inbound vs outbound |
| Speech-to-text | Billed by audio duration |
| Language model | Billed by tokens. Long conversations cost more per minute than short ones, because the history is re-sent each turn |
| Text-to-speech | Billed by characters generated; cloned or premium voices cost more |
| Platform margin | The vendor's own charge on top |
Three consequences worth knowing before you compare quotes:
- "Per minute" is not comparable across vendors unless you know whether telephony is included. Some quote the AI only and bill carrier charges separately.
- Silence is billable. A caller who puts you on hold is still consuming minutes.
- A cheaper per-minute rate can cost more overall if the agent is slower to resolve. Twenty extra seconds of fumbling on every call is real money at volume.
Ask for a worked example: a typical call for your business, minute by minute, with every line item.
Compliance you cannot skip
Two obligations catch people out, and both vary by jurisdiction — treat this as a prompt to get advice, not as advice.
- Recording consent. Many jurisdictions require callers to be told a call is recorded, and some require explicit consent from all parties. If you retain transcripts — and any useful system does — you are recording in substance even if you keep no audio.
- Disclosure that it is AI. A growing number of jurisdictions require it, and reputationally it is the right default regardless. Callers who discover mid-conversation that they were misled respond badly.
You also inherit the ordinary data-protection questions: where transcripts are stored, how long they are kept, and who can read them. Handling health or financial detail raises that scrutiny further.
When not to use a voice agent
- Complaints and cancellations. Someone already angry does not want to explain themselves to software. Route these straight to a person.
- Anything with legal or financial consequence. Contract terms, medical guidance, disputed charges.
- Very low call volume. If you take four calls a day, the setup effort will not repay itself. A shared inbox is a better answer.
- When your information is not written down anywhere. An agent can only answer from what it can read. If your pricing lives in a colleague's head, fix that first.
What a good deployment looks like
- Start with overflow, not replacement. The agent answers what rings out — after hours, during a rush. Nobody loses a call they would otherwise have taken.
- Pick three intents. Hours and location, availability, booking. Three done well beats thirty done adequately.
- Make escalation instant and obvious. "Let me get someone" should be one sentence away at all times.
- Read the transcripts. Every week, for the first month. The highest-value hour you will spend, and the step most people skip.
- Widen only when a category is genuinely solved.
This is also how you get a result quickly. Salesforce reports that 70% of organisations adopting AI agents see measurable value within 60 days (State of Service, 7th edition) — and the ones that do are almost always those who scoped tightly rather than attempting everything at once.
Frequently asked questions
Will callers know it is AI?
Most will, quickly — and that is fine. What annoys people is not the machine, it is the machine wasting their time. A clear opening ("you're speaking to an automated assistant; I can help with bookings or put you through to someone") sets expectations and improves the rest of the call.
Can it use my own voice?
Voice cloning from a short sample is standard. Whether you should is a separate question: cloning a named person's voice raises consent issues, and a neutral professional voice is often the safer choice for a business line.
What happens if it does not understand?
In a well-built agent, it asks once for clarification and then escalates. An agent that loops — "sorry, I didn't catch that" three times running — is worse than no agent, because the caller has wasted a minute and is annoyed before reaching a human.
Can it make outbound calls too?
Technically yes, and it is a different risk profile. Outbound calling attracts far more regulation than answering your own phone — consent, calling hours, do-not-call registers. Check the rules for every market you would call into before you start.
How many concurrent calls can it handle?
In principle many more than a human team, which is the main operational argument for it. In practice the ceiling is usually your telephony provider's concurrency limit and your budget, not the AI.
Does it work in languages other than English?
Multilingual support is common, but quality is uneven across languages and worse for regional accents and code-switching — which matters if your callers mix languages mid-sentence, as many do. Test in the languages your customers actually use, not the ones on the feature list.
The takeaway
A voice agent is a latency problem with a language model attached. Judge it on how a whole call feels on a real phone line, not on how clever the transcript looks; get a line-by-line cost example before comparing quotes; and deploy it on overflow first, where a failure costs you nothing you were not already losing.
Want to hear one? Serve AI answers calls in your own voice, books into your calendar and writes the lead into your CRM. See what it costs or have it call you.
