The Latency Trap: Moving Beyond TTFT for Voice Agents
When engineering real-time voice agents, developers frequently fixate on Time to First Token (TTFT). While essential, TTFT is fundamentally a measurement of when the LLM transitions from idle to active. In the context of a voice conversation, it is merely the starting line. True conversational fluidity requires an agent to minimize the time until the first audible sentence, a metric often referred to as Time to First Sentence (TTFS).
Because text-to-speech (TTS) engines require a complete clause or sentence to begin synthesizing audio, optimizing solely for TTFT can result in agents that appear responsive but suffer from delayed, jerky delivery. Building a production-grade agent requires managing a cumulative latency budget that spans inference, transcription, synthesis, and network transport.
Understanding the Voice Latency Budget
A high-performance voice-to-voice system must keep its total response time between 700ms and 1.2s. To stay within this, the latency must be distributed efficiently across the stack. Industry experts generally recommend partitioning this budget into four equal parts: transport, transcription (STT) and phrase detection, LLM generation, and synthesis (TTS). Pauses extending beyond 800ms are typically perceived by users as unnatural, breaking the illusion of conversational presence.
Benchmark Overview: Decoding Inference Performance
The following data highlights how various providers perform on the critical LLM first-chunk latency. These figures reflect a 10k input token workload, demonstrating that prompt size significantly impacts both TTFT and overall throughput.
| Provider | Model | TTFT (Seconds) | Output Speed (tok/s) |
|---|---|---|---|
| Baseten | gpt-oss-120b (high) | 0.23s | 266 |
| Baseten | gpt-oss-120b (low) | 0.24s | 271 |
| DeepInfra | Nemotron 3 Ultra | 0.28s | 371 |
| Cohere | Command A+ | 0.40s | 239 |
| Modular | Gemma 4 31B (NVFP4) | 0.44s | 243 |
| Cerebras | gpt-oss-120b (high) | 0.49s | 1,697 |
The Throughput vs. Latency Paradox
It is vital to distinguish between raw token generation speed and initial latency. Some hardware architectures, such as those used by Cerebras and Groq, offer exceptional throughput (exceeding 1,600 tok/s), which is excellent for streaming but does not automatically guarantee the lowest TTFT. Conversely, models designed for specific diffusion-based tasks, like Inception’s Mercury 2, may show high throughput but incur massive TTFT penalties exceeding 3 seconds, rendering them unsuitable for real-time voice interaction.
Real-time Model Performance
Recent benchmarks for speech-to-speech (S2S) models reveal the trade-offs between reasoning effort and latency. As models are tasked with more complex reasoning, the Time to First Audio (TTFA) increases.
| Model | TTFA (Seconds) | Speech Reasoning | Task Success |
|---|---|---|---|
| Deepslate Opal | 0.44s | 85% | – |
| Grok Voice Think Fast 2.0 High | 0.70s | 97% | 94.7% |
| OpenAI GPT-Realtime-1.5 | 0.81s | 81% | 85.1% |
| OpenAI GPT-Realtime-2.1 High | 1.21s | 96% | 91.5% |
| Google Gemini 3.1 Flash Live High | 2.99s | 97% | 71.8% |
Strategic Recommendations for Agent Architecture
- Colocate Services: Prioritize geographic proximity between your agent infrastructure and the inference provider. This is often more impactful than selecting a faster model.
- Configure Reasoning Effort: For most Realtime APIs, reasoning effort is a configurable parameter. Lowering it significantly reduces TTFT at a marginal cost to task accuracy.
- Implement Preemptive Generation: Explore STT models like Deepgram Flux that offer early end-of-turn signals, allowing the LLM to begin processing before the final transcript is locked.
- Instrument Everything: Do not rely on vendor marketing figures. Implement logging for end-to-end latency, LLM TTFT, and TTS time-to-first-byte. Focus on optimizing the p95 latency, as outliers are what cause agents to feel broken to the end user.
