Lowest-Latency Inference APIs for Voice and Realtime Brokers: A Time to First Token TTFT-First Benchmark

Time to first token (TTFT) is the metric groups use to choose an inference API for voice. Additionally it is the metric that misleads them. TTFT marks when era begins; a text-to-speech mannequin can’t converse till a full clause arrives. Between these two factors sits the distinction between an agent that feels conversational and one which will get interrupted. This piece benchmarks each layer of the voice stack together with LLM, speech-to-text, text-to-speech, and speech-to-speech.

Why TTFT Is the Proper Entry Level and the Fallacious End Line

A voice agent is a latency finances with a language mannequin inside it. Each stage spends milliseconds the consumer can hear.

Time to first token (TTFT) is the interval between sending an inference request and receiving the primary token again. IBM’s definition frames it because the second a system transitions from idle to visibly energetic.

For chat, TTFT is near the entire story. For voice, it’s one time period in a sum.

The reason being mechanical. A text-to-speech mannequin can’t synthesize half a phrase. It wants a whole clause or sentence earlier than it produces audio. LiveKit calls the ensuing metric time-to-first-sentence (TTFS), and argues in its Gemma 4 deployment publish that TTFS is what customers really really feel.

That provides you two knobs quite than one. TTFT controls when era begins. Tokens per second controls how briskly the primary sentence completes. A supplier that wins one and loses the opposite won’t really feel quick.

The Latency Price range: What One Voice Flip Really Prices

LiveKit’s voice brokers overview breaks a flip into STT at roughly 100–200ms, LLM at 300–500ms with streaming, TTS at 100–200ms, and community at 50–150ms over WebRTC. It places the sensible end-to-end goal at 700ms to 1.2s.

Kwindla Hultman Kramer, co-creator of Pipecat, has suggested concentrating on 800ms median voice-to-voice latency, with a looser 1,500ms acceptable for a proof of idea. His tough arithmetic splits that 4 methods at roughly 200ms every: transport and media processing, STT plus phrase endpointing, LLM inference, and TTS.

Each day’s earlier work on the quickest voice bot provides the human baseline. Typical human response time in dialog is round 500ms. Pauses past 800ms begin to really feel unnatural.

Each day’s February 2026 voice-agent LLM benchmark interprets that into an LLM requirement instantly. Pure dialog wants voice-to-voice beneath 1,500ms, which works out to roughly 700ms of TTFT finances for a text-mode LLM inside a transcription-to-LLM-to-voice harness.

That 700ms quantity is the bar to carry each supplier towards.

How one can Learn a TTFT Benchmark With out Being Misled

Earlier than the tables, 5 methodology information that change what the numbers imply:

1. Workload form dominates: Synthetic Evaluation modified its default workload in March 2026. The location now stories 10k enter token prompts quite than 1k. Longer prompts elevate each TTFT and output pace. LiveKit argues that is nearer to actuality for voice, as a result of manufacturing brokers front-load coverage, persona, escalation guidelines, retrieved knowledge, and gear schemas.

2. Server location is baked in: Synthetic Evaluation assessments from a digital machine in Google Cloud’s us-central1-a zone. It states plainly that TTFT contains community latency and should benefit or drawback suppliers based mostly on the place they serve.

3. Reasoning tokens depend: Within the Synthetic Evaluation definition, TTFT for a reasoning mannequin is the primary reasoning token, not the primary reply token. These are separate columns.

4. Measure from the receiving aspect: Each day notes that mannequin suppliers generally quote TTFT inside to their inference stacks. Each day measures from request ship to first usable token off the API.

5. Runs are usually not repeatable: Each day is blunt about this: TTFT varies considerably between benchmark runs, and suppliers change inference stacks and generally weights with out altering mannequin names.

Layer 1: LLM Time to First Token

Figures under are from the Synthetic Evaluation API suppliers leaderboard, retrieved August 30, 2026. The “first chunk” column is TTFT. Workload is 10k enter tokens, single immediate, median over 72 hours.

Lowest measured first-chunk latency

The throughput entice

Silicon distributors optimize for a distinct metric than voice brokers want.

Mercury 2 is the clearest illustration. It’s a diffusion-based language mannequin, and it generates 770 tokens per second. Its first chunk arrives at 3.07s. That’s 4 occasions all the LLM finances for a pure dialog.

Cerebras and Groq are a distinct case. Their TTFT is respectable and their throughput is phenomenal. For TTFS particularly, that mixture is robust, as a result of the sentence completes nearly instantly after the primary token lands.

Frontier and proprietary endpoints

Observe the identical mannequin on completely different hosts. GPT-5.6 Luna non-reasoning measures 0.59s on Amazon Bedrock and 0.74s on OpenAI’s personal API. Internet hosting and routing matter as a lot because the weights.

The seller-measured outlier

LiveKit publishes TTFT figures for its personal inference product. Gemma 4 31B on LiveKit Inference measured 192ms, towards Gemini 2.5 Flash at 911ms, GPT-5.5 at 966ms, GPT-4.1 at 1,006ms, and the identical Gemma 4 31B by way of OpenRouter at 1,876ms.

LiveKit is clear concerning the mechanism, which makes the declare extra credible than most. It runs Gemma behind SGLang with speculative decoding, and intentionally under-packs every GPU so queueing delay stays low. A heat request, it says, begins returning tokens in round 100ms. The tradeoff is value, at $1.20 per 1M output tokens.

The identical publish stories TTFS throughout full conversations: 354ms for Gemma 4 31B on LiveKit, 1,034ms for Gemini 2.5 Flash, 1,088ms for GPT-4.1, 1,267ms for Gemini 3.0 Flash, and 1,404ms for GPT-5.5.

Functionality numbers accompany it. On IFBench, independently scored by Synthetic Evaluation, Gemma 4 31B scores 75.6% towards GPT-5.5 at 75.9%, GPT-4.1 at 43%, and Gemini 2.5 Flash at 39%. On τ²-bench, GPT-5.5 leads at 93.9% with Gemma 4 31B at 76.9%.

Layer 2: Speech-to-Textual content and Flip Detection

For voice, STT latency will not be transcription pace. It’s how lengthy after the consumer stops speaking the pipeline is aware of the consumer stopped speaking.

Synthetic Evaluation measures two issues on its streaming STT leaderboard, each ranging from a SileroVAD-detected finish of speech: time to first partial transcript, and time to last transcript. Its AA-WER Streaming index attracts on roughly 8 hours of audio, weighted AA-AgentTalk 50%, VoxPopuli 25%, Earnings-22 25%.

Vendor-published latency figures:

Deepgram Flux is essentially the most architecturally attention-grabbing entry. It folds end-of-turn detection into the popularity mannequin quite than bolting a VAD on prime. Deepgram states this will minimize agent response latency by 200–600ms versus a standard STT-plus-VAD pipeline. It exposes eot_threshold (0.5–0.9), eager_eot_threshold (0.3–0.9), and an EagerEndOfTurn occasion that allows you to begin the LLM early.

That final functionality issues greater than the uncooked quantity. In the event you can start era on an keen sign, you progress LLM TTFT off the essential path completely when the prediction is true.

AssemblyAI Common-Streaming inverts the standard partials-then-finals mannequin by emitting immutable transcripts. AssemblyAI reported 307ms median phrase emission towards 516ms for Deepgram Nova-3 in its personal 2025 measurement. Its docs additionally suggest utilizing unformatted transcripts for voice brokers, since formatting arrives later and barely modifications LLM conduct.

Accuracy claims listed here are contested and vendor-published. AssemblyAI stories Common-3.5 Professional Realtime at 6.99% WER on the open Pipecat voice-agent benchmark, forward of Google Chirp3 at 9.04%, ElevenLabs Scribe v2 at 9.76%, and Deepgram Flux at 15.58%. Run it your self earlier than treating it as settled.

LiveKit additionally paperwork preemptive era, which begins the LLM on a partial transcript. The caveat is actual: if the reply needs to be regenerated after the ultimate transcript, you burn tokens and save nothing.

Layer 3: Textual content-to-Speech Time to First Audio

That is the place vendor numbers diverge most sharply from what customers expertise.

ElevenLabs states Flash v2.5 delivers roughly 75ms. Its personal docs qualify that fastidiously: 75ms refers to mannequin inference time solely. The corporate’s latency ideas web page goes additional, itemizing community round-trip at sometimes 20–200ms relying on geography, and noting that almost all audio gamers buffer earlier than playback, with 500ms buffering being widespread. It additionally states that Eleven v3 will not be constructed for real-time, and recommends Flash v2.5, Flash v2, or Multilingual v2 for its Brokers Platform.

Cartesia states sub-90ms TTS and 100ms transcript latency for Sonic-3.6 and Ink-2. Marktechpost’s protection of the Sonic-3.6 launch flagged each as vendor-stated mannequin latency, not measured end-to-end spherical journeys. Cartesia beforehand claimed 82ms end-to-end time-to-first-audio for Sonic 3.5. Sonic runs on state house fashions quite than transformers, which scale linearly quite than quadratically with sequence size.

On high quality, the Synthetic Evaluation Supplier Voice enviornment is blind-listener Elo, retrieved August 30, 2026:

The hole between Sonic 3.6 at 1,288 and Flash v2.5 at 1,083 is the standard value of the low-latency tier that almost all brokers really run on.

Layer 4: Speech-to-Speech Time to First Audio

Speech-to-speech fashions collapse STT, LLM, and TTS into one go. Fewer spherical journeys ought to imply decrease latency.

LiveKit is cautious right here, noting that realtime fashions are not assured to be sooner in each case, and {that a} well-tuned pipeline will be extremely aggressive.

The information helps that warning. From the Synthetic Evaluation speech-to-speech leaderboard, TTFA measured on Large Bench Audio, retrieved August 30, 2026:

Grok Voice Assume Quick 2.0 Excessive is the standout on this board: 0.70s TTFA with 97% speech reasoning and 94.7% job success.

The reasoning-effort penalty is seen inside single mannequin households. Gemini 3.1 Flash Stay strikes from 0.96s to 2.99s between Minimal and Excessive. OpenAI’s GPT-Realtime-2.1 strikes from 0.97s to 1.21s, shopping for 2.1 proportion factors of job success.

OpenAI shipped gpt-realtime-2.1 and gpt-realtime-2.1-mini in early July 2026, and said that improved caching minimize p95 latency by at the least 25% throughout its Realtime voice fashions. Tail latency is what makes a telephone agent really feel damaged, so that may be a extra helpful declare than a median enchancment could be.

The aptitude hole

Each day’s benchmark quantifies why most manufacturing brokers nonetheless use cascaded pipelines. On its aiwf_medium_context take a look at, GPT Realtime scored 86.7% towards GPT-4.1 at 94.9%. Ultravox 0.7 was, in Each day’s evaluation, the primary speech-to-speech mannequin to carry out nicely on lengthy multi-turn conversations, and it’s open weights.

Synthetic Evaluation additionally benchmarks 4 vendor “default cascaded programs,” which is beneficial context for what the platforms really ship: Deepgram Voice Agent (Nova-3 + GPT-4o Mini + Aura-2), ElevenLabs Brokers (Scribe v2 Realtime + Gemini 2.5 Flash + Eleven Flash v2), Cartesia Line (Ink + Gemini 2.5 Flash + Sonic), and Inworld Realtime (Inworld STT 1 + Gemini 2.5 Flash + Inworld TTS 1.5 Mini).

Three of the 4 run Gemini 2.5 Flash. That may be a revealing consensus.

Reference Budgets

Assembled from the verified element figures above. These are planning estimates, not measurements of a working system.

Aggressive cascaded pipeline, US-hosted, colocated:

That lands at or barely above the 800ms goal, which matches Kwindla’s framing that 800ms is tight however achievable.

Speech-to-speech, single mannequin:

Comparable, with much less observability and, per Each day’s benchmark, a measurable functionality hole on instrument calling and instruction following.

What to Do With This

• Decide the metric your structure is bounded by. If a TTS mannequin sits downstream, optimize TTFS, not TTFT. Which means TTFT and tokens per second collectively.
• Colocate earlier than you optimize fashions. LiveKit charges agent-model colocation as very excessive affect, above mannequin alternative. In the event you use SIP, hold the trunk geographically shut too.
• Cap reasoning effort explicitly. It’s the largest single lever within the tables above, and it’s a configuration flag on most trendy endpoints.
• Price range for instrument calls. Kwindla notes that any flip with a instrument name roughly doubles LLM latency. LiveKit recommends limiting max_tool_steps, consolidating exterior API calls, and taking part in a pondering sound so silence will not be the consumer’s solely suggestions.
• Instrument earlier than you tune. The LiveKit Brokers SDK exposes e2e_latency, LLM time to first token, and TTS time to first byte per flip. Pipecat exposes the equal by means of enable_metrics and observers. Retailer the logs externally and look ahead to regression.
• Measure p95, not simply p50. OpenAI’s personal important enchancment in July 2026 was a tail-latency discount, as a result of that’s the place voice brokers break.
• Look ahead to infrastructure footguns. LiveKit paperwork that self-hosted brokers on AWS burstable occasion sorts resembling t3 or t4g can hit extreme latency and turn-detection timeouts even at apparently low CPU utilization.

Key Takeaways

• Quickest independently measured first chunk on a 10k-token workload: Baseten serving gpt-oss-120b at 0.23s, per Synthetic Evaluation.
• Throughput and TTFT are completely different merchandise: Cerebras hits 1,697 tok/s however 0.49s TTFT; Inception’s Mercury 2 hits 770 tok/s at 3.07s.
• Vendor latency claims like ElevenLabs’ 75ms and Cartesia’s sub-90ms are mannequin inference time solely, excluding community.
• Reasoning effort is the one largest TTFT lever: Gemini 3.1 Flash Stay goes 0.96s to 2.99s between Minimal and Excessive.
• TTFT alone doesn’t predict how an agent feels. Time-to-first-sentence does, as a result of speech synthesis wants a full clause.

Sources

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its reputation amongst audiences.