Real-Time Voice AI: From Duplex to Native Speech-to-Speech

Every voice agent I've built has died the same way: the user speaks, a 1.5 second silence follows, and the person on the other end says "hello? hello?" twice before the answer finally surfaces. The model was never the bottleneck. The pipeline was. Transcription, reasoning, synthesis, playback. Each hop adds its own latency tax, and for voice, the tax compounds.

The last few weeks changed that equation. Google released Gemini 3.8 Live, a native speech-to-speech model that streams audio while it thinks. An open-weights project called Realtime-Venus does full-duplex conversation on a 9B parameter model. And Google Duplex, the 2018 system that started this whole thread, now reads as both quaint and prescient.

Here's the connection between all three: voice agents stopped being about making a call and started being about keeping a conversation alive while real work happens underneath.

Duplex taught the old rules

Google Duplex, announced in May 2018, was the first system that made phone calls sound like a person. It booked restaurant tables and hair salon appointments through the Google Assistant, calling real businesses and speaking to real receptionists. The research insight that made it work was a constraint: Duplex only operated in closed domains. A narrow task, like scheduling an appointment, could be explored exhaustively. General conversation was out of scope.

The architecture reads like a museum piece now. An RNN trained per domain on anonymized phone calls, fed by ASR output plus conversation history and task parameters. A concatenative TTS engine mixed with Tacotron and WaveNet for intonation control. Speech disfluencies, the "hmm"s and "uh"s, spliced in to signal that the system was still processing. A self-monitoring layer that flagged tasks the system couldn't complete and handed them to a human operator.

Two ideas from Duplex still matter today. First, latency was a design variable, not just something to minimize. When a user said "hello?", the system dropped to faster, lower-confidence models and occasionally heuristics, keeping response time under 100ms. But for complex sentences, Duplex deliberately introduced more latency, because an instant reply to a dense question feels wrong in human conversation. Second, the system had to know what it didn't know. Self-monitoring and human handoff kept a narrow system honest.

Native speech-to-speech collapses the stack

Native speech-to-speech models remove the cascade entirely. Gemini 3.8 Live, released in the Gemini API and Google AI Studio, is a single model that takes audio in and streams audio out, performing tasks while maintaining dialogue. The Extended Thinking variant adds configurable background reasoning for multi-step problems, and it ranks #1 on Artificial Analysis' Speech-to-Speech leaderboard.

The capability list reads like a checklist of every pain point from the cascade era:

• Async function calling. API and tool calls execute in the background while audio keeps streaming to the user. No more dead air while a lookup runs.
• Visual context. The model grounds dialogue in live video input, which matters for agents that watch what the user is doing.
• Alphanumeric precision. Confirmation codes, claim numbers, and technical data parse correctly, which cascaded stacks usually mangle.
• Incremental content updates. Real-time audio merges with structured data for context-aware responses.
• Multilingual support. Coverage for 97+ languages means you don't swap models per market.

Pricing works out to $0.005/min for audio input, about $0.30 an hour, and $0.018/min for output, roughly $1.08 an hour. For a typical support or concierge agent, that undercuts the GPU bill for self-hosting an equivalent open model.

Quick Take: Native speech-to-speech doesn't just cut latency. It changes what an agent can do while the user is still talking.

The latency budget is tighter than you think

The most useful analysis in the community discussion around this release wasn't about the model itself. It was the latency budget for real-time bidirectional voice. In a chatbot, a 1-2 second retrieval window for history or external context is acceptable. In live voice, human turn-taking demands a sub-500ms round-trip: speech-to-text, memory retrieval, LLM first token, text-to-speech. Any longer and the pause becomes jarring. If session state or caller memory takes more than 200ms to fetch, the illusion of natural conversation breaks. The rule that emerged: keep state retrieval deterministic and under 50ms when you're orchestrating memory alongside a streaming audio feed.

Key numbers: sub-500ms full round-trip for natural turn-taking; under 50ms for session-state retrieval; a 100ms response ceiling for simple utterances, which Duplex hit in 2018; about 2 seconds of silence before kids and callers start prompting the agent themselves.

That budget changes what you build. It means you can't bolt a vector search onto a voice agent and hope. The retrieval path needs to be cached, indexed, and predictable before the model cost even enters the conversation.

Transcription: where the average hides the hard cases

The companion release, Gemini 3.5 Transcribe, is the listening half of the stack. It hits a 4.0% average WER in streaming mode and 2.6% non-streaming across 85+ languages. Translation: on average one wrong word in every 25 while streaming, about one in 38 when you send a full audio file. Three features matter in production. Automatic code-switching handles intra-sentence and inter-sentential language switches without manual configuration. Custom vocabulary biasing steers recognition toward domain jargon, company names, and proper nouns with a custom_vocabulary list of up to 1,000 terms. Smart transcription mode strips filler words and disfluencies, producing reader-ready transcripts. The Interactions API extends this to files up to an hour long with structured timestamps and speaker labeling.

The "average" deserves suspicion. A developer who tested Spanish-English code-switching calls reported WER jumping to 15-20%, roughly four times the advertised figure. If your callers mix languages mid-sentence, engineer your confirmations and retries around that reality, not the marketing number.

The open-source route: Realtime-Venus

On the open side, inclusionAI published Realtime-Venus, a pair of checkpoints built on the MiniCPM-o 4.5 lineage. The Omni variant is a 9B parameter audio-visual model, which means it fits on a single consumer GPU with quantization. It does full-duplex interaction: it watches and listens continuously, decides whether and when to respond, and generates text and speech on a shared causal timeline. Proactive interaction is the standout feature. The model initiates a response when an event warrants it, without waiting for a user prompt. It distinguishes backchannels, interruptions, corrections, and redirections while it's speaking.

Delegation is the open-source answer to Gemini's async function calling. The model emits an in-stream <delegate> request on the causal timeline, and the Realtime-Venus-Harness runtime executes external tools asynchronously, feeding results back into the ongoing conversation. There's also training-free long-video memory that archives visually informative moments and retrieves relevant audio-visual context, plus native speech output via Token2wav with a reference voice.

The trade is yours to carry. You adopt the harness, the integrations, and the deployment. For teams with data-residency constraints or GPU already in the budget, that's a fair exchange. For everyone else, the closed API saves you a month of glue code.

Choosing between the stacks

The decision isn't about which is "better." It's about where your constraint sits.

Dimension	Gemini 3.8 Live	Realtime-Venus-Omni	Duplex (2018)
Model type	Native speech-to-speech (API)	Full-duplex audio-visual, open weights	Cascaded RNN + TTS
Best fit	Production agents needing tools mid-conversation	Self-hosted or on-prem interaction	Historical reference
Function calling	Async, background execution	<delegate> tokens + external harness	None
Latency handling	Streams audio while thinking	Continuous perceiving while speaking	Under 100ms on simple turns, deliberate pauses on complex ones
Language coverage	97+	Not stated (MiniCPM-o lineage supports zh/en)	English only
Proactive speech	No	Yes, initiates on event	No
Visual input	Yes	Yes (Omni variant)	No

What the community is saying

The async tool-call detail is the part worth pressure-testing first. Continuing the audio stream while a tool runs is great UX, but it forces you to define delivery semantics: what should the user hear if the tool times out, returns partial data, or completes after the conversation has moved on? I'd model each call with an idempotency key and a cancellable state machine, then surface queued, running, committed, and failed states instead of letting the voice layer imply success.

I run a voice interface on a low-power RISC-V board, where replies have to be short enough to hold an 8-year-old's attention. The bottleneck was never the model; it was the pipeline. Transcription, reasoning, TTS, transfer, playback. Every hop added latency, and kids notice about 2 seconds of silence. Running tool calls in the background while speech keeps streaming attacks exactly the right part of that problem. What I'd love documented more: how these models handle turn-taking with real-world noise. Barge-in and half-finished sentences are the norm with a child, not the exception.

Every voice agent I've built deadlocks the moment I need to call an API mid-conversation. But that 4.0% streaming WER across 85+ languages is suspicious. I tested transcription on mixed-language calls, Spanish-English code-switching, and the WER jumped to 15-20% easily. We need a per-language breakdown instead of an average that smooths over the hard cases.

Common pitfalls

• Don't trust the streaming WER average. The 4.0% figure is a mean over 85+ languages. On Spanish-English code-switching calls, a developer measured 15-20%. Test your actual language mix and build confirmations for anything alphanumeric.
• Treat async tool calls as state machines, not fire-and-forget. A tool can time out, return partial data, or complete after the conversation moved on. Give each call an idempotency key and model queued, running, committed, and failed states. Otherwise the voice layer asserts success that never happened.
• Spend your retrieval budget before you spend your model budget. If session state takes more than 200ms to fetch, the conversation feels dead. Keep state retrieval deterministic and under 50ms, and structure memory so it doesn't compete with the audio stream.
• Design for barge-in, not against it. Real-world noise means half-finished sentences and interruptions are normal, especially with children or call centers. Track pause counts and volume trends in the input stream, and treat interruption as a first-class signal.
• Match latency to expectation. Duplex deliberately slowed down on complex sentences because instant replies felt unnatural. Minimize latency on "hello?" and simple utterances, and give the model breathing room on dense questions.

One thing to remember

The models moved fast; the plumbing didn't. Duplex needed a hand-coded RNN per domain, cascaded ASR and TTS, and synthesized disfluencies to sound human. Gemini 3.8 Live and Realtime-Venus handle full-duplex audio natively and removed an entire class of pipeline latency. But you still own the boundaries: tool state, retrieval latency, barge-in, cancellation. The voice layer is happy to assert success. Your state machine decides whether that's true.

The bottom line

• If you're building a customer-facing voice agent that needs data lookups mid-conversation, adopt Gemini 3.8 Live. Async function calling keeps audio streaming while tools run, something a cascaded stack can't do without a 1-2 second stall.
• If you're constrained by data residency or GPU budget, skip the closed APIs and use Realtime-Venus-Audio. The 9B checkpoint runs on a single consumer GPU, but you adopt the harness as your own runtime and own the integration glue.
• One thing to watch: open-weights full-duplex models like Realtime-Venus are closing the gap fast. Expect sub-9B competitors and per-language WER reporting to become standard within six months, which will put real pressure on closed APIs to justify their pricing.