Local Voice Agent

Local Voice Agent

Design and evaluation plan, September 2026, with early prototype measurements. Full build and benchmarks scheduled.

A voice agent that runs entirely on one Apple M2 Max (96 GB). Small models make every split-second decision (has the caller finished, is that "mm-hm" an interruption, which skill handles this, is the reply safe to speak), and the LLM only writes the reply. An early prototype exists in a private repo; this page shows excerpts from it and the only measurements taken so far.

The problem

Voice agents demo well and then fail in four places.

• Turn-taking. A fixed silence timer either cuts people off mid-thought or leaves dead air after every sentence.
• Barge-in. A plain VAD treats every "yeah" and "right" as an interruption, so the agent stops talking whenever the caller signals they are listening. When a real interruption does land, most agents keep the whole unspoken reply in their context and carry on as if the caller heard it.
• Latency. The gap before the agent's first audio is spread across six stages. Without per-stage attribution nobody knows which one to fix.
• Reliability. A single-call demo says nothing about several callers sharing one machine.

What changed in 2026

1. End of turn is decided from audio. Smart Turn v3.2 (BSD-2, 23 languages, 16 kHz input up to 8 s, public train and test data) reads prosody and pace rather than a transcript. Daily measured the v3.1 8 MB CPU model at 9 ms per inference on an AWS c7a.2xlarge plus 7 ms preprocessing (Daily, 2025-12-03). LiveKit's Turn Detector v1 (2026-06-17) adds audio encoders to an LLM backbone, so it is multimodal rather than audio-only.
2. Eager replies. Pipecat v1.9.0 (2026-09-11) starts the reply on a predicted end of turn, holds it until the turn is confirmed, discards it if the user keeps talking, and never runs a tool call on an unconfirmed turn. The same release added LatencyBreakdown.contributions for per-stage latency.
3. ML barge-in. LiveKit's adaptive interruption (2026-03-19) reports 86% precision at 100% recall and 51% fewer false interruptions than VAD alone. These are vendor figures, and the model is free only on LiveKit Cloud.
4. Barge-in trims memory to what was heard. Deepgram Flux TTS (2026-08-13) returns spoken and unspoken text on interruption for exactly this purpose.
5. Streaming STT runs locally on Apple Silicon. Voxtral Realtime (arXiv 2602.11298, rev. 2026-04-06) is natively streaming, Apache-2.0, and matches Whisper at 480 ms delay. mlx-audio v0.5.6 (2026-09-24) runs it on Metal. Apple's ChipChat (arXiv 2509.00078) had already shown a sub-second cascade built entirely in MLX on a Mac Studio.

The bar is now set by hosted full-duplex models. OpenAI's GPT-Live-1 reached the API on 2026-09-10 at $0.05 per session minute. OpenAI reports 0.798 s turn-taking latency; that is a vendor turn-taking figure, and Artificial Analysis independently measures its time to first audio at about 1.24–1.34 s. Google's Gemini 3.8 Live and 3.8 Live Extended Thinking followed on 2026-09-15; Extended Thinking leads the Artificial Analysis S2S Index at 82.6 (accessed 2026-09-24). Open full-duplex models need NVIDIA GPUs, so on a Mac the agent is a cascade.

Architecture

flowchart LR
  MIC["Caller audio
browser or fixture WAV"] --> VAD["Silero VAD
CPU, gates the stream"]
  VAD --> EOT["Smart Turn v3.2
CPU, end of turn"]
  VAD --> STT["Voxtral Realtime
MLX, streaming"]
  VAD --> OVL{"Overlap decision
lexicon, then VAP / Laya"}
  STT --> OVL
  STT --> AGG["User turn aggregator"]
  EOT --> AGG
  AGG --> RTR{"Router
rules, then Laya"}
  RTR -- canned --> TTS
  RTR -- needs LLM --> LLM["Local LLM
mlx-lm, OpenAI API on localhost"]
  LLM --> GRD["Qwen3Guard-Stream
reply safety"]
  GRD --> TTS["Kokoro-82M
ONNX, CPU"]
  TTS --> CUR["Playback cursor
paced release, heard text"]
  CUR --> OUT["Speaker"]
  OVL -. "barge-in: rewrite context" .-> CUR

Everything runs in one process. VAD, Smart Turn and Kokoro use the CPU; Voxtral and the LLM share the Metal GPU. Every turn writes one JSONL row: Pipecat's latency breakdown plus route, guard wait and prompt risk.

Event flow

sequenceDiagram
  participant C as Caller
  participant V as VAD + Smart Turn
  participant S as STT
  participant L as LLM + Guard
  participant P as Playback cursor
  Note over C,P: 1. Normal turn
  C->>V: speech, then silence
  V->>V: Smart Turn: turn complete
  S->>L: final transcript
  L->>P: safe sentence, then TTS audio
  P->>C: audio at playback speed
  Note over C,P: 2. Backchannel while the agent talks
  C->>V: "mm-hm"
  V->>P: pause
  S->>V: final "mm-hm" matches lexicon
  V->>P: resume, no user turn
  Note over C,P: 3. Real interruption
  C->>V: "wait, stop, I meant..."
  V->>P: pause
  S->>V: partial transcript, not a backchannel
  V->>P: interruption
  P->>P: heard text = played sentences + played share of the cut one
  P->>L: rewrite last assistant turn to heard text before next LLM call

How each decision is made

Laya's own model card says it degrades on high-cardinality choices, so the intent taxonomy stays at 20 or fewer; past about 50 intents a plain fine-tuned encoder is the fallback. Its act/escalate head is not used; its confidence is.

Everything stays local. An early draft quoted Laya latency on a remote T4, but a network round trip to a shared GPU has no place inside a barge-in decision, where the agent's audio is already paused and the caller is waiting. The only latency that counts is on the M2 Max, and it is unmeasured, so it is a benchmark item. Review also noted that a VAP model predicts hold versus shift straight from audio with no STT, so it gets its own test arm.

Where the prototype stands. The prototype runs Smart Turn, the lexicon and rules for overlap, a rules router (handoff and goodbye get canned replies without calling the LLM), Qwen3Guard-Stream and the playback cursor. Laya and VAP plug in behind the same two interfaces and are not wired yet. The guard scores each finished sentence in one forward pass rather than token by token; attention is causal, so each token gets the verdict it would get streamed, but the reply waits for the sentence to end. Eager end of turn is not in the prototype: in Pipecat 1.10 it comes only from STT services that predict end of turn themselves, and neither Smart Turn nor Voxtral is wired to it.

Code walkthrough

The heard-text cursor. Audio is released at playback speed, so bytes released minus a small lead is bytes played. On interruption, the unplayed remainder is subtracted and the cut sentence contributes a time-proportional word prefix.

    def heard_text(self) -> str:
        """Whole sentences played, plus a time-proportional word prefix of the cut one."""
        heard, left = [], self.played
        for text, n in self.sentences:
            if left >= n:
                heard.append(text)
                left -= n
                continue
            words = text.split()
            k = int(len(words) * left / n) if n else 0
            if k:
                heard.append(" ".join(words[:k]) + "…")
            break
        return " ".join(heard).strip()

        if isinstance(frame, InterruptionFrame):
            await self._stop_release()
            unplayed = max(0.0, self._clock - time.monotonic()) * self._bps  # released, still queued
            self.cursor.played = max(0, self.cursor.played - int(unplayed))
            full, heard = self.cursor.full_text(), self.cursor.heard_text()
            if heard != full:  # every user turn start interrupts; only a real cut needs a rewrite
                self._pending = (full, heard)

voice/playback_cursor.py, local-voice-agent, branch adhoc-104-build (two non-contiguous parts of the file).

The overlap decision. Rules answer first. The classifier interface is where VAP or Laya plugs in, so swapping them changes no pipeline code.

LONG_OVERLAP_MS = 1200


class RulesOverlapClassifier:
    """Default (after the lexicon): long speech yields, short untranscribed sound is noise,
    any other words yield."""

    def classify(self, state: OverlapState) -> Overlap:
        if not _norm(state.user_partial):
            return "barge_in" if state.overlap_ms >= LONG_OVERLAP_MS else "noise"
        return "barge_in"


def decide_overlap(state: OverlapState, classifier: OverlapClassifier | None = None) -> Overlap:
    """Lexicon fast path (no model), then the pluggable classifier."""
    if is_backchannel(state.user_partial):
        return "backchannel"
    return (classifier or RulesOverlapClassifier()).classify(state)

voice/decisions.py, local-voice-agent, branch adhoc-104-build.

Wiring the turn strategies. The overlap logic starts user turns; Smart Turn ends them.

    context = LLMContext([{"role": "system", "content": SYSTEM}])
    cursor, stats = PlaybackCursor(), {}
    overlap = OverlapTurnStartStrategy(player=cursor)
    turns = UserTurnStrategies(
        start=[overlap],
        stop=[TurnAnalyzerUserTurnStopStrategy(turn_analyzer=LocalSmartTurnAnalyzerV3())],
    )
    aggregators = LLMContextAggregatorPair(
        context,
        user_params=LLMUserAggregatorParams(vad_analyzer=silero_v6(), user_turn_strategies=turns),
    )

voice/agent.py, build(), local-voice-agent, branch adhoc-104-build.

Early measurements

Early prototype measurement, one scripted turn set, single M2 Max laptop. n = 3 turns: one fixture WAV (fixtures/turn_capital.wav, synthetic speech) replayed three times through the full local cascade, in one run of about 23 s. Git SHA b13e077, clean tree, no API spend. Time is measured from the end of the caller's speech to the first agent audio, split by Pipecat's LatencyBreakdown. The machine was heavily loaded: the 1-, 5- and 15-minute load averages at start were 25.6, 21.4 and 19.7. These three rows say nothing about load, concurrency or any other machine.

In both later turns, the final Voxtral transcript takes the largest share, then Kokoro, then the LLM. The reply-safety check added 24–37 ms. All three turns were routed to chat and the prompt was scored Safe.

Versions recorded in the run manifest. pipecat-ai 1.10.0, mlx 0.32.2, mlx-audio 0.5.4, mlx-lm 0.31.3, silero-vad 6.2.1, onnxruntime 1.24.4, kokoro-onnx 0.6.1, torch 2.14.0, transformers 5.17.0. LLM mlx-community/Qwen3-4B-Instruct-2507-4bit, Kokoro voice af_heart, macOS 15.6.1 on an M2 Max with 96 GB. These differ from the plan, which pins Pipecat 1.11.0, mlx-audio 0.5.6 and Silero 6.2.x at the latest release; the newer Pipecat and mlx-audio releases were still inside a dependency quarantine window when the prototype was built, and nothing in it needs them.

How it will be measured

Every run records the git SHA, pinned versions, hardware samples and raw JSONL, and charts are generated only from those files. Published figures appear as labelled references, never mixed into our series.

• End of turn: LiveKit eot-bench (Apache-2.0, 14 languages). Smart Turn v3.2 against a VAD-silence baseline. Chart: x = added latency (ms), y = false-cutoff rate (%). LiveKit's published v1 points (9.9% at 300 ms, 4.5% at 600 ms) are overlaid as vendor-run references.
• STT: pipecat stt-benchmark. Voxtral Realtime at several delay settings and MLX Whisper. Chart: x = median time to final segment (ms), y = semantic WER (%). Published references: Nova-3 at 247 ms / 1.62%, Soniox at 249 ms / 1.29%.
• Duplex behaviour: Full-Duplex-Bench v1.5 subset (background speech, backchannel, interruption). The external anchor is Qwen-Audio-3.1-Realtime's reported drop in responses to background speech from 73.0% to 13.0% (2026-09-21).
• Barge-in tests. The FDB subset plus scripted clips with an interruption or backchannel at known offsets. Arms: a fixed 500 ms debounce, lexicon only, lexicon plus Laya, VAP only, and ML confirmation with and without cursor truncation. Metrics: false-yield rate, missed-interrupt rate, p50/p95 stop latency, and the share of turns whose context equals the played text.
• Routing: classifier vs LLM. Fine-tuned Laya against a local-LLM router and a plain fine-tuned encoder on the owner's intent set. Metrics: accuracy, calibration error, and escalation rate at a fixed routed precision.
• Latency attribution. Replayed turns through the full cascade, as horizontal stacked bars: y = configuration (eager on/off, STT, TTS), x = ms split by stage.
• Concurrent-caller load, ramp and hold (the LiveKit method, scaled to one laptop). Synthetic callers over WebSocket ramp 1, 2, 4, 6, 8, 12, 16 sessions at 2-minute steps, then hold the highest level that meets the SLO for 5 minutes. Chart: x = concurrent sessions, y = voice-to-voice p50/p95 (ms) with an SLO line at p95 ≤ 1.5 s; a second panel plots CPU % and GPU % against sessions, with errors annotated. With one Metal GPU shared by STT and the LLM, the knee is expected to be low; it will be reported as a single-laptop result.
• Hosted comparison. GPT-Live-1 and Gemini 3.8 Live on the same fixture audio only, under a capped API budget, at concurrency 1 and 5. Time to first response audio and stop latency are measured on our side, on the same axes as the cascade.

τ-Voice (2026-03-14) is a reminder of what comes after latency: voice agents completed only 31–51% of grounded tasks with clean audio, against 85% for text.

Proof of capability

The planned proof is an unedited screen recording of a live browser call to the local cascade on the M2 Max. A readout shows each turn's measured latency breakdown from LatencyBreakdown. The call includes a mid-sentence pause that does not trigger a reply, an "mm-hm" that does not interrupt, and a real interruption, after which the on-screen context shows the assistant turn cut to the words the caller heard. The per-turn JSONL for the call is published alongside it.

What's new versus the earlier telephony work

The earlier Real-Time Voice AI study covered telephony voice agents: Twilio Media Streams, VAD-gated Whisper, semantic endpointing on the transcript, per-clause TTS overlap and debounced barge-in. This project changes:

• Turn detection moves from text to audio. The earlier design called an is_complete() check on the transcript; here Smart Turn decides from prosody before STT finishes.
• Backchannels are classified, not debounced. A lexicon, a long-speech rule and a pluggable VAP or Laya classifier replace a speech-duration debounce.
• The agent's memory matches what was heard. The earlier study described using Twilio mark echoes to find the playback position; this project implements a played-audio cursor that rewrites the context on every real interruption.
• No cloud in the call path. Streaming STT, the LLM, the safety check and TTS all run on one laptop, instead of managed Deepgram or a separate GPU service.
• Measured, not budgeted. Every turn logs a per-stage breakdown, and the claims wait for the benchmarks above.

Sources

• Introducing GPT-Live-1 in the API, OpenAI developer community, 2026-09-10
• Artificial Analysis Speech-to-Speech leaderboard, accessed 2026-09-24
• Introducing Gemini 3.8 Live and 3.8 Live Extended Thinking, Google, 2026-09-15
• Pipecat v1.9.0 release, 2026-09-11; Pipecat releases, v1.11.0 on 2026-09-18
• Improved accuracy in Smart Turn v3.1, Daily, 2025-12-03; pipecat-ai/smart-turn, v3.2, accessed 2026-09-24
• Solving end-of-turn detection, LiveKit, 2026-06-17
• Adaptive interruption handling, LiveKit, 2026-03-19
• How to load test voice agents, LiveKit, 2026-03-23
• LiveKit eot-bench, accessed 2026-09-24
• Deepgram launches Flux TTS, audioXpress, 2026-08-13
• Benchmarking STT for voice agents, Daily, 2026-02-13
• Voxtral Realtime, arXiv 2602.11298, 2026-02-11, revised 2026-04-06
• mlx-audio, v0.5.6, 2026-09-24
• ChipChat, arXiv 2509.00078, 2025-08-26, revised 2026-07-19
• Qwen3Guard technical report, arXiv 2510.14276, 2025-10
• τ-Voice, arXiv 2603.13686, 2026-03-14
• Qwen-Audio-3.1-Realtime, arXiv 2609.25176, 2026-09-21