Text-only LLM work is well-trodden by now. The requests landing on our desk in 2026 increasingly start with "we want users to talk to it" — a support line that answers in under a second, a clinical dictation tool that gets drug names right, a drive-through order taker, a field technician who cannot type with gloves on. Voice changes the engineering problem completely: quality is no longer the hard part, latency and streaming are.
This tutorial builds the whole path in PyTorch: adapt Whisper to your domain vocabulary with LoRA, wrap it in a streaming transcriber that emits partial text as audio arrives, bolt on VAD-based turn detection, and hit a conversational latency budget end to end. The code is runnable; the numbers are the ones we hold client deployments to.
The latency budget is the spec
Humans perceive a conversation as natural when the gap between "I stopped talking" and "it started talking" stays under roughly 800 ms. Past about 1.2 s it feels like a bad phone connection and users start talking over the bot. That single number constrains every component:
| Stage | Budget | Notes |
|---|---|---|
| Network + audio buffering | 50-100 ms | 20 ms Opus frames, jitter buffer |
| Endpointing (VAD silence hold) | 200-400 ms | The largest and most tunable cost |
| Final ASR pass on the last chunk | 80-150 ms | Streaming means most audio is already transcribed |
| LLM time-to-first-token | 150-300 ms | Prefill only; prompt length matters here |
| TTS time-to-first-audio | 80-200 ms | Streaming TTS, not file-at-a-time |
| Total to first audio | 600-1100 ms |
Two design consequences fall straight out of that table. First, ASR must be streaming: if you wait for the utterance to end before you start transcribing, you pay full ASR latency inside the budget instead of overlapping it with the user's own speech. Second, every stage must stream into the next — first token into TTS, first TTS chunk onto the socket. A pipeline that is fast but batch-shaped misses the budget anyway.
Step 1: adapt Whisper to your domain with LoRA
Off-the-shelf Whisper large-v3 is strong on general speech and weak exactly where it costs you money: product SKUs, drug names, street names, carrier jargon, accents underrepresented in training. Full fine-tuning a 1.5B encoder-decoder is unnecessary; LoRA on the attention projections gets most of the win from a few hours of labelled audio on one GPU.
import torch
from datasets import Audio, load_dataset
from transformers import (WhisperForConditionalGeneration, WhisperProcessor,
Seq2SeqTrainer, Seq2SeqTrainingArguments)
from peft import LoraConfig, get_peft_model
MODEL = "openai/whisper-large-v3"
processor = WhisperProcessor.from_pretrained(MODEL, language="en", task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(MODEL, torch_dtype=torch.bfloat16)
model.config.forced_decoder_ids = None
model = get_peft_model(model, LoraConfig(
r=32, lora_alpha=64, lora_dropout=0.05, bias="none",
target_modules=["q_proj", "k_proj", "v_proj", "out_proj"],
))
model.print_trainable_parameters() # ~1% of params trainable
ds = load_dataset("your-org/support-calls").cast_column("audio", Audio(sampling_rate=16_000))
def prepare(batch):
audio = batch["audio"]
batch["input_features"] = processor.feature_extractor(
audio["array"], sampling_rate=16_000).input_features[0]
batch["labels"] = processor.tokenizer(batch["text"]).input_ids
return batch
ds = ds.map(prepare, remove_columns=ds["train"].column_names, num_proc=4)
The collator is the part people get wrong — Whisper wants fixed-length 30 s mel features but variable-length labels, and the BOS token must be stripped because the model prepends it during teacher forcing:
from dataclasses import dataclass
@dataclass
class WhisperCollator:
processor: WhisperProcessor
def __call__(self, features):
batch = self.processor.feature_extractor.pad(
[{"input_features": f["input_features"]} for f in features],
return_tensors="pt")
labels = self.processor.tokenizer.pad(
[{"input_ids": f["labels"]} for f in features], return_tensors="pt")
ids = labels["input_ids"].masked_fill(labels.attention_mask.ne(1), -100)
if (ids[:, 0] == self.processor.tokenizer.bos_token_id).all():
ids = ids[:, 1:]
batch["labels"] = ids
return batch
trainer = Seq2SeqTrainer(
model=model,
args=Seq2SeqTrainingArguments(
output_dir="whisper-lora-support",
per_device_train_batch_size=8, gradient_accumulation_steps=2,
learning_rate=1e-3, warmup_steps=50, num_train_epochs=3,
bf16=True, gradient_checkpointing=True,
predict_with_generate=True, eval_strategy="epoch",
remove_unused_columns=False, label_names=["labels"],
),
train_dataset=ds["train"], eval_dataset=ds["validation"],
data_collator=WhisperCollator(processor),
)
trainer.train()
model.save_pretrained("whisper-lora-support") # adapter only, ~80 MB
Practical notes from real engagements:
- Normalize your transcripts the way you will evaluate them. Decide up front whether "twenty dollars" or "$20" is the target, and be consistent in both training labels and WER scoring. Half the "fine-tuning made it worse" reports we investigate are normalization mismatches, not model regressions.
- LoRA learning rates are high:
1e-3atr=32is a reasonable start, roughly 100x what you would use for full fine-tuning. - Three to ten hours of in-domain audio usually cuts domain WER by 20-40%. Below one hour, prefer prompt biasing (the
initial_prompt/ previous-text tokens) over training. - Evaluate entity error rate, not just WER. Nobody cares about a dropped "the"; everyone cares about a wrong dosage or order number. Score the spans that matter separately.
Step 2: make it stream
Whisper is an offline 30-second-window model. Making it stream is an inference-loop problem: keep a rolling audio buffer, re-decode it as new audio lands, and commit the prefix that has stopped changing between passes. That last idea — LocalAgreement — is what makes partial output stable enough to display and to forward downstream.
import numpy as np, torch
class StreamingTranscriber:
"""Rolling-buffer Whisper with LocalAgreement-2 prefix commitment."""
def __init__(self, model, processor, sr=16_000, window_s=25.0, step_s=0.75):
self.model, self.processor, self.sr = model, processor, sr
self.window, self.step = int(window_s * sr), int(step_s * sr)
self.buf = np.zeros(0, dtype=np.float32)
self.prev_tail = "" # previous pass's uncommitted text
self.committed = "" # stable text, safe to send downstream
self.since_decode = 0
@torch.inference_mode()
def _decode(self):
feats = self.processor.feature_extractor(
self.buf, sampling_rate=self.sr, return_tensors="pt").input_features
feats = feats.to(self.model.device, dtype=self.model.dtype)
ids = self.model.generate(feats, language="en", task="transcribe",
num_beams=1, max_new_tokens=180)
return self.processor.batch_decode(ids, skip_special_tokens=True)[0].strip()
def accept_audio(self, pcm_f32):
"""Feed 20-100 ms of float32 mono 16 kHz audio. Returns (committed, partial)."""
self.buf = np.concatenate([self.buf, pcm_f32])
self.since_decode += len(pcm_f32)
if self.since_decode < self.step:
return self.committed, self.prev_tail
self.since_decode = 0
text = self._decode()
# LocalAgreement-2: commit the longest word prefix two passes agree on.
new_words, old_words = text.split(), self.prev_tail.split()
agree = 0
while agree < min(len(new_words), len(old_words)) and \
new_words[agree] == old_words[agree]:
agree += 1
if agree:
self.committed = (self.committed + " " + " ".join(new_words[:agree])).strip()
self.prev_tail = text
# Trim once the buffer fills the window, keeping 5 s of left context.
if len(self.buf) > self.window:
self.buf = self.buf[-5 * self.sr:]
self.prev_tail = ""
return self.committed, " ".join(new_words[agree:])
Tuning that loop:
step_sis the latency/compute dial. Re-decoding every 0.5-1.0 s of new audio keeps a single L4 or A10 comfortably ahead of real time withlarge-v3-turbo; below 0.3 s you burn GPU re-transcribing the same samples.- Use greedy decoding (
num_beams=1) while streaming. Beams cost latency and buy almost nothing on short windows. - Buffer trimming is where streaming implementations leak accuracy. Always keep several seconds of left context, and prefer trimming at a committed word or a VAD silence boundary rather than mid-word.
large-v3-turbohas a 4-layer decoder instead of 32 and is roughly 5-8x faster for a small WER cost. For streaming it is almost always the right default; keep full large-v3 for offline batch re-transcription if you need the last point of accuracy.- Whisper hallucinates confident text on silence and on music. Gate every decode behind VAD — never feed it non-speech.
Step 3: turn detection, the part that decides how it feels
Endpointing owns 200-400 ms of the budget and nearly all of the perceived rudeness. A fixed 700 ms silence timer feels sluggish for quick answers and interrupts anyone who pauses to think. Silero VAD is a tiny torch-hub model that runs on CPU in microseconds per frame:
vad, _ = torch.hub.load("snakers4/silero-vad", "silero_vad")
class Endpointer:
def __init__(self, sr=16_000, hold_ms=350, thresh=0.5):
self.sr, self.hold, self.thresh = sr, hold_ms, thresh
self.silence_ms, self.spoke = 0, False
def __call__(self, frame_f32): # exactly 512 samples @ 16 kHz = 32 ms
p = vad(torch.from_numpy(frame_f32), self.sr).item()
if p >= self.thresh:
self.spoke, self.silence_ms = True, 0
else:
self.silence_ms += 32
done = self.spoke and self.silence_ms >= self.hold
if done:
self.spoke, self.silence_ms = False, 0
return done, p
Then make hold_ms adaptive instead of constant. Cheap heuristics that measurably improve how an agent feels:
- Short hold (~250 ms) when the committed transcript already reads as a complete clause ending in terminal punctuation; long hold (~700 ms) after a filler or trailing conjunction ("and...", "so...", "um").
- Longer hold mid-number or mid-spelling — people dictate account numbers in bursts.
- Speculative response: on the first silence tick, start the LLM prefill on the current committed transcript. If the user resumes, cancel. You recover 150-300 ms of time-to-first-token for the price of some wasted prefill, and prefix caching makes the retry nearly free.
- Support barge-in: if VAD fires while TTS is playing, stop playback immediately and truncate the assistant turn in history at what was actually spoken, not what was generated. Skipping that truncation is the classic bug where the agent believes it said things the user never heard.
Step 4: wiring the full loop
async def voice_turn(ws, asr, ep, llm, tts, history):
async for frame in ws.audio_frames(): # 32 ms float32 mono
done, _ = ep(frame)
committed, partial = asr.accept_audio(frame)
await ws.send_json({"type": "partial", "text": f"{committed} {partial}".strip()})
if done and committed:
history.append({"role": "user", "content": committed})
spoken = []
async for tok in llm.stream(history): # OpenAI-compatible endpoint
spoken.append(tok)
async for pcm in tts.stream(tok): # sentence-chunked TTS
await ws.send_audio(pcm)
if ep.interrupted(): # barge-in
history.append({"role": "assistant",
"content": "".join(spoken)})
return
history.append({"role": "assistant", "content": "".join(spoken)})
return
A serving shape that works in production: ASR and TTS as separate GPU services with their own dynamic batching, the LLM behind vLLM with prefix caching enabled (the system prompt is identical on every call — cache it), and a thin stateful per-session process holding the buffers and the socket. Keep the LLM prompt short: prefill latency is linear in prompt length and sits directly inside your first-token budget. A 4,000-token system prompt is a quarter second of dead air on every turn.
Also do the boring arithmetic. Hosted streaming ASR and realtime speech APIs are excellent; a self-hosted PyTorch stack wins when data residency, per-minute cost at volume, or domain WER on your specific vocabulary is the binding constraint. Decide which of those applies before you build.
What to measure
- Turn latency, p50 and p95, from the last user speech frame to the first audio byte out. This is the product metric; everything else is diagnostic.
- WER and entity error rate on a frozen in-domain set, streaming path versus offline path. Streaming should cost 1-3 points of WER, not 10.
- Commit lag: seconds between a word being spoken and being committed. Past ~1.5 s the LLM is reasoning on stale input.
- Interruption accuracy: false barge-ins (agent stops for background noise) versus missed barge-ins (agent talks over the user).
- Real-time factor per ASR replica under concurrency, to size the fleet. Keep it below 1.0 with headroom or requests queue and latency cascades.
Takeaways
Voice agents are not an LLM feature; they are a real-time systems problem with an LLM inside. The model work — LoRA on Whisper for your vocabulary — is one afternoon and delivers a large, measurable accuracy win. The engineering work — streaming buffers, prefix commitment, adaptive endpointing, barge-in, speculative prefill — is what separates a demo from something people will talk to twice. Instrument turn latency on day one and tune against it.
Building a speech or voice pipeline on PyTorch and want a second pair of eyes on the latency budget? Get in touch — streaming ASR adaptation and real-time serving architecture are squarely what our PyTorch consultants do.