Speech-to-Text Pipeline: Whisper & Diarization Guide

Q: Should I use the Whisper API or self-host Whisper?

Run the volume math and the data-residency check, because the crossover is steep. Managed transcription APIs price around $0.006 to $0.01 per audio minute, which makes low and moderate volumes trivially cheap: a thousand hours costs a few hundred dollars, with zero operational burden and useful extras bundled — built-in diarization, streaming endpoints, word confidence. Self-hosting flips the economics at sustained scale: faster-whisper (the CTranslate2 reimplementation) transcribes at 10-20 times real-time on a single modern GPU, so one rented card can chew through a call center's entire daily volume for a fraction of per-minute pricing; at 100,000 hours a year the API bill exceeds $36,000 while the equivalent GPU time costs a small fraction of that. The second decisive factor is residency: if recordings contain customer conversations, health information, or anything that contractually cannot leave your infrastructure, self-hosting is not an optimization but a requirement. The pragmatic path most teams follow: start on an API to validate the use case in days, instrument cost per hour, and migrate the bulk batch workload to self-hosted faster-whisper when the line item or the compliance review forces the question — keeping the API for spiky overflow if useful.

Q: What is speaker diarization and how accurate is it?

Diarization answers "who spoke when": it partitions an audio stream into speaker turns and assigns anonymous labels (Speaker A, Speaker B), which alignment code then combines with the ASR's word timestamps to produce a speaker-attributed transcript. It is a separate model family from transcription — the open-source standard is pyannote-audio — working by embedding short voice segments and clustering the embeddings by voice similarity. Accuracy expectations should be set honestly: on clean audio with distinct voices and polite turn-taking, modern diarization is very good; error concentrates precisely where conversations get interesting — overlapping speech, short interjections, similar-sounding voices, and far-field microphones — and an 8-15 percent attribution error on messy multi-speaker meetings is normal even for strong systems. Two practical rules follow. First, when the recording source can separate speakers by channel — telephony records each party on its own stereo channel — split the channels and skip diarization entirely for that audio: it is free, perfect attribution. Second, treat speaker labels as probabilistic downstream: surface confidence, and do not build workflows that punish an individual based on an unverified diarization label. Identifying who Speaker A actually is (speaker identification) is a further step requiring enrolled voice profiles and appropriate consent.

Q: Why does Whisper hallucinate text and how do I prevent it?

Whisper is a sequence-to-sequence model trained to always produce text, so when it receives audio containing no intelligible speech — silence, music, background noise, or a non-speech segment — its failure mode is not an empty result but confident, fluent, entirely fabricated sentences; repeated phrases and looping outputs are the classic signature, and low-resource languages show it more. The defense is architectural, not prompt-level: never let non-speech reach the model. A voice-activity-detection (VAD) stage in front of the ASR — faster-whisper ships with an integrated VAD filter, and standalone options like Silero VAD are widely used — segments audio into actual utterances and drops silence and noise, which both eliminates most hallucination triggers and cuts compute, since you stop paying to transcribe dead air that can be a third of a raw call recording. Complementary guards: check Whisper's per-segment metadata (average log-probability and the no-speech probability it emits) and discard or flag segments where the model itself signals low confidence; cap and deduplicate pathological repeats in post-processing; and on long files, chunk at VAD-detected pause boundaries so a hallucination in one segment cannot cascade across the stitch. With VAD gating plus confidence filtering, hallucination drops from a routine embarrassment to a rare, catchable event.

Q: How do I transcribe long audio files with Whisper?

Whisper processes audio in 30-second windows, so long files need a chunking strategy, and the naive approach — slicing at fixed 30-second offsets — reliably cuts words in half and produces boundary errors. The production pattern: run VAD first to find natural pause boundaries, split the audio into chunks of roughly 30 seconds that end at those pauses, include a small overlap between consecutive chunks, transcribe chunks in parallel batches on the GPU, then stitch the transcripts using the word timestamps to reconcile the overlap regions. Long-form wrappers in the ecosystem — faster-whisper's built-in handling and community pipelines like WhisperX — implement this pattern with timestamp alignment. Practical additions that matter at scale: preserve original-audio offsets through every stage so each transcribed word carries an absolute timecode into the source file (indispensable for diarization alignment, audio citation, and redaction); process chunks as an idempotent batch job so a failed file resumes rather than restarts; and for very long recordings like multi-hour meetings, parallelize across chunks rather than files to keep GPU utilization high. Done this way, a single GPU works through hours of audio in minutes, with transcripts that read continuously across chunk boundaries.

Q: How do I make Whisper recognize custom vocabulary and product names?

Out of the box, ASR mangles exactly the words your organization cares most about — product names, people, acronyms, domain jargon — because they are rare or absent in training data, and every downstream system inherits the corruption: search indexes miss them, summaries garble them, RAG retrieves nothing. Whisper's built-in lever is the initial prompt: a string passed with each request that biases decoding toward the vocabulary it contains, so stuffing it with your product names, speaker names, and key terms measurably improves their recognition — cheap and worth doing on every deployment. Managed APIs expose stronger versions: keyword boosting or custom vocabulary lists with adjustable weights, and some offer custom model adaptation. Where the term list is large or the stakes are high, add a post-processing correction pass: a dictionary of known-bad-to-good mappings (fuzzy-matched against the transcript) fixes systematic errors deterministically, and an LLM pass with the vocabulary in context can repair the long tail — the transcript equivalent of spell-check with a domain dictionary. For self-hosted deployments with persistent needs, fine-tuning Whisper on a few hours of in-domain audio is genuinely effective but rarely the first move. Sequence: initial prompt everywhere, correction dictionary for the systematic offenders, then heavier options only if the error rate on critical terms still hurts.

Q: What does a speech-to-text pipeline cost at scale?

Anchor on the two cost shapes. Managed APIs charge per minute — roughly $0.006 to $0.01 for major providers, with features like diarization sometimes tiered — so cost scales linearly with audio volume forever: 1,000 hours costs $360-600, 10,000 hours costs $3,600-6,000, and 100,000 hours a year runs $36,000-60,000. Self-hosting converts that to compute time: faster-whisper on a single modern GPU transcribes at 10-20 times real-time, meaning one GPU-hour processes 10-20 audio-hours, so 100,000 audio-hours needs roughly 5,000-10,000 GPU-hours a year — a few thousand dollars of rented compute, or less on owned hardware running batch overnight. Diarization adds its own pass (pyannote runs near real-time on GPU), typically adding 20-50 percent to compute. The hidden costs that decide real projects: storage for audio and transcripts (trivial), engineering time for the pipeline (the real self-hosting cost — VAD, chunking, alignment, monitoring), and accuracy QA sampling. The honest crossover: below roughly 1,000-2,000 hours a month, APIs win on total cost once engineering time is priced in; above it, self-hosted batch wins and keeps widening; and data-residency requirements override the math entirely in favour of self-hosting.

Q: How do I run RAG or LLM summarization over transcripts?

Treat transcripts as first-class documents with three transcript-specific adaptations. First, chunk by conversational structure, not fixed tokens: speaker turns and topic shifts are the natural boundaries, and each chunk should carry its metadata — speakers present, absolute timestamps, source recording ID — so retrieval can filter ("only customer-call transcripts from March") and answers can cite the exact audio moment, the transcript version of a source link. Second, clean before you embed: raw ASR output full of filler words, false starts, and mangled jargon embeds poorly; run the custom-vocabulary correction pass and light normalization first, since retrieval quality downstream is capped by transcript quality upstream — garbage in, confidently-wrong answers out. Third, for summarization, feed the LLM speaker-attributed text with timestamps and instruct it to ground claims in specific turns; for long meetings, summarize hierarchically (per-section, then overall) rather than stuffing hours of transcript into one context. Confidence metadata earns its keep here: low-confidence regions can be flagged to the summarizer or excluded from high-stakes extraction like commitments and action items. Wired this way, recorded meetings and calls become a searchable, citable knowledge corpus rather than an audio graveyard.

Q: What is the difference between batch transcription and real-time voice AI?

They are different architectures optimized for opposite constraints, and conflating them produces bad designs in both directions. Batch transcription — this article's subject — processes recorded audio after the fact: latency is measured in minutes and nobody is waiting mid-sentence, so the design optimizes accuracy and cost-per-hour: run the largest model that is economical, apply full VAD and chunking, add diarization and post-processing passes, parallelize on GPUs at 10-20 times real-time. Real-time voice AI — live agents, phone bots, meeting captions — inverts every priority: a human is in the loop and the total round trip (hear, understand, respond, speak) must land under roughly 500 milliseconds to feel natural, so it uses streaming ASR that emits partial hypotheses while audio is still arriving, accepts a modest accuracy sacrifice for speed, cannot batch, and often colocates or fuses models to shave hops. The architectural consequence: do not force a streaming stack to do bulk historical transcription (you pay the latency-optimized premium for no benefit) and do not put a batch pipeline behind a live conversation (multi-second gaps kill the interaction). Most organizations need both eventually — a real-time path for live interactions and a batch path for the archive — sharing vocabulary and post-processing but differing in engine configuration and infrastructure.