You feed Whisper a recording with a few seconds of silence, and it confidently transcribes "okay okay okay okay" — or the classic "Thank you. Thanks for watching!" over dead air. It is not a bug in your code. It is a direct consequence of how the model is built, and once you understand the mechanism, the fix is short and reliable.

🎬 The 60-second version is on Instagram — watch the reel. This post is the full engineering version: the why, the code, and the production gotchas the reel didn't have room for.

TL;DR

Whisper always outputs something for every audio window, even when there is no speech. Trained on speech, it reads noise as words, and its decoder can lock into a repetition loop. Three changes fix the vast majority of real-world cases:

  • A VAD gate (Silero-VAD) removes non-speech audio before it ever reaches Whisper.
  • no_repeat_ngram_size=3 breaks the decoder's repetition feedback loop.
  • temperature=0 forces greedy, deterministic decoding — no invented tokens.

Why Whisper invents words on silence

Whisper is an encoder–decoder model. The encoder turns 30 seconds of audio into a representation; the decoder is an autoregressive language model that generates text token by token, conditioned on that representation. The crucial detail: the decoder is a language model first. It was trained on a very large corpus of speech, and it always produces a plausible transcript.

When the audio is silence or non-speech noise, the encoder's representation carries little useful signal. The decoder does what language models do in the absence of evidence — it falls back on its prior, emitting the kinds of phrases that were common in training data. That is why the hallucinations are so recognizable: "Thank you," "Thanks for watching," "Please subscribe." Whisper's training set included a great deal of subtitled video, so those phrases sit at high probability when nothing else is competing.

The second half of the problem is the autoregressive loop. Each token the decoder emits becomes context for the next one. Once "okay" appears, the most likely next token becomes "okay" again, which makes the one after that even more likely. With no real audio to pull it out of the rut, the model spirals into repetition. Silence plus an autoregressive decoder is a feedback loop waiting to happen.

Fix 1: A VAD gate so silence never reaches the model

The most robust fix attacks the root cause: if Whisper never sees non-speech audio, it cannot hallucinate words from it. Voice Activity Detection (VAD) is a lightweight classifier that labels each chunk of audio as speech or non-speech. Silero-VAD is the popular choice — it is a tiny model (roughly a megabyte) and fast enough to run as a pre-filter with negligible overhead.

The pattern is simple: run VAD first, drop the non-speech segments, and hand only the speech to Whisper. In practice you rarely wire this up by hand, because production Whisper runtimes build it in. faster-whisper exposes it as a single flag, vad_filter=True, which runs Silero-VAD internally and transcribes only the retained speech regions.

Fixes 2 and 3: Break the loop and kill the randomness

VAD removes most hallucinations, but tuning the decoder handles the rest — including repetition inside genuine speech.

no_repeat_ngram_size=3 tells the decoder that no sequence of three tokens may appear twice. The instant a 3-gram would repeat, that path is blocked and the model is forced to choose a different continuation. This directly severs the "okay okay okay" feedback loop.

temperature=0 switches decoding to greedy: the single highest-probability token at every step, with zero sampling randomness. By default Whisper uses a temperature fallback schedule ([0.0, 0.2, ... 1.0]), escalating to higher temperatures when its internal quality checks fail — and higher temperature is exactly when invented words creep in. Pinning it to 0 keeps decoding deterministic and reproducible, which is also what you want for a testable pipeline.

Here is the whole fix in faster-whisper, which supports all three in one call:

python
from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

segments, info = model.transcribe(
    "meeting.wav",
    vad_filter=True,                  # Silero-VAD drops non-speech before decoding
    temperature=0,                    # greedy decoding, no random fallback
    no_repeat_ngram_size=3,           # block any 3-gram from repeating
    condition_on_previous_text=False, # stop a loop leaking into later windows
)

for seg in segments:
    print(f"[{seg.start:.2f} -> {seg.end:.2f}] {seg.text}")

The production gotchas the 60 seconds skipped

Three parameters clear most cases. A few more are worth knowing when you push this to real workloads:

  • condition_on_previous_text=False. By default, Whisper feeds each window's transcript in as context for the next one. That helps coherence on clean audio — but if one window hallucinates, that garbage becomes the prompt for the following windows and the error propagates. Turning it off transcribes each segment independently, which is more robust on noisy or long recordings at a small cost to cross-segment coherence.
  • The built-in thresholds. Whisper ships with its own guards you can tune: no_speech_threshold (how confident it must be that a window is silent before it emits nothing) and compression_ratio_threshold (a repetitive transcript compresses suspiciously well, so a high ratio flags a likely hallucination). If VAD plus decoding tweaks still leak the odd artifact, these are the next dials.
  • VAD is not free of tradeoffs. Aggressive VAD settings can clip quiet speech, soft consonants, or trailing words. If you start losing real content, loosen the VAD threshold rather than removing it.
  • Deterministic output is a feature. temperature=0 plus a fixed model means the same audio yields the same transcript every run — essential if you want regression tests over your ASR pipeline.

Key Takeaways

  • Whisper hallucinates on silence because its decoder is a language model that always emits something, falling back on training-data phrases when there is no real signal.
  • Repetition ("okay okay okay") is the autoregressive loop feeding on itself; there is nothing in the audio to break it.
  • VAD is the highest-leverage fix — stop non-speech from reaching the model at all.
  • no_repeat_ngram_size=3 breaks the loop; temperature=0 removes the randomness; condition_on_previous_text=False stops errors from propagating across windows.
  • For stubborn cases, tune no_speech_threshold and compression_ratio_threshold.

This is the deep-dive companion to a @teqnersinsights reel. Follow @teqnersinsights for the 60-second versions, and browse the full reel library — more Whisper and ASR deep-dives are on the way.

Further reading: faster-whisper transcription options · faster-whisper (SYSTRAN)

WhisperASRSpeech RecognitionPythonMachine Learning