Transcribe an hour-long recording with Whisper and something subtle goes wrong: the words are right, but by the end the timestamps are tens of seconds out of sync. Subtitles land after the speaker has moved on; your word-level alignment is useless. The transcript is accurate — the timing is not — and the reason is baked into how Whisper tells time.
🎬 The 60-second version is on Instagram — watch the reel. This post explains why the drift happens and the two fixes, with the tradeoff between them.
TL;DR
- Whisper does not read timestamps off a clock. It predicts timestamp tokens from the decoder, within 30-second windows, and small errors compound across a long file.
- Fix 1 — VAD chunking: cut the audio on silence (Silero-VAD) and re-anchor each segment to real audio time, so errors can't accumulate across the whole recording.
- Fix 2 — WhisperX: run a forced-alignment pass (wav2vec2) to get word-level timestamps accurate to roughly ±50 ms.
Why the drift happens
Whisper processes audio in 30-second windows. Within each window, it does not measure time directly — it emits special timestamp tokens interleaved with the text, and those tokens are predictions from the decoder, just like the words. The model is essentially guessing "this word started around here" from learned patterns, not consulting a sample-accurate clock.
On a short clip that is fine. Over a long recording it is not, because the errors are cumulative. Each 30-second window is stitched onto the previous ones, and any small offset in where a window's content is placed carries forward and adds to the next. A fraction of a second of slippage per chunk, repeated across an hour of audio, is how you end up 40 seconds out of sync by the end. The transcript text stays correct because recognizing words is a different, more robust task than pinning them to an exact moment.
Fix 1: VAD chunking
The first fix stops the errors from compounding by re-establishing ground truth periodically. Voice Activity Detection (Silero-VAD is the common choice) finds the silences and cuts the audio into speech segments on those boundaries. Each segment is then transcribed against its own known start time in the original file.
The key benefit is re-anchoring. Because every segment is tied back to a real position in the audio, a timing error inside one segment stays inside that segment — it cannot leak forward and accumulate across the whole recording the way it does with naive 30-second windowing. You also get a nice side effect: cutting on silence avoids slicing through the middle of a word. This is a lightweight fix that dramatically bounds the drift, though the per-word precision is still only as good as Whisper's own token timing within each segment.
Fix 2: WhisperX and forced alignment
When you need genuinely precise, word-level timestamps, the answer is to stop trusting Whisper's timing entirely and compute it directly from the audio. WhisperX does this with a second pass: after Whisper produces the transcript, WhisperX runs a wav2vec2 forced-alignment model that maps each transcribed word onto the acoustic signal, aligning phonemes to the waveform. The result is word-level timestamps accurate to about ±50 ms — far tighter than Whisper's native token timing.
The tradeoff is cost and complexity: WhisperX loads a second model and runs an extra alignment stage, which adds compute and dependencies. So choose by requirement. If you need roughly-right segment timing and want to stay lightweight, VAD chunking is enough. If you are building karaoke-style captions, dubbing, or anything that needs each word pinned to the audio, reach for WhisperX.
Key Takeaways
- Whisper's timestamps are predicted tokens, not measured time, so they drift and compound over long audio.
- VAD chunking re-anchors each segment to real audio time and bounds the drift — cheap and effective for segment-level timing.
- WhisperX adds a wav2vec2 forced-alignment pass for ~±50 ms word-level accuracy, at the cost of a second model.
- Pick by need: lightweight and roughly-synced → VAD; word-perfect alignment → WhisperX.
This is the deep-dive companion to a @teqnersinsights reel. If Whisper is in your stack, the sibling post Why Whisper Hallucinates on Silence — and the 3-Line Fix is worth a read. Follow @teqnersinsights for more, and browse the full reel library.