Whisper is an open-source speech recognition model family that OpenAI released in 2022. It was trained on roughly 680,000 hours of multilingual, weakly supervised audio scraped from the web, and it transcribes or translates across close to 99 languages. Because the weights ship free under an MIT license, Whisper became the default open baseline that much of the industry benchmarks against.
A brief history of Whisper
OpenAI released Whisper in 2022 and described its method in a paper on large-scale weak supervision. Instead of chasing a cleaner training set, the team scaled up the messy alternative, pairing a huge pile of web audio with the transcripts that already accompanied it.
The lineup grew over time. The first release ran in sizes from tiny to large, a stronger large-v3 followed in 2023, and a faster turbo variant arrived later, each keeping the same encoder-decoder design.
How Whisper works
Whisper is an encoder-decoder Transformer. Incoming audio is split into 30-second chunks, converted to a log-mel spectrogram, and fed to the encoder, which turns it into a sequence of vector embeddings. The decoder then generates text tokens one at a time.
Whisper is a multitask model steered by special tokens: they tell it whether to transcribe or translate, which language to assume, and whether to emit timestamps. It ships in sizes from tiny to large-v3, plus a faster turbo variant, trading accuracy for speed and memory.
Whisper's strengths and weaknesses
Whisper's strength is that it works well out of the box. Its large, varied training set lets it handle accents, background noise, and unfamiliar domains without any tuning, and its language coverage is broad.
Whisper's weaknesses matter just as much. The base model works on 30-second windows and does not natively stream, so live use needs extra engineering.
Whisper has no built-in diarization, so it cannot tell you who spoke. And it is prone to hallucination, inventing fluent text on silence, music, or steady noise.
Why Whisper matters
Whisper powers a large share of hosted APIs and self-hosted deployments, and many commercial engines are Whisper derivatives or benchmark themselves against it.
When you compare providers, it helps to know which ones run a Whisper variant and which built their own architecture, like wav2vec descendants or Conformer systems. That choice explains a lot about a provider's language coverage, its pricing, and the specific failure modes you should test for, silence hallucination first among them.
Frequently asked questions
How does OpenAI Whisper work? It converts audio into a log-mel spectrogram, encodes it with a Transformer, and decodes text token by token. Special tokens control language, transcription versus translation, and timestamps.
Does Whisper support speaker diarization? No. Whisper transcribes what was said but not who said it. For multi-speaker audio you layer a separate diarization system, such as pyannote.audio or NeMo, on top of Whisper's output.
How many languages does Whisper support? Whisper handles transcription and translation across close to 99 languages, though accuracy varies widely by language depending on how much training data existed for each.
Is OpenAI Whisper free? The model weights are open under an MIT license, so running Whisper yourself is free apart from compute. OpenAI's hosted transcription API is a separate paid service.