Speaker diarization is the process of partitioning audio by who spoke when. It splits a recording into segments and labels each one Speaker 1, Speaker 2, and so on, without knowing who those people actually are. It answers "who spoke when," a separate question from what was said and from matching a voice to a named person.
How speaker diarization works
A diarization pipeline usually runs in stages. Voice activity detection first marks which parts of the audio contain speech. That speech is cut into short segments, and each segment is turned into a speaker embedding, a numerical fingerprint of the voice such as an x-vector.
The embeddings are clustered so segments with similar fingerprints share a label, and the number of clusters gives the speaker count. Harder audio adds overlapped-speech detection for moments when two people talk at once.
Quality is scored with diarization error rate, which measures how much of the audio was assigned to the wrong speaker, including missed and overlapping speech.
A brief history of speaker diarization
Diarization grew out of speaker segmentation and clustering research in the late 1990s, applied first to broadcast news and recorded telephone conversations. It took modern shape through the NIST Rich Transcription evaluations of the 2000s, which framed the "who spoke when" task and standardized diarization error rate for scoring it.
Diarization versus identification
Diarization and speaker identification are easy to confuse, but they answer different questions. Diarization produces anonymous, relative labels: it knows there are three distinct voices but not whose they are.
Speaker identification goes further and matches a voice against known, enrolled profiles to name the person, which pulls in voice biometrics and its consent and privacy obligations. Many transcription products offer diarization out of the box; identification is a separate, more sensitive capability.
Speaker diarization use cases
Diarization earns its place anywhere a transcript carries more than one voice. Meeting transcription uses it to attribute action items to the right person. Contact-center transcription separates agent from caller so analytics can score each side of the call.
Interviews, podcasts, and medical and legal recordings all rely on it to turn a wall of text into a readable, attributable transcript.
Why diarization matters for speech-to-text
For meetings, interviews, podcasts, and calls, a transcript without speaker labels is far less useful, so diarization is a common add-on to a speech-to-text API. Support is uneven, though, and often model-specific: a provider may offer diarization on one engine and not another, or only for pre-recorded audio rather than live streams.
If speaker labels matter to your use case, confirm which model and which mode, batch or streaming, actually support them before you commit.
Frequently asked questions
What is speaker diarization and how does it work? Speaker diarization partitions audio by who spoke when. It detects speech, splits it into segments, turns each into a voice embedding, and clusters those embeddings so each speaker gets a consistent label like Speaker 1 or Speaker 2.
Does OpenAI Whisper support speaker diarization? Not on its own. Whisper transcribes speech but does not label speakers. Diarization is usually added with a separate library such as pyannote, or by a provider that bundles the two behind one API.
What is the best diarization API? No single diarization API wins across the board. Accuracy varies by audio type, number of speakers, and overlap, and it is measured with diarization error rate. Test candidates on your own recordings rather than trusting a headline number.