Beam search is the decoding algorithm that turns a model's step-by-step predictions into a final transcript. Instead of greedily taking the single best token at each step, it keeps the top few candidate sequences, the "beam," and extends them in parallel. That way it can pick the sentence that scores best overall rather than the one that looked best early.
How beam search works
Beam search holds a fixed number of partial hypotheses at each decoding step, a count set by the beam width. It expands each hypothesis with the next possible tokens, scores every resulting candidate using the model's probabilities and often an external language model, then prunes back to the best few.
When decoding ends, the algorithm returns the hypothesis with the highest total score. A wider beam explores more of the search space and usually improves accuracy, at the cost of more computation per step. Beam widths of 4 to 16 are common in practice, with diminishing returns past that.
A brief history of beam search
Beam search grew out of early speech recognition research. The Harpy system at Carnegie Mellon University, introduced in a 1976 dissertation, was the first use of the technique, and the term beam search was in circulation by 1977.
The origin explains the design. Beam search is a memory-limited variant of best-first search, built to explore only the most promising paths when tracking every candidate was too costly for the hardware of the day. That same tradeoff, accuracy against compute, still sets the beam width in modern decoders.
Greedy decoding versus beam search
Greedy decoding is the simplest strategy: take the single most probable token at every step. It is fast but shortsighted, because a locally attractive choice can lock the system out of a better complete sentence.
Beam search spends extra compute to avoid that trap, carrying several competing paths forward before it commits. The gap matters most when an early sound is ambiguous and later context resolves it, exactly the case greedy decoding cannot recover from.
With CTC models the same idea appears as beam search decoding over blank-and-token sequences.
Why beam search matters
In beam search, beam width is a direct accuracy-versus-latency knob. Streaming systems tend to use narrow beams to keep responses fast, while offline batch transcription can afford wider beams for a small accuracy gain.
Beam search also produces alternatives for free: an n-best list of ranked transcripts, or a lattice of competing paths that lattice rescoring can re-score with a heavier language model afterward. If a provider exposes decoding settings, beam width is often the lever, which is why the same model can feel either snappier or slightly more accurate depending on how it is tuned.
Frequently asked questions
What is beam search? It is a decoding method that keeps several high-scoring candidate sequences alive at once and extends them together, choosing the best complete transcript instead of committing to one token at a time.
How does beam search work? At each step it expands every kept hypothesis with possible next tokens, scores the results, and prunes back to the top few set by the beam width, returning the highest-scoring full sequence at the end.
How does beam search differ from greedy search? Greedy search takes the single most likely token at every step and never reconsiders. Beam search carries multiple paths forward, so a strong later context can rescue an early ambiguous choice.
How do you choose a beam width? Wider beams raise accuracy but cost compute and latency. Streaming favors narrow beams; batch jobs can go wider, with gains flattening out beyond roughly 16.