Why are YouTube's auto captions wrong sometimes?

Noise, music, overlapping speakers, accents and fast speech all blur the sounds the acoustic model hears, and the language model then guesses from context. On the stage, loud noise leaves sounds no word fits, and fast speech merges a doubled sound so a word disappears.

How does YouTube know when to show each caption?

The recognizer knows which frames of audio each word came from, so every word has a start and end time. Captions are grouped from those words and shown over the same stretch of the video.

Why are auto captions better for some videos than others?

A recognizer does best on speech like the speech it learned from. Google's 2009 launch was trained on English broadcast news, and its team wrote that it worked well on a single speaker talking clearly and struggled with background sound and varied speech.

Why does it write the wrong word when the words sound the same?

Homophones like right and write have the same sounds, so only the language model can choose, from the words around them. When context is weak, it can pick the wrong one. Turn the language model off on the second stage and “right” becomes “rite”.

Live model · YouTube auto captions

How YouTube Auto Captions Actually Work

YouTube's auto captions run speech recognition on the audio. The audio becomes a spectrogram, a slice every 10 milliseconds. An acoustic model gives every slice a probability for every sound, the likeliest path collapses into sounds, and a search with a language model turns those into words, timed to the frames they came from.

Sound to sounds: every frame scored, then collapsed

Someone says “turn right at the light”: 47 frames of sound, 1,880 ms. Each column of the spectrogram is one frame, and each phoneme leaves its own pattern across the bands. Silence sits at both ends, and between words when the speech is unhurried.

StopCTC

SaidBackground noiseSpeechSeed

CH1 A run of one label, merged

Watch the Python behind this model run, line by line

How to read the stage

Someone says “turn right at the light” in a busy room: 47 frames. The acoustic model scores every frame against every sound, and the line marks the top label. Merge the runs and drop the blanks and 14 sounds are left, one more than was said: noise turned one frame into an L. Try loud noise under Break it, then no noise at all.

Why the blank

This is CTC, connectionist temporal classification, from Graves and colleagues in 2006: a network labels every frame, and the collapse merges repeats, then removes blanks. Merging is what lets one sound span many frames. It's also why a doubled sound needs a blank between: say “at two” fast and the two T's merge into one, so the sounds come out AE T UW.

From sounds to words

R AY T could be right, write or rite; only context tells them apart. The second stage searches word by word, scoring each caption by how well its words fit the sounds and how likely the words are in order, and keeps the best 4. It repairs the stray L and writes “turn right at the light”. With the language model off, the same sounds come out “turn rite at the light”.

Sounds to words: a beam search with a language model

The first stage heard 14 phonemes: T ER N R L AY T AE T DH AH L AY T. Now find words that sound like them. Each caption so far keeps a cost, sound plus language, and only the best 4 survive each step.

StopBeam search

SaidBackground noiseSpeechBeam widthLanguage modelSeed

Step0In the beam0 of 4

Heard: the phonemes from the first stage

The search: captions kept (beam of 4) and dropped, with their cost

The beam, best first

Caption so far	Sound	Language	Total
(none)

The caption

On	Off	Text
(none)

CH1 Kept in the beam

Watch the Python behind this model run, line by line

What YouTube adds

Google launched automatic captions on YouTube in 2009, using the speech recognition behind Google Voice; the first models worked best on clear speech in English. Since 2017 they also caption [APPLAUSE], [MUSIC] and [LAUGHTER], from a separate model of sounds. Today's recognizers are end-to-end networks: Google's Conformer reads 80 filterbank channels from a 25 ms window every 10 ms.

Timing and lines

Each word keeps the frames it came from, so the caption knows when to appear: here from 0:00.08 to 0:01.80. Caption guidelines then shape the text for reading. The Described and Captioned Media Program's key asks for lines of 32 characters or fewer, one or two lines on screen, and no more than 130 to 160 words a minute depending on the audience.

What the model leaves out

The sounds here are hand-written patterns over 10 bands, not learned from recordings, and the acoustic model compares patterns instead of running a trained network. Real decoders sum every path that collapses to the same sounds instead of taking only the best one, and search sounds and words together. The language model knows 14 words; YouTube's knows a language, and its vocabulary is open.

Questions

Why are YouTube's auto captions wrong sometimes?: Noise, music, overlapping speakers, accents and fast speech all blur the sounds the acoustic model hears, and the language model then guesses from context. On the stage, loud noise leaves sounds no word fits, and fast speech merges a doubled sound so a word disappears.
How does YouTube know when to show each caption?: The recognizer knows which frames of audio each word came from, so every word has a start and end time. Captions are grouped from those words and shown over the same stretch of the video.
Why are auto captions better for some videos than others?: A recognizer does best on speech like the speech it learned from. Google's 2009 launch was trained on English broadcast news, and its team wrote that it worked well on a single speaker talking clearly and struggled with background sound and varied speech.
Why does it write the wrong word when the words sound the same?: Homophones like right and write have the same sounds, so only the language model can choose, from the words around them. When context is weak, it can pick the wrong one. Turn the language model off on the second stage and “right” becomes “rite”.