TL;DR
Earlybird is a compact, 29M-parameter streaming speech recognition model that achieves "negative latency" by predicting words correctly before the speaker even finishes saying them. By training on randomly truncated audio and learning to guess the next two words, it hits a median latency of -90ms on clean speech—beating much larger models while fitting into a tiny 16.8 MB (4-bit) footprint.
What Problem Does This Paper Address?
In the world of voice assistants, every millisecond of delay matters. Traditional streaming Automatic Speech Recognition (ASR) models typically wait until a word is finished—and often a few hundred milliseconds of "look-ahead" audio—before committing to a transcript. This creates a lag that prevents voice agents from planning replies or starting searches while the user is still talking.
Humans don't work this way; we often recognize a word from its first few phonemes and the surrounding context. Earlybird aims to give machines this same predictive capability, allowing them to output text early with a reliable confidence score so downstream systems can act faster.
Methods
Earlybird is built on the Moonshine streaming Tiny architecture (an encoder-decoder model) but introduces a novel training recipe:
- Training on Cut-off Audio: Instead of training on full utterances, the authors cut audio at random 80ms intervals. The model must predict the full word currently being spoken (even if it's only heard the start) plus a two-word "guess" of what comes next.
- Monotonic Targets: To ensure stability, the target at any time $t+1$ always extends the target at time $t$. This teaches the model not to output words it might have to "take back" later.
- Selective Distillation: It uses a larger "teacher" model (Moonshine Medium) to provide a distillation loss, but only at positions where the teacher is correct. This helps Earlybird learn to spread its probability over likely candidates when uncertain.
- Padding Frames: At each step, the encoder sees three frames of silence after the current audio. This helps the model "finish" a word as if the speaker had just stopped.
- Quantization: The model undergoes 4-bit quantization-aware training, shrinking it to just 16.8 MB without significant accuracy loss.
Datasets
- Training: A mix of LibriSpeech (960h), People’s Speech clean (~300h), and synthetic speech from Kokoro (~25h).
- Evaluation: Six diverse test sets including LibriSpeech (test-clean/other), TED-LIUM 3, Earnings-22, and a held-out synthetic set.
- Benchmarks: The model was also validated on the eight English sets of the Open ASR Leaderboard.
Key Insights and Ideas
- The Power of Negative Latency: Earlybird is the only model under 1 GB whose median word is predicted correctly before it ends. On LibriSpeech test-clean, 69% of words are predicted correctly before the speaker finishes them.
- Confidence is Key: For a predictive model to be useful, it must know when it's guessing. The authors found that the model's raw token probabilities are remarkably well-calibrated (Expected Calibration Error of 3.7–5.5), meaning a "90% confidence" actually corresponds to being right ~90% of the time, even for unfinished words.
- Architecture vs. Recipe: Notably, these gains were achieved with no changes to the underlying architecture of the base Moonshine model, proving that the training objective (predictive prefix training) is the primary driver of earliness.
Key Results
- Speed & Size: At 16.8 MB, Earlybird runs roughly twice as fast as real-time on a single CPU thread.
- Latency: It predicts words ~360ms sooner than the standard Moonshine Tiny runtime and ~160-200ms sooner than Whisper base.
- Accuracy: Despite its focus on speed, it remains competitive. It beats Moonshine Tiny on all eight Open ASR Leaderboard sets and roughly ties with Whistle, a similarly sized model.
The trade-off for this earliness is revisions. Because Earlybird commits early, it "takes back" words more often than models that wait (1.29 revisions per word on clean speech vs. 0.16 for Whistle). However, these revisions are often corrected within milliseconds as more audio arrives.
Limitations and Future Work
- Background Noise: While Earlybird excels on clean speech, its accuracy degrades more than larger models (like Whisper) in noisy environments or "chatter."
- Revisions: The high revision rate on spontaneous or noisy speech (2–6 per word) might be jarring for a UI, though it is manageable for a back-end voice agent.
- Language Support: The current release is English-only.
- Guessing Accuracy: While the model guesses the next words, only about 12-16% of these guesses are correct at a high confidence level before the word actually begins. Improving the "language model" aspect of the recognizer could enhance these future-lookahead guesses.
Summary updated