Earlybird: Predictive Streaming Speech Recognition in 29M Parameters

Sahil MahendrakararXiv 2026 · arXiv

TL;DR

Earlybird is a compact, 29M-parameter streaming speech recognition model that achieves "negative latency" by predicting words correctly before the speaker even finishes saying them. By training on randomly truncated audio and learning to guess the next two words, it hits a median latency of -90ms on clean speech—beating much larger models while fitting into a tiny 16.8 MB (4-bit) footprint.

What Problem Does This Paper Address?

In the world of voice assistants, every millisecond of delay matters. Traditional streaming Automatic Speech Recognition (ASR) models typically wait until a word is finished—and often a few hundred milliseconds of "look-ahead" audio—before committing to a transcript. This creates a lag that prevents voice agents from planning replies or starting searches while the user is still talking.

Humans don't work this way; we often recognize a word from its first few phonemes and the surrounding context. Earlybird aims to give machines this same predictive capability, allowing them to output text early with a reliable confidence score so downstream systems can act faster.

Methods

Earlybird is built on the Moonshine streaming Tiny architecture (an encoder-decoder model) but introduces a novel training recipe:

  1. Training on Cut-off Audio: Instead of training on full utterances, the authors cut audio at random 80ms intervals. The model must predict the full word currently being spoken (even if it's only heard the start) plus a two-word "guess" of what comes next.
  2. Monotonic Targets: To ensure stability, the target at any time $t+1$ always extends the target at time $t$. This teaches the model not to output words it might have to "take back" later.
  3. Selective Distillation: It uses a larger "teacher" model (Moonshine Medium) to provide a distillation loss, but only at positions where the teacher is correct. This helps Earlybird learn to spread its probability over likely candidates when uncertain.
  4. Padding Frames: At each step, the encoder sees three frames of silence after the current audio. This helps the model "finish" a word as if the speaker had just stopped.
  5. Quantization: The model undergoes 4-bit quantization-aware training, shrinking it to just 16.8 MB without significant accuracy loss.

Datasets

  • Training: A mix of LibriSpeech (960h), People’s Speech clean (~300h), and synthetic speech from Kokoro (~25h).
  • Evaluation: Six diverse test sets including LibriSpeech (test-clean/other), TED-LIUM 3, Earnings-22, and a held-out synthetic set.
  • Benchmarks: The model was also validated on the eight English sets of the Open ASR Leaderboard.

Key Insights and Ideas

  • The Power of Negative Latency: Earlybird is the only model under 1 GB whose median word is predicted correctly before it ends. On LibriSpeech test-clean, 69% of words are predicted correctly before the speaker finishes them.
  • Confidence is Key: For a predictive model to be useful, it must know when it's guessing. The authors found that the model's raw token probabilities are remarkably well-calibrated (Expected Calibration Error of 3.7–5.5), meaning a "90% confidence" actually corresponds to being right ~90% of the time, even for unfinished words.
  • Architecture vs. Recipe: Notably, these gains were achieved with no changes to the underlying architecture of the base Moonshine model, proving that the training objective (predictive prefix training) is the primary driver of earliness.
Figure 1: Word latency against accuracy; lower left is better. WER is the mean over the full Open ASR sets (Whistle: about 200 clips per set), with Earlybird at 4 bits. Latency is on the same 67 LibriSpeech clean utterances for every model except Parakeet, which was re-decoded every 240 ms on 150 other utterances (Earlybird: −80 ms on those). Hollow: published leaderboard WER.

Key Results

  • Speed & Size: At 16.8 MB, Earlybird runs roughly twice as fast as real-time on a single CPU thread.
  • Latency: It predicts words ~360ms sooner than the standard Moonshine Tiny runtime and ~160-200ms sooner than Whisper base.
  • Accuracy: Despite its focus on speed, it remains competitive. It beats Moonshine Tiny on all eight Open ASR Leaderboard sets and roughly ties with Whistle, a similarly sized model.
Figure 2: Size against accuracy (left) and against word latency (right); lower left is better. Full-precision models are shown at their 16-bit size, Nemotron (streaming, 80 ms chunks) at its int8 size. Hollow: published leaderboard WER; filled: our runs. Latency as in Figure 1.

The trade-off for this earliness is revisions. Because Earlybird commits early, it "takes back" words more often than models that wait (1.29 revisions per word on clean speech vs. 0.16 for Whistle). However, these revisions are often corrected within milliseconds as more audio arrives.

Table 1: Median word latency and WER across test sets. Negative numbers indicate prediction before the word ends.

Limitations and Future Work

  • Background Noise: While Earlybird excels on clean speech, its accuracy degrades more than larger models (like Whisper) in noisy environments or "chatter."
  • Revisions: The high revision rate on spontaneous or noisy speech (2–6 per word) might be jarring for a UI, though it is manageable for a back-end voice agent.
  • Language Support: The current release is English-only.
  • Guessing Accuracy: While the model guesses the next words, only about 12-16% of these guesses are correct at a high confidence level before the word actually begins. Improving the "language model" aspect of the recognizer could enhance these future-lookahead guesses.

Summary updated

Abstract

Streaming speech recognizers output a word only after the speaker has finished it, typically 90-400 ms later. For a voice agent every millisecond counts: the sooner the words are outputted, the sooner the agent can start a search or plan its reply. We present Earlybird, a 29M-parameter streaming recognizer (16.8 MB at 4 bits). Earlybird learns from audio cut off at random points, including partway through a word, and is asked to predict every word that is due by then in full, plus a guess of the next two words. Its raw token probabilities give a confidence for every word including the one still being spoken. On clean audiobook speech (LibriSpeech test-clean), the median word is predicted correctly 90 ms before the speaker finishes it, while released streaming models get it 90 to 400 ms after it ends. Earlybird is the earliest on all six test sets we time, covering audiobooks, TED talks, public speech, earnings calls, and synthetic speech. On the same audiobooks, 69% of words are predicted correctly before they end, against 3-25% for the other models. For words still being spoken, confidence is calibrated, with an expected calibration error (ECE) of 3.7 to 5.5. On the eight English Open ASR Leaderboard sets, it is more accurate than Moonshine Tiny on every set and roughly ties Whistle, a model of the same size.
BibTeX
@misc{mahendrakar2026earlybird,
  author = {Sahil Mahendrakar},
  title = {{Earlybird: Predictive Streaming Speech Recognition in 29M Parameters}},
  year = {2026},
  eprint = {2611.14421},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14421},
}
Figure