Scaling Speech Models with Iterative Pseudo-Labeling

187 concepts143 papers2 proposals

Building a state-of-the-art automatic speech recognition (ASR) system traditionally demands thousands of hours of meticulously transcribed audio. However, manual transcription is expensive, time-consuming, and scales poorly. In contrast, untranscribed audio data—from customer interactions, radio broadcasts, or web videos—is virtually infinite.

As Karthik Visweswariah et al. demonstrated in Task adaptation of acoustic and language models based on large quantities of data, leveraging this raw operational data can yield substantial performance gains even without human intervention. The core mechanism that unlocks this data is iterative pseudo-labeling (or self-training). Instead of waiting for human annotators, an existing "teacher" model processes the unlabeled audio to generate transcriptions. These machine-generated transcripts serve as "pseudo-labels" to train a "student" model.

As Chunxi Liu et al. note in Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions, this pipeline allows high-capacity models to learn from datasets orders of magnitude larger than anything manually transcribed. In fact, Weiyi Zheng et al. leveraged this exact paradigm in Scaling ASR Improves Zero and Few Shot Learning to scale English ASR models to a massive 10 billion parameters trained on 4.5 million hours of audio.

Why It Works: The Intuition Behind Self-Training

A natural skepticism arises: if the teacher model makes mistakes, won't the student model simply memorize and amplify those errors? Why does training a model on its own imperfect predictions actually improve performance?

The answer lies in regularization, data diversity, and acoustic alignment. When a student model is trained on pseudo-labels, it isn't just memorizing text strings; it is learning to map a highly diverse set of acoustic environments, accents, and background noises to a compressed phonetic or textual space. The massive volume of unlabeled data exposes the model to real-world acoustic variations that are absent from clean, supervised datasets.

To prevent the student from inheriting and compounding the teacher's errors, modern frameworks introduce deliberate asymmetries and filtering mechanisms:

1. Confidence Filtering and Re-calibration

Not all pseudo-labels are created equal. Early work by Dong Yu et al. in Semi-supervised GMM and DNN acoustic model training with multi-system combination and confidence re-calibration showed that filtering out low-confidence predictions prevents noisy labels from corrupting the model. Similarly, Anton Ragni et al. in Data augmentation for low resource languages utilized confidence thresholds to ensure only the most reliable automated transcriptions update the model parameters, acting as an automated quality control gate.

2. Asymmetric Data Augmentation

In a teacher-student framework, we can make the student's job harder than the teacher's. As shown by Yiming Li et al. in A Hybrid System of Sound Event Detection Transformer and Frame-Wise Model for DCASE 2022 Task 4, the teacher model processes a clean version of the audio to generate a stable pseudo-label. The student model is then forced to predict that same label from a heavily augmented, noisy version of the same clip. This forces the student to learn robust, noise-invariant features rather than tracking superficial acoustic patterns.

3. Consistency Regularization

Instead of relying on a single hard prediction, models can enforce consistency across multiple views of the data. In Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation, Zhehuai Chen et al. introduced Sequential MixMatch, which averages and sharpens predictions from different noise applications on untranscribed speech. This creates highly stable acoustic representations that smooth out the model's decision boundaries.

┌─────────────────┐     Clean Audio     ┌─────────────────┐
│ Unlabeled Audio ├────────────────────►│  Teacher Model  │
└────────┬────────┘                     └────────┬────────┘
         │                                       │ Generates
         │ Distorts                              │ Pseudo-Labels
         ▼                                       ▼
┌─────────────────┐                     ┌─────────────────┐
│ Augmented Audio ├────────────────────►│  Student Model  │
└─────────────────┘   Computes Loss     └─────────────────┘

Bridging the Gap in Low-Resource and Out-of-Domain Scenarios

The true power of pseudo-labeling emerges when adapting models to new domains, dialects, or low-resource languages where human labels are completely unavailable.

When an ASR model encounters an unseen accent or dialect, its performance typically degrades. However, by using the model to transcribe unlabeled target-domain audio, it can bootstrap its own adaptation. Cong-Thanh Do et al. demonstrated in Domain Adaptive Self-supervised Training of Automatic Speech Recognition that even when the initial teacher model is imperfect, the resulting pseudo-labels provide enough acoustic signal to capture domain-specific features.

This strategy is highly effective for building equitable speech technology. For instance, Shefali Garg in Improving Speech Recognition for African American English With Audio Classification used a teacher model to generate pseudo-labels for unlabeled audio segments identified via dialect classification, drastically reducing performance disparities without requiring manual transcription.

In extreme low-resource scenarios, the seed model can even be trained on entirely different languages. Anton Ragni et al. explored this in Language independent and unsupervised acoustic models for speech recognition and keyword spotting, where a multilingual base model generated initial hypotheses for a zero-resource target language, bootstrapping a language-specific model entirely through confidence-filtered unsupervised learning.

Expanding Beyond Standard Text Transcription

The intuition of iterative self-training extends far beyond mapping audio to a standard stream of words. The same core loop can be applied to capture complex structural, behavioral, and acoustic patterns:

By treating a model's own predictions not as absolute truths, but as noisy, structured guidance, iterative pseudo-labeling transforms the vast ocean of raw, untranscribed audio into a highly effective training ground.

Key Foundational Work

Exploiting large quantities of spontaneous speech for unsupervised training of acoustic models (2005) — contributed 2 concept elements to this direction.
Task adaptation of acoustic and language models based on large quantities of data (2004) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_leveraging_untranscribed_data_66039143,
  title = {leveraging untranscribed data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/leveraging-untranscribed-data/rdr_660391435dd2467c937095fad34665f3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_leveraging_untranscribed_data_66039143_v1,
  title = {leveraging untranscribed data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/leveraging-untranscribed-data/rdr_660391435dd2467c937095fad34665f3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake