A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale

Cal Peyser, Michael Picheny, Kyunghyun Cho, Rohit Prabhavalkar, W. Ronny Huang, Tara N. Sainath

Abstract

Unpaired text and audio injection have emerged as dominant methods for improving ASR performance in the absence of a large labeled corpus. However, little guidance exists on deploying these methods to improve production ASR systems that are trained on very large supervised corpora and with realistic requirements like a constrained model size and CPU budget, streaming capability, and a rich lattice for rescoring and for downstream NLU tasks. In this work, we compare three state-of-the-art semi-supervised methods encompassing both unpaired text and audio as well as several of their combinations in a controlled setting using joint training. We find that in our setting these methods offer many improvements beyond raw WER, including substantial gains in tail-word WER, decoder computation during inference, and lattice density.

Introduction

Semi-supervised learning (SSL) has traditionally been the primary tool for developing Automatic Speech Recognition (ASR) systems in low-resource environments. By leveraging unlabeled audio or text, models can learn robust acoustic and linguistic representations that supervised data alone cannot provide. However, the utility of SSL in high-resource, production-scale settings—where labeled data exceeds 200,000 hours—remains under-explored. This study, a collaboration between NYU and Google, evaluates SSL techniques within a 160M-parameter streaming Conformer system, focusing on metrics beyond average Word Error Rate (WER), such as rare word recognition, computational efficiency, and lattice richness.

Methodology

The study utilizes a cascading Conformer architecture, which incorporates a low-latency causal encoder and a delayed non-causal encoder. Three primary SSL methods were evaluated through joint training, where supervised and unsupervised losses are optimized simultaneously to prevent "catastrophic forgetting" common in pre-train/fine-tune cycles.

Cascading Conformer architecture with integrated SSL methods.
  1. JOIST (Method A): A joint training approach where the model performs Masked Language Modeling (MLM) on text. A learned text frontend and upsampling mechanism are used to align text representations with audio features.
  2. TTS Augmentation (Method B): A Text-to-Speech system generates synthetic audio features from unlabeled text, which are then used as additional supervised training data.
  3. BEST-RQ (Method C): An unsupervised audio injection method that masks segments of unlabeled audio and requires the encoder to predict quantized targets from a random, fixed codebook.

Experimental Setup

The models were trained on a supervised dataset of ~200,000 hours (~4 million utterances). Unlabeled data included ~600 million audio utterances and ~230 billion text examples. Evaluation was conducted across several dimensions, including standard Voice Search (VS), Noisy environments, and Rare Proper Nouns (RPN) to measure performance on the "tail" of the distribution.

Results and Discussion

The experimental results reveal a "saturation point" for general ASR performance. For a 160M parameter model, 200k hours of supervised data appears sufficient to reach a ceiling on common phrases, resulting in 0% relative WER improvement on the standard Voice Search (VS) set across all SSL configurations.

Relative WER changes across different test sets and SSL configurations.

However, significant gains were observed in other critical areas:

  • Tail-Word Performance: JOIST (Method A) achieved relative WER reductions of 4.7% on rare proper nouns and 5.0% on the rare-LM set, demonstrating its strength in linguistic injection.
  • Computational Efficiency: SSL methods, particularly JOIST, acted as a "decoder shortcut." By enhancing the encoder's linguistic certainty, JOIST reduced the number of decoding states expanded during beam search by 7.2% to 9.9%, directly lowering CPU requirements during inference.
  • Lattice Richness: All SSL methods improved lattice density by approximately 12-15% on the VS set. A denser lattice provides a more diverse set of hypotheses, increasing the likelihood that the correct transcription is available for downstream Natural Language Understanding (NLU) tasks.

Conclusion

While SSL may not improve average WER in high-resource streaming ASR, it provides substantial benefits for production systems. The improvements in tail-word accuracy, decoding efficiency, and lattice density demonstrate that SSL can optimize models for the "long tail" of language and for the computational constraints of real-world deployment. Future research will investigate whether larger model capacities can unlock further general WER gains from these massive unlabeled datasets.

Summary updated

Abstract

Unpaired text and audio injection have emerged as dominant methods for improving ASR performance in the absence of a large labeled corpus. However, little guidance exists on deploying these methods to improve production ASR systems that are trained on very large supervised corpora and with realistic requirements like a constrained model size and CPU budget, streaming capability, and a rich lattice for rescoring and for downstream NLU tasks. In this work, we compare three state-of-the-art semi-supervised methods encompassing both unpaired text and audio as well as several of their combinations in a controlled setting using joint training. We find that in our setting these methods offer many improvements beyond raw WER, including substantial gains in tail-word WER, decoder computation during inference, and lattice density.
BibTeX
@inproceedings{peyser2023comparison,
  author = {Cal Peyser and Michael Picheny and Kyunghyun Cho and Rohit Prabhavalkar and W. Ronny Huang and Tara N. Sainath},
  title = {{A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=EzkaYU83kQ},
}
Figure