A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale

Cal Peyser, Michael Picheny, Kyunghyun Cho, Rohit Prabhavalkar, W. Ronny Huang, Tara N. Sainath

This paper, a collaboration between NYU and Google, investigates how semi-supervised learning (SSL) techniques—usually the heroes of low-resource settings—behave when applied to a massive, production-scale streaming Automatic Speech Recognition (ASR) system.

TL;DR

When you already have 200,000 hours of labeled data, standard semi-supervised learning doesn't improve your average Word Error Rate (WER). However, it significantly boosts performance on rare "tail" words, increases the richness of the generated hypothesis lattice, and reduces the computational load on the decoder during inference.

What Problem Does This Paper Address?

In the world of ASR, Semi-Supervised Learning (SSL) is a well-known trick for training models when labeled data is scarce. By using unlabeled audio or text, models can learn better acoustic and linguistic representations.

However, most SSL research focuses on "low-resource" scenarios or uses massive, non-streaming architectures that wouldn't fit on a mobile phone. The authors ask: What happens when we apply these techniques to a state-of-the-art, 160M-parameter streaming Conformer already trained on a massive supervised dataset? In a production environment, we care about more than just average accuracy; we care about rare word recognition, CPU budgets, and "lattice density" (the variety of alternative transcriptions the model considers).

Methods

The researchers compared three primary SSL methods using a cascading Conformer architecture (which includes both a low-latency "causal" encoder and a slightly delayed "non-causal" encoder):

  1. TTS Augmentation (Method B): Using a Text-to-Speech system to turn unlabeled text into synthetic audio features, which are then treated as standard supervised data.
  2. JOIST (Method A): A "joint" training method where the model performs a Masked Language Modeling (MLM) task on text. To make text "look" like audio to the encoder, they use a learned text frontend and upsample the representations.
  3. BEST-RQ (Method C): An unsupervised audio injection method. It masks segments of unlabeled audio and tasks the encoder with predicting quantized targets derived from a random, fixed codebook.

Crucially, they used joint training (optimizing supervised and unsupervised tasks simultaneously) rather than the more common pre-train/fine-tune cycle, which they found leads to "forgetting" on very large datasets.

Datasets

  • Supervised ($S$): ~4 million utterances (~200k hours) of labeled speech.
  • Unlabeled Audio ($U^S$): ~600 million utterances.
  • Unlabeled Text ($U^T$): ~230 billion examples.
  • Evaluation Sets:
    • VS/Noisy: Standard and noisy voice search traffic.
  • RPN (Rare Proper Nouns): Utterances with names/nouns appearing $<5$ times in the training set.
  • R-LM / C-LM: Sets targeting rare vs. common words found in the unlabeled text corpus to measure how well text injection works.

Key Insights and Ideas

  • The Saturation Point: For a 160M parameter model, 200k hours of labeled data is enough to hit a performance ceiling for general WER. SSL doesn't move the needle on "easy" or common phrases.
  • JOIST as a Decoder Shortcut: JOIST (text injection) significantly reduced the number of states the decoder had to explore during beam search. By strengthening the encoder's linguistic knowledge, the model becomes more "certain," making inference more computationally efficient.
  • Lattice Richness: All SSL methods increased "lattice density." In production, ASR isn't just about the top-1 result; it produces a "lattice" of possibilities for downstream Natural Language Understanding (NLU) tasks. A denser lattice means the correct answer is more likely to be "hidden" in the options for the NLU to find.
  • Complementary Strengths: JOIST is great for rare words but can struggle with noisy audio. Combining it with BEST-RQ (audio injection) helps recover that lost robustness while keeping the rare-word gains.

Key Results

  • General WER: 0% improvement on the standard Voice Search (VS) test set across all methods.
  • Tail-Word Performance: JOIST (Method A) achieved a 11.1% relative reduction in decoding states and a 4.7% to 5.0% relative WER improvement on rare proper nouns and rare-LM sets.
  • Efficiency: JOIST reduced the average number of decoding states expanded by 7.2% to 9.9%, directly translating to lower CPU usage.
  • Lattice Density: All methods improved lattice density by ~12-15% on the Voice Search set, providing a much richer signal for downstream tasks.

Limitations and Future Work

The authors note that the lack of improvement in general WER might be due to model capacity. A 160M parameter model might simply be "full" after 200k hours of supervised training. They suggest that larger models might see more general gains from SSL.

Future work could involve investigating how this increased lattice density specifically benefits downstream tasks like intent classification or named entity recognition in a production pipeline.

Summary updated

Abstract

Unpaired text and audio injection have emerged as dominant methods for improving ASR performance in the absence of a large labeled corpus. However, little guidance exists on deploying these methods to improve production ASR systems that are trained on very large supervised corpora and with realistic requirements like a constrained model size and CPU budget, streaming capability, and a rich lattice for rescoring and for downstream NLU tasks. In this work, we compare three state-of-the-art semi-supervised methods encompassing both unpaired text and audio as well as several of their combinations in a controlled setting using joint training. We find that in our setting these methods offer many improvements beyond raw WER, including substantial gains in tail-word WER, decoder computation during inference, and lattice density.
BibTeX
@inproceedings{peyser2023comparison,
  author = {Cal Peyser and Michael Picheny and Kyunghyun Cho and Rohit Prabhavalkar and W. Ronny Huang and Tara N. Sainath},
  title = {{A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=EzkaYU83kQ},
}

Loading related content...

Figure