Distillation Strategies for Discriminative Speech Recognition Rescoring

Prashanth Gurunath Shivakumar, Jari Kolehmainen, Yile Gu, Ankur Gandhe, Ariya Rastrow, Ivan Bulyko

Introduction

In Automatic Speech Recognition (ASR), accuracy is paramount, but latency is the ultimate constraint for real-time applications. State-of-the-art systems often employ a two-pass architecture: a fast first pass generates an n-best list of hypotheses, and a second-pass rescorer re-ranks them to select the most accurate transcription. While large BERT-based models have significantly improved rescoring accuracy—especially when trained with discriminative objectives like Minimum Word Error Rate (MWER)—their size often makes them too slow for streaming voice assistants.

This research explores effective strategies to distill the performance of a large, high-capacity teacher model (170M parameters) into a compact, production-ready student model (5M parameters) without sacrificing the gains achieved through discriminative training.

The Challenge: Latency vs. Accuracy

Modern rescorers benefit from MWER loss, which directly penalizes models based on actual word errors in the top-ranked hypothesis. However, these gains are often model-size dependent; shrinking a 170M parameter model to 5M parameters typically results in a significant loss of the MWER-driven accuracy boost. To bridge this gap, the authors propose moving beyond standard knowledge distillation—which typically focuses on general language understanding—toward Discriminative Distillation.

Proposed Method: Discriminative Distillation

The core innovation lies in distilling the teacher's knowledge after it has been fine-tuned with MWER. This ensures the student learns the specific logic required to discriminate between similar hypotheses rather than just learning general syntax.

The proposed distillation framework transfers knowledge from a large 170M RescoreBERT to a compact 5M student model.

The researchers evaluated several loss functions for the student:

  1. Cross-Entropy over n-best posteriors ($L_{post}$): Matching the teacher's probability distribution.
  2. Oracle-based posterior correction ($L_{pOracle}$): Adjusting the teacher's signal toward the "Oracle" (best possible) hypothesis.
  3. Mean Squared Error (MSE): Directly matching the raw scores assigned by the teacher.
  4. Hybrid Losses: Combining these approaches with the original MWER objective.

Experimental Results

The strategies were tested on the public LibriSpeech benchmark and a massive internal dataset comprising 95,000 hours of voice-assistant audio across domains like Music, Shopping, and Navigation.

Closing the Performance Gap

The results demonstrate that task-specific distillation is highly effective. In several domains, the distilled 5M model achieved performance nearly identical to the 170M teacher. Specifically, the proposed techniques reduced the performance gap between the student and teacher by 62% to 100%.

Detailed Word Error Rate Reduction (WERR) and gap closure percentages across various domains using different distillation loss functions.

Key findings include:

  • Relative WER Improvement: The distilled student showed up to a 7% relative improvement over a student trained with standard MWER.
  • Latency Gains: By utilizing the 5M parameter model, the system achieves a 1350% relative improvement in latency compared to the 170M teacher, making it viable for real-time streaming.
  • Domain-Specific Success: The "Music" domain saw particularly strong gains, likely due to the model's improved ability to handle the unique acoustic and linguistic patterns of music-related queries.

Conclusion

This work demonstrates that the sophisticated knowledge captured by large discriminative rescorers can be successfully compressed into much smaller models. By focusing on discriminative distillation rather than general language modeling, developers can deploy high-accuracy ASR components in latency-sensitive environments. Future research may explore distilling from even larger teachers or further reducing student model size for edge device deployment.

Summary updated

Abstract

Second-pass rescoring is employed in most state-of-the-art speech recognition systems. Recently, BERT based models have gained popularity for re-ranking the n-best hypothesis by exploiting the knowledge from masked language model pre-training. Further, fine-tuning with discriminative loss such as minimum word error rate (MWER) has shown to perform better than likelihood-based loss. Streaming applications with low latency requirements impose significant constraints on the size of the models, thereby limiting the word error rate (WER) performance gains. In this paper, we propose effective strategies for distilling from large models discriminatively trained with the MWER objective. We experiment on Librispeech and production scale internal dataset for voice-assistant. Our results demonstrate relative improvements of upto 7% WER over student models trained with MWER. We also show that the proposed distillation can reduce the WER gap between the student and the teacher by 62% upto 100%.
BibTeX
@inproceedings{shivakumar2023distillation,
  author = {Prashanth Gurunath Shivakumar and Jari Kolehmainen and Yile Gu and Ankur Gandhe and Ariya Rastrow and Ivan Bulyko},
  title = {{Distillation Strategies for Discriminative Speech Recognition Rescoring}},
  booktitle = {Interspeech},
  year = {2023},
  doi = {10.21437/interspeech.2023-1981},
}
Figure