Introduction
In Automatic Speech Recognition (ASR), accuracy is paramount, but latency is the ultimate constraint for real-time applications. State-of-the-art systems often employ a two-pass architecture: a fast first pass generates an n-best list of hypotheses, and a second-pass rescorer re-ranks them to select the most accurate transcription. While large BERT-based models have significantly improved rescoring accuracy—especially when trained with discriminative objectives like Minimum Word Error Rate (MWER)—their size often makes them too slow for streaming voice assistants.
This research explores effective strategies to distill the performance of a large, high-capacity teacher model (170M parameters) into a compact, production-ready student model (5M parameters) without sacrificing the gains achieved through discriminative training.
The Challenge: Latency vs. Accuracy
Modern rescorers benefit from MWER loss, which directly penalizes models based on actual word errors in the top-ranked hypothesis. However, these gains are often model-size dependent; shrinking a 170M parameter model to 5M parameters typically results in a significant loss of the MWER-driven accuracy boost. To bridge this gap, the authors propose moving beyond standard knowledge distillation—which typically focuses on general language understanding—toward Discriminative Distillation.
Proposed Method: Discriminative Distillation
The core innovation lies in distilling the teacher's knowledge after it has been fine-tuned with MWER. This ensures the student learns the specific logic required to discriminate between similar hypotheses rather than just learning general syntax.
The researchers evaluated several loss functions for the student:
- Cross-Entropy over n-best posteriors ($L_{post}$): Matching the teacher's probability distribution.
- Oracle-based posterior correction ($L_{pOracle}$): Adjusting the teacher's signal toward the "Oracle" (best possible) hypothesis.
- Mean Squared Error (MSE): Directly matching the raw scores assigned by the teacher.
- Hybrid Losses: Combining these approaches with the original MWER objective.
Experimental Results
The strategies were tested on the public LibriSpeech benchmark and a massive internal dataset comprising 95,000 hours of voice-assistant audio across domains like Music, Shopping, and Navigation.
Closing the Performance Gap
The results demonstrate that task-specific distillation is highly effective. In several domains, the distilled 5M model achieved performance nearly identical to the 170M teacher. Specifically, the proposed techniques reduced the performance gap between the student and teacher by 62% to 100%.
Key findings include:
- Relative WER Improvement: The distilled student showed up to a 7% relative improvement over a student trained with standard MWER.
- Latency Gains: By utilizing the 5M parameter model, the system achieves a 1350% relative improvement in latency compared to the 170M teacher, making it viable for real-time streaming.
- Domain-Specific Success: The "Music" domain saw particularly strong gains, likely due to the model's improved ability to handle the unique acoustic and linguistic patterns of music-related queries.
Conclusion
This work demonstrates that the sophisticated knowledge captured by large discriminative rescorers can be successfully compressed into much smaller models. By focusing on discriminative distillation rather than general language modeling, developers can deploy high-accuracy ASR components in latency-sensitive environments. Future research may explore distilling from even larger teachers or further reducing student model size for edge device deployment.
Summary updated