Abstract
We present results from Alexa speech teams on semi-supervised learning (SSL) of acoustic models (AM) with experiments spanning over 3000 hours of GPU time, making our study one of the largest of its kind. We discuss SSL for AMs in a small footprint setting, showing that a smaller capacity model trained with 1 million hours of unsupervised data can outperform a baseline supervised system by 14.3% word error rate reduction (WERR). When increasing the supervised data to seven-fold, our gains diminish to 7.1% WERR; to improve SSL efficiency at larger supervised data regimes, we employ a step-wise distillation into a smaller model, obtaining a WERR of 14.4%. We then switch to SSL using larger student models in low data regimes; while learning efficiency with unsupervised data is higher, student models may outperform teacher models in such a setting. We develop a theoretical sketch to explain this behavior.
Introduction
The demand for on-device Automatic Speech Recognition (ASR) is growing, driven by the need for lower latency and improved user privacy. However, deploying high-performance acoustic models (AMs) on edge devices is challenging due to strict memory and computational constraints. Semi-Supervised Learning (SSL) offers a pathway to leverage vast amounts of unlabeled data by using a high-capacity "teacher" model to generate labels for a smaller "student" model.
Despite its promise, SSL faces two primary obstacles: learning saturation, where the benefits of unlabeled data diminish as supervised data grows, and the capacity gap, where a small student model struggles to learn directly from a much larger teacher. This paper, a collaboration between Amazon’s Alexa team and the University of Edinburgh, investigates these challenges using a massive 1-million-hour unsupervised dataset.
Methodology: Bridging the Capacity Gap
The researchers utilized a hybrid HMM-LSTM architecture, focusing on Student-Teacher Distillation. In this framework, a large Bidirectional LSTM (BLSTM) teacher provides "soft labels" to train a smaller Unidirectional LSTM (Uni-LSTM) student.
To overcome the capacity gap, the team introduced Step-wise Knowledge Distillation (TAKD). Instead of a direct transfer from a 78M parameter teacher to an 8M parameter student, they employed an intermediate 28M parameter "Teacher Assistant" (TA). This multi-stage process ensures that the final on-device model receives better-calibrated labels. The training process was scaled to an unprecedented 1 million hours of unsupervised data, requiring over 3,000 GPU hours.
Experimental Results
The experiments utilized supervised datasets ranging from 100 to 52,000 hours and a 1-million-hour unsupervised set. Performance was measured using Relative Word Error Rate Reduction (WERR).
The Efficacy of TAKD
In high-data regimes (50k hours supervised), direct Knowledge Distillation (KD) showed signs of saturation. However, the TAKD approach significantly improved efficiency. As shown in the results table below, TAKD achieved a 14.4% WERR at the cross-entropy (CE) stage, compared to only 9.2% for direct KD.
Students Outperforming Teachers
One of the most intriguing findings occurred in low-resource settings (e.g., 100 hours of labeled data). In these cases, the student model actually outperformed its teacher. The authors hypothesize that the student’s restricted capacity acts as a form of regularization, allowing it to generalize more effectively from the massive unsupervised data without overfitting to the small labeled set.
Scaling and Saturation
The study confirmed that while scaling to 1 million hours provides substantial gains, the rate of improvement slows down. For instance, in the 52k-hour supervised regime, most gains were realized by 540,000 hours of unsupervised data. This underscores the importance of architectural innovations like TAKD over simple data scaling.
Conclusion
This work demonstrates that large-scale SSL can dramatically improve on-device ASR performance, with WERR gains of up to 25.7% when combined with discriminative training (sMBR). The introduction of Teacher Assistant Knowledge Distillation (TAKD) provides a robust method for transferring knowledge across significant capacity gaps. Future research will focus on extending these findings to End-to-End (E2E) architectures and exploring sequence-level distillation techniques.
Summary updated