Exploiting Large-Scale Teacher-Student Training for On-Device Acoustic Models

Jing Liu, Rupak Vignesh Swaminathan, Sree Hari Krishnan Parthasarathi, Chunchuan Lyu, Athanasios Mouchtaris, Siegfried Kunzmann

Abstract

We present results from Alexa speech teams on semi-supervised learning (SSL) of acoustic models (AM) with experiments spanning over 3000 h of GPU time, making our study one of the largest of its kind. We discuss SSL for AMs in a small footprint setting, showing that a smaller capacity model trained with 1 million hours of unsupervised data can outperform a baseline supervised system by 14.3% word error rate reduction (WERR). When increasing the supervised data to seven-fold, our gains diminish to 7.1% WERR; to improve SSL efficiency at larger supervised data regimes, we employ a step-wise distillation into a smaller model, obtaining a WERR of 14.4%. We then switch to SSL using larger student models in low data regimes; while learning efficiency with unsupervised data is higher, student models may outperform teacher models in such a setting. We develop a theoretical sketch to explain this behavior.
BibTeX
@inproceedings{liu2021exploiting,
  author = {Jing Liu and Rupak Vignesh Swaminathan and Sree Hari Krishnan Parthasarathi and Chunchuan Lyu and Athanasios Mouchtaris and Siegfried Kunzmann},
  title = {{Exploiting Large-Scale Teacher-Student Training for On-Device Acoustic Models}},
  booktitle = {TDS},
  year = {2021},
  doi = {10.1007/978-3-030-83527-9\_35},
}

Loading related content...

Figure