Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?
Lists
Only you see this
Abstract
Compared to vision and language applications, self-supervised pre-training approaches for ASR are challenged by three unique problems: (1) There are multiple sound units in each input utterance, (2) With audio-only pre-training, there is no lexicon of sound units, and (3) Sound units have variable lengths with no explicit segmentation. In this paper, we propose the Hidden-Unit BERT (HUBERT) model which utilizes a cheap k-means clustering step to provide aligned target labels for pre-training of a BERT model. A key ingredient of our approach is applying the predictive loss over the masked regions only. This allows the pre-training stage to benefit from the consistency of the unsupervised teacher rather that its intrinsic quality. Starting with a simple k-means teacher of 100 cluster, and using two iterations of clustering, the HUBERT model matches the state-of-the-art wav2vec 2.0 performance on the ultra low-resource Libri-light 10h, 1h, 10min supervised subsets.
BibTeX
@inproceedings{hsu2021hubert,
author = {Wei-Ning Hsu and Yao-Hung Hubert Tsai and Benjamin Bolte and Ruslan Salakhutdinov and Abdelrahman Mohamed},
title = {{Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2021},
url = {https://openreview.net/forum?id=rcWO1Acjfc},
}Loading related content...