wav2vec: Unsupervised Pre-training for Speech Recognition

Steffen Schneider, Alexei Baevski, Ronan Collobert, Michael Auli

Introduction

State-of-the-art Automatic Speech Recognition (ASR) systems are data-hungry, typically requiring thousands of hours of precisely transcribed audio to perform well. However, transcribing audio is a slow, expensive, and labor-intensive process. In contrast, unlabeled audio is abundant and easy to collect.

While unsupervised pre-training has revolutionized Computer Vision (via ImageNet) and Natural Language Processing (via models like BERT), its application to speech recognition has historically been limited. The authors of wav2vec sought to bridge this gap, creating a general-purpose audio representation that makes ASR more data-efficient.

Methods

The authors propose a fully convolutional architecture that takes raw audio as input and processes it through two main components:

  1. The Encoder Network ($f$): A five-layer CNN that maps raw audio samples into a low-frequency latent feature representation ($z$). Each latent vector represents about 30ms of audio, with a stride of 10ms.
  2. The Context Network ($g$): A nine-layer CNN that mixes multiple latent representations from the encoder to produce a single contextualized tensor ($c$). This network has a total receptive field of about 210ms.
wav2vec architecture overview showing the encoder network $f$, context network $g$, and contrastive loss $\mathcal{L}$.

Contrastive Loss

To train the model without labels, the authors use a Noise Contrastive Estimation (NCE) task. The model is asked to distinguish a "true" future audio sample from a set of "distractor" samples (negatives) drawn from the same sequence. By forcing the model to predict future segments based on current context, it learns to capture the underlying structure of speech.

Datasets and Evaluation

The researchers evaluated wav2vec using several standard benchmarks:

  • TIMIT: Used for phoneme recognition (approx. 3 hours of labeled data).
  • Wall Street Journal (WSJ): Used for the primary ASR evaluation (approx. 81 hours of labeled data).
  • Librispeech: A large corpus of 960 hours of clean and noisy speech, used primarily for large-scale unsupervised pre-training.

Key Results and Insights

One of the most striking findings was the model's data efficiency in low-resource settings. As shown in the performance benchmarks, wav2vec significantly reduces error rates when labeled data is scarce.

Word Error Rate (WER) on WSJ benchmarks as a function of labeled training data, highlighting the efficiency of wav2vec pre-training.
  • State-of-the-Art Performance: On the WSJ nov92 test set, wav2vec achieved a 2.43% WER. This outperformed Deep Speech 2 (3.1% WER), despite Deep Speech 2 being trained on 12,000 hours of labeled data—roughly 150 times more than the labeled data used for wav2vec.
  • Data Efficiency: In a simulated "low-resource" environment with only 8 hours of labeled data, wav2vec reduced the Word Error Rate (WER) by up to 40% compared to a baseline model.
  • Transferability: Pre-training on the larger Librispeech dataset improved performance even when the downstream task was on a different dataset (WSJ), proving the representations are highly transferable.
  • Raw Audio vs. Hand-crafted Features: wav2vec learns directly from the raw waveform. The results show that these learned embeddings can outperform traditional log-mel filterbanks, which have been the industry standard for decades.

Limitations and Future Work

While increasing the context helped, predicting more than 12 steps (120ms) into the future did not yield further improvements. Additionally, the authors noted that fine-tuning the encoder/embedding network during the supervised phase did not meaningfully improve performance but did increase training time.

Future directions include exploring even more diverse architectures and testing the approach on a wider variety of ASR tasks and languages. The authors also aim to investigate how these representations can be used in entirely end-to-end systems without traditional lexicons or language models.

Summary updated

BibTeX
@inproceedings{schneider2019wav,
  author = {Steffen Schneider and Alexei Baevski and Ronan Collobert and Michael Auli},
  title = {{wav2vec: Unsupervised Pre-training for Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=-AdtyCiP_ze},
}
Figure