Learning Robust and Multilingual Speech Representations

Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, Aäron van den Oord

Abstract

Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. However, most research has been focused on evaluating the representations in terms of their ability to improve the performance of speech recognition systems on read English (e.g. Wall Street Journal and LibriSpeech). This evaluation methodology overlooks two important desiderata that speech representations should have: robustness to domain shifts and transferability to other languages. In this paper we learn representations from up to 8000 hours of diverse and noisy speech data and evaluate the representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. We find that our representations confer significant robustness advantages to the resulting recognition systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets and the features likewise provide improvements in 25 phonetically diverse languages including tonal languages and low-resource languages.

Introduction

Automatic Speech Recognition (ASR) has reached impressive milestones, yet most state-of-the-art models are specialized for "read English" datasets like LibriSpeech. These models often struggle with "domain shifts"—variations in recording environments, background noise, or spontaneous speaking styles—and rarely generalize to low-resource languages. This research, conducted by teams at DeepMind and the University of Oxford, explores the creation of universal, robust speech representations by scaling unsupervised learning to 8,000 hours of diverse, uncurated audio.

Methodology: Scaling Contrastive Predictive Coding

The researchers utilized Contrastive Predictive Coding (CPC), a technique where a model learns by predicting future segments of an audio signal in a latent space. By distinguishing correct future segments from "negative" noise samples using a contrastive loss (InfoNCE), the model captures essential phonetic structures without needing explicit labels.

Key Technical Innovations

  • Bidirectional CPC: Unlike the original unidirectional CPC, this model employs two autoregressive networks—one forward-looking and one backward-looking—to capture a more comprehensive context.
  • Dense Residual Connections: The autoregressive component uses a 13-layer causal convolution architecture with dense skip connections, facilitating better gradient flow and complex feature extraction.
  • Unprecedented Scale: The pre-training dataset was expanded from the standard 1,000 hours of clean audio to 8,000 hours of diverse, noisy, and multilingual data (CPC-8k).
The two-stage pipeline: unsupervised pre-training with CPC followed by semi-supervised speech recognition using a CTC loss.

Robustness and Multilingual Transfer

A central finding of the study is that diversity trumps purity. Pre-training on 8,000 hours of noisy, uncurated data (CPC-8k) yielded features that outperformed those trained on 1,000 hours of clean English (CPC-LibriSpeech), even when evaluated on clean English benchmarks. The diverse data teaches the model to ignore irrelevant factors like reverb and background noise.

Domain Robustness

The robustness of these features is most evident in domain-shift scenarios. When an ASR model trained on the clean Wall Street Journal (WSJ) dataset was tested on the noisy Switchboard telephone dataset, standard Log-Filterbank features resulted in a 96.44% Word Error Rate (WER). In contrast, using CPC-8k features reduced the WER to 69.13%, a massive leap in cross-domain usability.

Key Results across Languages

The model successfully learned a "universal" phonetic space that generalizes across 25 phonetically diverse languages. This is particularly impactful for low-resource languages where labeled data is scarce.

  • Sundanese (su): The most dramatic improvement was seen in Sundanese, where switching to CPC-8k features reduced the WER from 27.85% to 11.49%.
  • Low-Resource African Languages: Significant gains were observed in languages like Amharic, Swahili, and Wolof. For Amharic, the WER dropped from near-total failure with English-only pre-training to competitive levels with the multilingual 8k features.
  • Sample Efficiency: The representations are highly efficient; using just 10% of labeled LibriSpeech data with CPC-8k features achieved performance comparable to using 100% of the data with standard features.
Relative improvement in Word Error Rate (WER) across 25 phonetically diverse languages when using CPC-8k features compared to standard baselines.

Conclusion

By scaling unsupervised pre-training to massive, diverse datasets, this work demonstrates that it is possible to learn speech representations that are both robust to environmental noise and transferable across a wide linguistic spectrum. These "universal" features provide a powerful foundation for building ASR systems that work reliably in the real world and for all the world's languages.

Summary updated

Abstract

Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. However, most research has been focused on evaluating the representations in terms of their ability to improve the performance of speech recognition systems on read English (e.g. Wall Street Journal and LibriSpeech). This evaluation methodology overlooks two important desiderata that speech representations should have: robustness to domain shifts and transferability to other languages. In this paper we learn representations from up to 8000 hours of diverse and noisy speech data and evaluate the representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. We find that our representations confer significant robustness advantages to the resulting recognition systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets and the features likewise provide improvements in 25 phonetically diverse languages including tonal languages and low-resource languages.
BibTeX
@inproceedings{kawakami2020learning,
  author = {Kazuya Kawakami and Luyu Wang and Chris Dyer and Phil Blunsom and Aäron van den Oord},
  title = {{Learning Robust and Multilingual Speech Representations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=ZpVQvpxSpPq},
}
Figure