Learning Robust and Multilingual Speech Representations

Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, Aäron van den Oord

Abstract: Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, Aaron van den Oord. Findings of the Association for Computational Linguistics: EMNLP 2020.

2020.

Introduction

Most modern Automatic Speech Recognition (ASR) systems rely on hand-engineered features like MFCCs or representations learned from "clean" datasets like LibriSpeech. While effective in controlled environments, these systems often struggle with "domain shifts"—the transition from clear, read speech to noisy, conversational, or multilingual environments. This research addresses the gap in robustness and multilingual transferability of unsupervised speech models, arguing that a truly effective representation should be invariant to task-irrelevant factors like background noise while remaining highly transferable across languages.

Methodology: Scaling Contrastive Predictive Coding

The core of this approach is Contrastive Predictive Coding (CPC). CPC learns representations by predicting future segments of an audio signal based on past context. It utilizes a contrastive loss (InfoNCE) to distinguish the correct future segment from a set of negative "noise" samples.

The authors introduce several critical enhancements to the standard CPC framework:

  • Bidirectional Context: Unlike original unidirectional models, this version employs two autoregressive networks to capture both forward and backward context.
  • Dense Residual Connections: Incorporating DenseNet-style skip connections within autoregressive layers improves gradient flow and model capacity.
  • Massive Scale: The model is pre-trained on an "8k" corpus—8,000 hours of diverse, uncurated audio including Audio Set, AVSpeech, and Common Voice.
  • Frozen Feature Evaluation: To rigorously test representation quality, the pre-trained encoder is frozen, and only a small downstream ASR model is trained on top.
Overview of the CPC pre-training and downstream speech recognition pipeline.

Datasets and Robustness

The "8k" pre-training corpus provides a messy, diverse foundation far removed from the "purity" of standard benchmarks. Evaluation spans English datasets like Wall Street Journal (read) and Switchboard (noisy telephone), alongside 25 diverse languages.

A key finding is that diversity trumps purity. Training on 8,000 hours of noisy data (CPC-8k) yielded better results even on clean English tasks than training solely on clean English data. This diversity forces the model to learn universal phonetic invariants that are robust to environmental noise.

Key Results: Multilingual Transfer and Sample Efficiency

The most dramatic improvements appear in difficult domains and low-resource languages. On the noisy Switchboard dataset, CPC-8k features achieved a 69.13% WER (without a language model), a massive leap from the 96.44% seen with standard LogFilterbank features.

In multilingual settings, the model demonstrates a "universal" phonetic space. For languages like Sundanese, switching to CPC-8k features dropped the WER from 27.85% to 11.49%. Furthermore, the features are highly sample-efficient: using CPC-8k features with only 10% of labeled data often outperformed standard features using 100% of labels.

Relative improvement in WER across 21 diverse languages using CPC-8k features.

Conclusion

This work demonstrates that unsupervised pre-training on large-scale, diverse audio data creates speech representations that are significantly more robust and transferable than traditional features. By capturing underlying phonetic structures shared across human speech, these models pave the way for high-performance speech technology in low-resource and noisy real-world environments.

Summary updated

BibTeX
@inproceedings{kawakami2020learning,
  author = {Kazuya Kawakami and Luyu Wang and Chris Dyer and Phil Blunsom and Aäron van den Oord},
  title = {{Learning Robust and Multilingual Speech Representations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=ZpVQvpxSpPq},
}
Figure