Abstract: Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, Aaron van den Oord. Findings of the Association for Computational Linguistics: EMNLP 2020.
2020.
Introduction
Most modern Automatic Speech Recognition (ASR) systems rely on hand-engineered features like MFCCs or representations learned from "clean" datasets like LibriSpeech. While effective in controlled environments, these systems often struggle with "domain shifts"—the transition from clear, read speech to noisy, conversational, or multilingual environments. This research addresses the gap in robustness and multilingual transferability of unsupervised speech models, arguing that a truly effective representation should be invariant to task-irrelevant factors like background noise while remaining highly transferable across languages.
Methodology: Scaling Contrastive Predictive Coding
The core of this approach is Contrastive Predictive Coding (CPC). CPC learns representations by predicting future segments of an audio signal based on past context. It utilizes a contrastive loss (InfoNCE) to distinguish the correct future segment from a set of negative "noise" samples.
The authors introduce several critical enhancements to the standard CPC framework:
- Bidirectional Context: Unlike original unidirectional models, this version employs two autoregressive networks to capture both forward and backward context.
- Dense Residual Connections: Incorporating DenseNet-style skip connections within autoregressive layers improves gradient flow and model capacity.
- Massive Scale: The model is pre-trained on an "8k" corpus—8,000 hours of diverse, uncurated audio including Audio Set, AVSpeech, and Common Voice.
- Frozen Feature Evaluation: To rigorously test representation quality, the pre-trained encoder is frozen, and only a small downstream ASR model is trained on top.
Datasets and Robustness
The "8k" pre-training corpus provides a messy, diverse foundation far removed from the "purity" of standard benchmarks. Evaluation spans English datasets like Wall Street Journal (read) and Switchboard (noisy telephone), alongside 25 diverse languages.
A key finding is that diversity trumps purity. Training on 8,000 hours of noisy data (CPC-8k) yielded better results even on clean English tasks than training solely on clean English data. This diversity forces the model to learn universal phonetic invariants that are robust to environmental noise.
Key Results: Multilingual Transfer and Sample Efficiency
The most dramatic improvements appear in difficult domains and low-resource languages. On the noisy Switchboard dataset, CPC-8k features achieved a 69.13% WER (without a language model), a massive leap from the 96.44% seen with standard LogFilterbank features.
In multilingual settings, the model demonstrates a "universal" phonetic space. For languages like Sundanese, switching to CPC-8k features dropped the WER from 27.85% to 11.49%. Furthermore, the features are highly sample-efficient: using CPC-8k features with only 10% of labeled data often outperformed standard features using 100% of labels.
Conclusion
This work demonstrates that unsupervised pre-training on large-scale, diverse audio data creates speech representations that are significantly more robust and transferable than traditional features. By capturing underlying phonetic structures shared across human speech, these models pave the way for high-performance speech technology in low-resource and noisy real-world environments.
Summary updated