TL;DR
This paper introduces a novel self-supervised learning approach for robust automatic speech recognition (ASR) by integrating a speech reconstruction task into the wav2vec 2.0 framework. This method significantly improves ASR performance in noisy conditions without requiring a separate speech enhancement front-end, achieving state-of-the-art results on real-world noisy speech with limited labeled data.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems are crucial for many applications, but their performance drastically degrades in the presence of background noise, a common issue in real-world scenarios. Traditionally, this has been tackled by using a cascaded approach: a speech enhancement module first cleans up the noisy audio, and then the enhanced speech is fed to the ASR system. However, speech enhancement can introduce its own distortions, and jointly training these modules can lead to complex architectures. This paper aims to build noise robustness directly into the speech representation learning process itself, bypassing the need for a separate enhancement front-end.
Methods
The core of this work is an extension of the wav2vec 2.0 self-supervised learning framework. Wav2vec 2.0 learns speech representations by using contrastive learning to distinguish correct quantized speech representations from distractors. The authors augment this by adding a speech reconstruction module. This module is trained concurrently with the contrastive task during a "continual pre-training" phase on noisy speech data. The goal is for the model to learn representations that not only capture the acoustic information for ASR but also contain enough detail to reconstruct the original clean speech waveform from the noisy input. This reconstruction module is only used during this pre-training stage and is discarded during fine-tuning and inference, making the final ASR system efficient. The reconstruction module itself is built using a two-layer bidirectional LSTM (BLSTM) followed by a CNN decoder, inspired by Convolutional Recurrent Networks (CRNs) used in speech enhancement. The overall training objective is a multi-task loss combining the original contrastive loss, a diversity loss, and the new reconstruction loss.
Datasets
- LibriSpeech: Used for generating synthesized noisy speech. The
train-clean-100subset was used for training, andtest-cleanandtest-othersubsets were used for evaluation. - DNS-Challenge: Noise files from this challenge were used to create synthesized noisy speech by mixing with LibriSpeech data at various Signal-to-Noise Ratios (SNRs).
- MUSAN corpus: Used to create noisy speech for evaluation by mixing unseen noise files with test utterances.
- CHiME-4 challenge: Real-world noisy speech data from this challenge was used for evaluation. Specifically, the 1-channel track (
dev-realandeval-realsubsets) was utilized.
Key Insights and Ideas
- Implicit Denoising through Reconstruction: By forcing the model to reconstruct clean speech from noisy input, the learned representations are implicitly encouraged to discard noise-related information and retain speech-relevant features. This is a more integrated way to achieve noise robustness than traditional cascaded systems.
- Reconstruction Module as Auxiliary Task: The reconstruction task acts as a powerful auxiliary objective during pre-training. It helps the model learn richer, more robust representations without needing to be present during the final ASR inference.
- Balancing Clean and Noisy Performance: The proposed method demonstrates the ability to improve performance on noisy speech while maintaining or even slightly improving performance on clean speech, a common trade-off in many noise-robustness techniques.
- Low-Resource ASR Potential: The results on CHiME-4 highlight that this approach can achieve competitive, even state-of-the-art, performance with significantly less labeled data compared to supervised methods.
Key Results
- Synthesized Noisy LibriSpeech: The proposed method achieved substantial Word Error Rate (WER) reductions. For example, with a 4-gram language model, WER dropped from 20.9% (baseline) to 7.5% on
test-cleanand from 40.0% to 20.2% ontest-otherwhen compared to a baseline without the reconstruction module. - Real-World Noisy CHiME-4: On the CHiME-4 1-channel track, the approach achieved state-of-the-art ASR performance without any denoising front-end. With only 16% of labeled data, it achieved a 6.5% average WER, outperforming a supervised approach by 0.4%. Using all labeled data, the WER reached 4.1%, significantly lower than the best supervised approach.
- Ablation Study: The study showed that attaching the reconstruction module to the contextual representations (before quantization) yielded the best results. Replacing the CRN-inspired reconstruction module with a BLSTM resulted in slightly worse performance on noisy conditions.
Limitations and Future Work
- Paired Data Requirement: The current continual pre-training requires paired noisy and clean speech data. This is a limitation as such paired data is not always available.
- Implicit Constraints: The reconstruction process imposes only implicit constraints on the speech representations.
- Future Work: The authors propose to mitigate the paired data requirement by introducing a masker layer for intermediate representations and incorporating mixture-invariant training. They also aim to explore more advanced techniques to further improve noise robustness.
Summary updated