Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction

Heming Wang, Yao Qian, Xiaofei Wang, Yiming Wang, Chengyi Wang, Shujie Liu, Takuya Yoshioka, Jinyu Li, DeLiang Wang

TL;DR

This paper introduces a novel self-supervised learning approach for robust automatic speech recognition (ASR) by integrating a speech reconstruction task into the wav2vec 2.0 framework. This method significantly improves ASR performance in noisy conditions without requiring a separate speech enhancement front-end, achieving state-of-the-art results on real-world noisy speech with limited labeled data.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are crucial for many applications, but their performance drastically degrades in the presence of background noise, a common issue in real-world scenarios. Traditionally, this has been tackled by using a cascaded approach: a speech enhancement module first cleans up the noisy audio, and then the enhanced speech is fed to the ASR system. However, speech enhancement can introduce its own distortions, and jointly training these modules can lead to complex architectures. This paper aims to build noise robustness directly into the speech representation learning process itself, bypassing the need for a separate enhancement front-end.

Methods

The core of this work is an extension of the wav2vec 2.0 self-supervised learning framework. Wav2vec 2.0 learns speech representations by using contrastive learning to distinguish correct quantized speech representations from distractors. The authors augment this by adding a speech reconstruction module. This module is trained concurrently with the contrastive task during a "continual pre-training" phase on noisy speech data. The goal is for the model to learn representations that not only capture the acoustic information for ASR but also contain enough detail to reconstruct the original clean speech waveform from the noisy input. This reconstruction module is only used during this pre-training stage and is discarded during fine-tuning and inference, making the final ASR system efficient. The reconstruction module itself is built using a two-layer bidirectional LSTM (BLSTM) followed by a CNN decoder, inspired by Convolutional Recurrent Networks (CRNs) used in speech enhancement. The overall training objective is a multi-task loss combining the original contrastive loss, a diversity loss, and the new reconstruction loss.

Datasets

  • LibriSpeech: Used for generating synthesized noisy speech. The train-clean-100 subset was used for training, and test-clean and test-other subsets were used for evaluation.
  • DNS-Challenge: Noise files from this challenge were used to create synthesized noisy speech by mixing with LibriSpeech data at various Signal-to-Noise Ratios (SNRs).
  • MUSAN corpus: Used to create noisy speech for evaluation by mixing unseen noise files with test utterances.
  • CHiME-4 challenge: Real-world noisy speech data from this challenge was used for evaluation. Specifically, the 1-channel track (dev-real and eval-real subsets) was utilized.

Key Insights and Ideas

  • Implicit Denoising through Reconstruction: By forcing the model to reconstruct clean speech from noisy input, the learned representations are implicitly encouraged to discard noise-related information and retain speech-relevant features. This is a more integrated way to achieve noise robustness than traditional cascaded systems.
  • Reconstruction Module as Auxiliary Task: The reconstruction task acts as a powerful auxiliary objective during pre-training. It helps the model learn richer, more robust representations without needing to be present during the final ASR inference.
  • Balancing Clean and Noisy Performance: The proposed method demonstrates the ability to improve performance on noisy speech while maintaining or even slightly improving performance on clean speech, a common trade-off in many noise-robustness techniques.
  • Low-Resource ASR Potential: The results on CHiME-4 highlight that this approach can achieve competitive, even state-of-the-art, performance with significantly less labeled data compared to supervised methods.

Key Results

  • Synthesized Noisy LibriSpeech: The proposed method achieved substantial Word Error Rate (WER) reductions. For example, with a 4-gram language model, WER dropped from 20.9% (baseline) to 7.5% on test-clean and from 40.0% to 20.2% on test-other when compared to a baseline without the reconstruction module.
  • Real-World Noisy CHiME-4: On the CHiME-4 1-channel track, the approach achieved state-of-the-art ASR performance without any denoising front-end. With only 16% of labeled data, it achieved a 6.5% average WER, outperforming a supervised approach by 0.4%. Using all labeled data, the WER reached 4.1%, significantly lower than the best supervised approach.
  • Ablation Study: The study showed that attaching the reconstruction module to the contextual representations (before quantization) yielded the best results. Replacing the CRN-inspired reconstruction module with a BLSTM resulted in slightly worse performance on noisy conditions.

Limitations and Future Work

  • Paired Data Requirement: The current continual pre-training requires paired noisy and clean speech data. This is a limitation as such paired data is not always available.
  • Implicit Constraints: The reconstruction process imposes only implicit constraints on the speech representations.
  • Future Work: The authors propose to mitigate the paired data requirement by introducing a masker layer for intermediate representations and incorporating mixture-invariant training. They also aim to explore more advanced techniques to further improve noise robustness.

Summary updated

Abstract

Noise robustness is essential for deploying automatic speech recognition (ASR) systems in real-world environments. One way to reduce the effect of noise interference is to employ a preprocessing module that conducts speech enhancement, and then feed the enhanced speech to an ASR backend. In this work, instead of suppressing background noise with a conventional cascaded pipeline, we employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We propose to combine a reconstruction module with contrastive learning and perform multi-task continual pre-training on noisy data. The reconstruction module is used for auxiliary learning to improve the noise robustness of the learned representation and thus is not required during inference. Experiments demonstrate the effectiveness of our proposed method. Our model substantially reduces the word error rate (WER) for the synthesized noisy LibriSpeech test sets, and yields around 4.1/7.5% WER reduction on noisy clean/other test sets compared to data augmentation. For the real-world noisy speech from the CHiME-4 challenge (1-channel track), we have obtained the state of the art ASR performance without any denoising front-end. Moreover, we achieve comparable performance to the best supervised approach reported with only 16% of labeled data.
BibTeX
@inproceedings{wang2022improving,
  author = {Heming Wang and Yao Qian and Xiaofei Wang and Yiming Wang and Chengyi Wang and Shujie Liu and Takuya Yoshioka and Jinyu Li and DeLiang Wang},
  title = {{Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746220},
}
Figure