Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition

Yiming Wang, Jinyu Li, Heming Wang, Yao Qian, Chengyi Wang, Yu Wu

Abstract

The goal of self-supervised learning (SSL) for automatic speech recognition (ASR) is to learn good speech representations from a large amount of unlabeled speech for the downstream ASR task. However, most SSL frameworks do not consider noise robustness which is crucial for real-world applications. In this paper we propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech via contrastive learning. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets of each other. By doing this, it enforces the network to have consistent predictions for the original and noisy speech, thus allows to learn contextualized representation with noise robustness. Our experiments on synthesized and real noisy data show the effectiveness of our method: it achieves 2.9–4.9% relative word error rate (WER) reduction on the synthesized noisy LibriSpeech data without deterioration on the original data, and 5.7% on CHiME-4 real 1-channel noisy data compared to a data augmentation baseline even with a strong language model for decoding. Our results on CHiME-4 can match or even surpass those with well-designed speech enhancement components.

Introduction

Self-supervised learning (SSL) has revolutionized Automatic Speech Recognition (ASR) by allowing models to learn from massive amounts of unlabeled data. However, most SSL frameworks—including the popular wav2vec 2.0—are "noise-blind." They are typically trained on relatively clean datasets like LibriSpeech and struggle significantly when deployed in messy, real-world environments with background noise or poor recording quality.

Traditionally, researchers solve this by plugging in a dedicated "speech enhancement" front-end to denoise the audio before it reaches the ASR system. While effective, this adds significant complexity to the neural network architecture and doesn't always result in the best ASR performance, as the denoising objective isn't always perfectly aligned with the recognition objective.

Methods

The authors introduce wav2vec-Switch, which modifies the pre-training phase of wav2vec 2.0 to bake noise robustness directly into speech representations.

The "Switch" Mechanism

In standard wav2vec 2.0, the model tries to predict a "quantized" (discretized) version of a speech segment from its surrounding context. Wav2vec-Switch takes an original audio clip ($X$) and creates a noisy version ($\tilde{X}$) by adding noise on-the-fly.

The wav2vec-Switch architecture showing the cross-prediction of quantized targets between original and noisy speech streams.

Both clips are fed through the network in parallel. Instead of just calculating the standard loss, the model performs a "switch":

  1. It tries to predict the original target from the noisy context.
  2. It tries to predict the noisy target from the original context.

This forces the "context network" (the part of the model that understands the sequence) to ignore the noise and focus on the underlying speech structure that is common to both versions.

The Consistency Constraint

A critical technical detail is that the internal states of the network—specifically the dropout masks and the positions of the masked segments—must be identical for both the original and noisy versions in a pair. If these are different, the model's representations diverge, and the "switch" task becomes too chaotic for the model to learn effectively.

Key Results

  • Synthesized Data: On noisy LibriSpeech, wav2vec-Switch achieved a 7.1–11.0% relative reduction in Word Error Rate (WER) compared to a standard data-augmentation baseline (when no language model was used).
  • Real-World Data: On the CHiME-4 real 1-channel track, the model achieved a 5.7% relative WER reduction even when using a strong language model for decoding.
Performance comparison on the CHiME-4 real 1-channel noisy dataset.
  • Competitive Performance: Remarkably, this self-supervised approach matched or outperformed several state-of-the-art supervised systems that used complex, dedicated speech enhancement front-ends. Unlike speech enhancement methods that add new layers or modules, wav2vec-Switch only changes the loss function during pre-training. The final model used for inference is the exact same size as a standard wav2vec 2.0 model.

Conclusion

Wav2vec-Switch demonstrates that noise robustness can be treated as a consistency problem during self-supervised pre-training. By enforcing consistent predictions across original and noisy pairs, the model learns noise-invariant features without architectural overhead. Future work may involve applying this consistency logic to other SSL objectives, such as masked language modeling (e.g., HuBERT or WavLM), and exploring its scalability on even larger unlabeled datasets.

Summary updated

Abstract

The goal of self-supervised learning (SSL) for automatic speech recognition (ASR) is to learn good speech representations from a large amount of unlabeled speech for the downstream ASR task. However, most SSL frameworks do not consider noise robustness which is crucial for real-world applications. In this paper we propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech via contrastive learning. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets of each other. By doing this, it enforces the network to have consistent predictions for the original and noisy speech, thus allows to learn contextualized representation with noise robustness. Our experiments on synthesized and real noisy data show the effectiveness of our method: it achieves 2.9--4.9% relative word error rate (WER) reduction on the synthesized noisy LibriSpeech data without deterioration on the original data, and 5.7% on CHiME-4 real 1-channel noisy data compared to a data augmentation baseline even with a strong language model for decoding. Our results on CHiME-4 can match or even surpass those with well-designed speech enhancement components.
BibTeX
@inproceedings{wang2022wav,
  author = {Yiming Wang and Jinyu Li and Heming Wang and Yao Qian and Chengyi Wang and Yu Wu},
  title = {{Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746929},
}
Figure