Wav2vec-Switch: Contrastive Learning from Original-Noisy Speech Pairs for Robust Speech Recognition

Yiming Wang, Jinyu Li, Heming Wang, Yao Qian, Chengyi Wang, Yu Wu

TL;DR

Wav2vec-Switch is a simple yet powerful modification to the wav2vec 2.0 framework that bakes noise robustness directly into speech representations. By "switching" targets between original and noisy speech pairs during contrastive pre-training, the model learns to produce consistent, noise-invariant features without requiring complex external enhancement modules.

What Problem Does This Paper Address?

Self-supervised learning (SSL) has revolutionized Automatic Speech Recognition (ASR) by allowing models to learn from massive amounts of unlabeled data. However, most SSL frameworks—including the popular wav2vec 2.0—are "noise-blind." They are typically trained on relatively clean datasets like LibriSpeech and struggle significantly when deployed in messy, real-world environments with background noise or poor recording quality.

Traditionally, researchers solve this by plugging in a dedicated "speech enhancement" front-end to denoise the audio before it reaches the ASR system. While effective, this adds significant complexity to the neural network architecture and doesn't always result in the best ASR performance, as the denoising objective isn't always perfectly aligned with the recognition objective.

Methods

The authors introduce wav2vec-Switch, which modifies the pre-training phase of wav2vec 2.0.

The "Switch" Mechanism

In standard wav2vec 2.0, the model tries to predict a "quantized" (discretized) version of a speech segment from its surrounding context. Wav2vec-Switch takes an original audio clip ($X$) and creates a noisy version ($\tilde{X}$) by adding noise on-the-fly.

Both clips are fed through the network in parallel. Instead of just calculating the standard loss, the model performs a "switch":

  1. It tries to predict the original target from the noisy context.
  2. It tries to predict the noisy target from the original context.

This forces the "context network" (the part of the model that understands the sequence) to ignore the noise and focus on the underlying speech structure that is common to both versions.

The Consistency Constraint

A critical technical detail is that the internal states of the network—specifically the dropout masks and the positions of the masked segments—must be identical for both the original and noisy versions in a pair. If these are different, the model's representations diverge, and the "switch" task becomes too chaotic for the model to learn effectively.

Datasets

  • Pre-training: The 960-hour LibriSpeech corpus, augmented with noise from the MUSAN dataset (music and general noise).
  • Fine-tuning: The 100-hour "clean" subset of LibriSpeech and the CHiME-4 dataset.
  • Evaluation:
    • Synthesized noisy LibriSpeech (various Signal-to-Noise Ratios).
  • CHiME-4: A challenging real-world benchmark featuring speech recorded in noisy environments like buses, cafes, and street junctions.

Key Insights and Ideas

  • Noise Invariance via Contrast: The core "aha moment" is that noise robustness can be treated as a consistency problem. If a model is truly robust, its high-level understanding of a sentence shouldn't change just because a bus drove by in the background.
  • Zero Architectural Overhead: Unlike speech enhancement methods that add new layers or modules, wav2vec-Switch only changes the loss function during pre-training. The final model used for inference is the exact same size as a standard wav2vec 2.0 model.
  • The Importance of Random State Syncing: The ablation study highlights that keeping dropout and masking identical is not just a minor detail—it's essential. Without this synchronization, performance on original speech actually degrades.

Key Results

  • Synthesized Data: On noisy LibriSpeech, wav2vec-Switch achieved a 7.1–11.0% relative reduction in Word Error Rate (WER) compared to a standard data-augmentation baseline (when no language model was used).
  • Real-World Data: On the CHiME-4 real 1-channel track, the model achieved a 5.7% relative WER reduction even when using a strong language model for decoding.
  • Competitive Performance: Remarkably, this self-supervised approach matched or outperformed several state-of-the-art supervised systems that used complex, dedicated speech enhancement front-ends.

Limitations and Future Work

  • Pair Requirement: The method currently relies on having "original-noisy" pairs during pre-training. While easy to synthesize, it might be harder to apply to scenarios where only noisy data is available without a clean reference.
  • Scaling: The authors suggest exploring even larger unlabeled datasets to see if the robustness gains scale linearly with data size.
  • Beyond Contrastive Learning: Future work could involve applying this "switch" or consistency logic to other SSL objectives, such as masked language modeling (like HuBERT or WavLM).

Summary updated

Abstract

The goal of self-supervised learning (SSL) for automatic speech recognition (ASR) is to learn good speech representations from a large amount of unlabeled speech for the downstream ASR task. However, most SSL frameworks do not consider noise robustness which is crucial for real-world applications. In this paper we propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech via contrastive learning. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets of each other. By doing this, it enforces the network to have consistent predictions for the original and noisy speech, thus allows to learn contextualized representation with noise robustness. Our experiments on synthe-sized and real noisy data show the effectiveness of our method: it achieves 2.9–4.9% relative word error rate (WER) reduction on the synthesized noisy LibriSpeech data without deterioration on the original data, and 5.7% on CHiME-4 real 1-channel noisy data compared to a data augmentation baseline even with a strong language model for decoding. Our results on CHiME-4 can match or even surpass those with well-designed speech enhancement components.
BibTeX
@inproceedings{wang2022wav,
  author = {Yiming Wang and Jinyu Li and Heming Wang and Yao Qian and Chengyi Wang and Yu Wu},
  title = {{Wav2vec-Switch: Contrastive Learning from Original-Noisy Speech Pairs for Robust Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746929},
}

Loading related content...

Figure