Robust Data2VEC: Noise-Robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learning

Qiu-Shi Zhu, Long Zhou, Jie Zhang, Shujie Liu, Yu-Chen Hu, Li-Rong DaiICASSP 2022 · OpenReview

TL;DR

This paper introduces "Robust Data2vec," a self-supervised learning approach for speech recognition that combines regression and contrastive learning to improve robustness against noisy audio. By cleverly constructing "non-semantic" negative samples and filtering out easily distinguishable ones during contrastive learning, the model achieves significantly better performance on noisy speech tasks.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle with noisy audio, which is a common real-world scenario. While self-supervised learning (SSL) methods like contrastive learning (e.g., wav2vec 2.0) and regression-based approaches (e.g., data2vec) have shown promise in leveraging vast amounts of unlabeled data to improve ASR, their combined effect on noise robustness and the optimal strategies for contrastive learning in this context remain underexplored. Specifically, the paper questions how to best design negative samples for contrastive learning to enhance robustness and whether combining regression and contrastive learning offers benefits beyond their individual strengths.

Methods

The core of this work is an enhanced version of the data2vec framework. The authors propose a joint training strategy that combines a regression loss (predicting target representations from a teacher model) with an improved contrastive loss. The novelties lie in two key modifications to the contrastive learning component:

  1. Patch-based Non-Semantic Negative Samples: Instead of relying solely on standard negative samples (other frames or sequences), they create "non-semantic" negative samples by dividing the feature representations into patches of varying sizes and then shuffling these patches. This aims to expose the model to more diverse and challenging perturbations, forcing it to learn higher-level, noise-robust representations.
  2. Removal of Distinguishable Negative Samples: By analyzing the cosine similarity distribution between predicted and negative samples, they observe that as training progresses, some negative samples become too easy to distinguish from positive ones. The paper proposes an incremental removal of these "easily distinguishable" negative samples, arguing that focusing on harder negatives can improve the model's discriminative capacity and lead to more robust representations.

The overall model architecture follows the data2vec paradigm, featuring a feature encoder and a Transformer encoder, with a teacher model generated via exponential moving average (EMA) of the student model's parameters.

Datasets

  • Training/Pre-training: MUSAN dataset (109 hours of speech, music, and noise) and CHiME-4 noise dataset. Noisy speech for pre-training is synthesized by mixing clean speech with noise from these sources at various Signal-to-Noise Ratios (SNRs) from 0 to 25 dB.
  • Evaluation: CHiME-4 dataset (real and artificially simulated noisy speech). The single-channel close-by noisy data is used for validation and testing.

Key Insights and Ideas

  • Joint Training Synergy: Combining regression and contrastive learning in SSL for ASR is not just additive; it offers synergistic benefits, particularly in improving noise robustness and potentially mitigating model collapse issues seen in regression-only approaches.
  • The Power of "Bad" Samples: Creating "non-semantic" negative samples by patching and shuffling features is a surprisingly effective way to push the model towards learning representations that are less sensitive to noise. This suggests that exposing the model to structured but semantically irrelevant perturbations can be more beneficial than random noise.
  • Focusing on the Hard Stuff: Not all negative samples are created equal. Actively filtering out "easy" negatives and focusing on harder ones during contrastive learning can significantly boost the model's ability to discriminate between correct and incorrect speech features, leading to better performance in challenging conditions.
  • Noise Robustness is Achievable: The work demonstrates that with careful design of SSL objectives and negative sample strategies, significant improvements in ASR performance on noisy data can be achieved without requiring explicit noise reduction pre-processing during fine-tuning.

Key Results

  • The proposed "Robust Data2vec" model, when fine-tuned on the CHiME-4 dataset, achieved a Word Error Rate (WER) of 3.1% on the validation set and 5.8% on the test set, outperforming previous state-of-the-art self-supervised methods on this noisy ASR benchmark.
  • Ablation studies confirmed the effectiveness of each proposed component:
  • Joint regression and standard contrastive loss: WER reduced from 9.5/15.7 (regression only) to 8.9/14.2.
  • Adding non-semantic negative samples: WER further reduced to 8.5/13.3.
  • Adding removal of distinguishable negatives: Achieved the best result of 8.3/12.8 before joint decoding with an LM.
  • The paper also showed that joint training of regression and contrastive learning can help avoid model collapse, a problem sometimes observed when only regression is used.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper mentions that the choice of the number of standard vs. non-semantic negative samples matters, with 50 of each yielding the best results. This suggests potential sensitivity to hyperparameter tuning for optimal performance.
  • Computational Cost: While not explicitly detailed, joint training with multiple loss functions and complex negative sample generation likely increases computational requirements.
  • Future Work: The authors suggest that further exploration into different strategies for constructing negative samples and analyzing their impact on various downstream tasks could be beneficial. Investigating the theoretical underpinnings of why these specific negative sample strategies improve noise robustness could also be a valuable direction.

Summary updated

Abstract

Self-supervised pre-training methods based on contrastive learning or regression tasks can utilize more unlabeled data to improve the performance of automatic speech recognition (ASR). However, the robustness impact of combining the two pre-training tasks and constructing different negative samples for contrastive learning still remains unclear. In this paper, we propose a noise-robust data2vec for self-supervised speech representation learning by jointly optimizing the contrastive learning and regression tasks in the pre-training stage. Furthermore, we present two improved methods to facilitate contrastive learning. More specifically, we first propose to construct patch-based non-semantic negative samples to boost the noise robustness of the pre-training model, which is achieved by dividing the features into patches at different sizes (i.e., so-called negative samples). Second, by analyzing the distribution of positive and negative samples, we propose to remove the easily distinguishable negative samples to improve the discriminative capacity for pre-training models. Experimental results on the CHiME-4 dataset show that our method is able to improve the performance of the pre-trained model in noisy scenarios. We find that joint training of the contrastive learning and regression tasks can avoid the model collapse to some extent compared to only training the regression task.
BibTeX
@inproceedings{zhu2022robust,
  author = {Qiu-Shi Zhu and Long Zhou and Jie Zhang and Shujie Liu and Yu-Chen Hu and Li-Rong Dai},
  title = {{Robust Data2VEC: Noise-Robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=fuVAMH2VRB},
}
Figure