Analyzing Robustness of End-to-End Neural Models for Automatic Speech Recognition

Goutham Rajendran, Wei ZouarXiv 2022 · OpenReview

Abstract

We investigate robustness properties of pre-trained neural models for automatic speech recognition. Real life data in machine learning is usually very noisy and almost never clean, which can be attributed to various factors depending on the domain, e.g. outliers, random noise and adversarial noise. Therefore, the models we develop for various tasks should be robust to such kinds of noisy data, which led to the thriving field of robust machine learning. We consider this important issue in the setting of automatic speech recognition. With the increasing popularity of pre-trained models, it's an important question to analyze and understand the robustness of such models to noise. In this work, we perform a robustness analysis of the pre-trained neural models wav2vec2, HuBERT and DistilHuBERT on the LibriSpeech and TIMIT datasets. We use different kinds of noising mechanisms and measure the model performances as quantified by the inference time and the standard Word Error Rate metric. We also do an in-depth layer-wise analysis of the wav2vec2 model when injecting noise in between layers, enabling us to predict at a high level what each layer learns. Finally for this model, we visualize the propagation of errors across the layers and compare how it behaves on clean versus noisy data. Our experiments conform the predictions of Pasad et al. [2021] and also raise interesting directions for future work.

Introduction

Automatic Speech Recognition (ASR) has been transformed by massive pre-trained models. However, most of these models are benchmarked on "clean" datasets that do not reflect the chaos of the real world—think background chatter, hardware glitches, or varying speech speeds. This research addresses a fundamental gap: the lack of understanding regarding the robustness of these end-to-end models. For ASR to be deployed in safety-critical or high-stakes environments, such as autonomous driving or medical transcription, it is essential to determine how much noise a model can tolerate before performance collapses and which architectural components are most vulnerable to corruption.

Methodology

Input Perturbation

The study modified raw input waveforms using three types of simulated noise:

  • White Noise: Adding random Gaussian noise with a mixing probability ($\rho$).
  • Speed Perturbation: Adjusting the playback speed of the audio without altering the pitch.
  • Chunk Drop: Randomly deleting segments of the audio to simulate "dropped frames" or packet loss.

Layer-wise Analysis

The researchers conducted a "surgical" analysis of the wav2vec2 model by intervening during inference. They injected additive and multiplicative noise directly into the outputs of specific layers to identify noise-sensitive regions. Additionally, they tracked activation distances (L2) between clean and noisy audio to observe how the network processes and potentially mitigates noise as the signal travels through the layers.

Datasets

Two industry-standard benchmarks were utilized:

  • LibriSpeech: A sub-sampled version of the test-clean split.
  • TIMIT: Used primarily to compare wav2vec2 against DistilHuBERT, a compressed version of HuBERT.

Results and Discussion

Comparative Robustness

Experiments on LibriSpeech revealed that HuBERT consistently maintained a lower Word Error Rate (WER) than wav2vec2 across all noise levels for white noise and speed perturbations. While HuBERT is approximately 25% slower in terms of inference time, its resilience makes it a superior choice for noisy environments.

Word Error Rate (WER) vs. White Noise intensity for wav2vec2 and HuBERT.

Furthermore, the relationship between speech speed and error was found to be convex. Models handle slight deviations well, but performance drops sharply once the speed deviates significantly from the original recording. This suggests that while the models are robust to minor temporal variations, they lack the generalization required for extreme speed shifts.

Effect of speed perturbation on WER, showing the convex relationship for both models.

Architectural Insights

  • Semantic Bottleneck: In wav2vec2, layers 6 through 8 were found to be the most sensitive to additive noise, suggesting these layers are critical for high-level semantic processing. Corrupting signals at this stage leads to dramatic spikes in WER.
  • Denoising Stack: The transformer architecture inherently performs a form of denoising. The difference between clean and noisy activations decreases in later layers, indicating the model "eats" the noise during transcription.
  • Layer 11 Anomaly: Layer 11 in wav2vec2 consistently behaved as a functional outlier, showing extreme sensitivity to multiplicative noise and breaking the general denoising trend observed in other layers.

Conclusion

This work demonstrates a clear trade-off between efficiency and robustness in ASR models. While wav2vec2 offers faster inference, HuBERT provides significantly greater resilience to input corruption. Future research should focus on more realistic environmental noise patterns, such as cafe chatter or traffic, and investigate adversarial robustness to mathematically crafted noise designed to deceive modern transformer models.

Summary updated

Abstract

We investigate robustness properties of pre-trained neural models for automatic speech recognition. Real life data in machine learning is usually very noisy and almost never clean, which can be attributed to various factors depending on the domain, e.g. outliers, random noise and adversarial noise. Therefore, the models we develop for various tasks should be robust to such kinds of noisy data, which led to the thriving field of robust machine learning. We consider this important issue in the setting of automatic speech recognition. With the increasing popularity of pre-trained models, it's an important question to analyze and understand the robustness of such models to noise. In this work, we perform a robustness analysis of the pre-trained neural models wav2vec2, HuBERT and DistilHuBERT on the LibriSpeech and TIMIT datasets. We use different kinds of noising mechanisms and measure the model performances as quantified by the inference time and the standard Word Error Rate metric. We also do an in-depth layer-wise analysis of the wav2vec2 model when injecting noise in between layers, enabling us to predict at a high level what each layer learns. Finally for this model, we visualize the propagation of errors across the layers and compare how it behaves on clean versus noisy data. Our experiments conform the predictions of Pasad et al. [2021] and also raise interesting directions for future work.
BibTeX
@misc{rajendran2022analyzing,
  author = {Goutham Rajendran and Wei Zou},
  title = {{Analyzing Robustness of End-to-End Neural Models for Automatic Speech Recognition}},
  year = {2022},
  eprint = {2208.08509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.08509},
}
Figure