An investigation of modularity for noise robustness in conformer-based ASR

Louise Coppieters de Gibson, Philip N. Garner, Pierre-Edouard HonnetarXiv 2024 · DOI · arXiv

Automatic Speech Recognition (ASR) has reached incredible heights, but it still has a "kryptonite": unexpected background noise. When a model trained on clean studio audio meets a windy street or a clattering cafe, performance often plummets. In a recent paper, researchers from Idiap and Telepathy Labs investigate whether modularity—breaking a model into specialized "experts"—can make ASR more robust without exploding the computational cost.

TL;DR

By introducing specialized "expert" blocks into a Conformer-based ASR model, researchers significantly improved accuracy in noisy environments. Surprisingly, they found that a simple split between "clean" and "noisy" experts performs better than having many experts for specific noise types (like buses vs. cafes), while also allowing the model to train much faster.

What Problem Does This Paper Address?

Modern ASR models are massive, often based on the Conformer architecture (which combines the local processing of CNNs with the global reach of Transformers). While powerful, these "monolithic" models face two big hurdles:

  1. Negative Interference: Learning to handle "bus noise" might accidentally degrade the model's ability to recognize "clean speech."
  2. Data Scarcity: We often have very little data for specific new environments (e.g., a specific factory floor), making it hard to adapt a giant model without it "forgetting" what it already knows.

The authors wanted to see if modularity—specifically a "Mixture of Experts" (MoE) approach—could allow a model to specialize in different acoustic conditions effectively.

Methods

The researchers modified the standard Conformer encoder by introducing modular layers. A modular layer consists of:

  • Experts: Multiple versions of a Conformer block.
  • Router: A mechanism that decides which expert should process a specific piece of audio.
  • Aggregator: A component that combines the outputs.

They tested two routing strategies:

  1. Fixed Routing: The "ground truth" approach. If the data label says the audio is from a cafe, the "cafe expert" is used. This acts as an upper bound for performance.
  2. Learned Routing: A more realistic scenario where a small pre-trained classifier (3 CNN blocks) listens to the audio and "guesses" which expert to send it to.

To keep the comparison fair, the model only uses one expert per utterance, ensuring the computational cost during inference remains the same as the non-modular baseline.

Datasets

The study used the CHiME4 dataset, a benchmark for noisy speech recognition. It includes:

  • Clean data: Wall Street Journal (WSJ) utterances.
  • Noisy data: Utterances recorded in or mixed with four environments: Bus, Cafe, Street, and Pedestrian area.
  • Real vs. Simulated: It contains both real-world noisy recordings and "simulated" noise (clean speech mixed with noise digitally).

Key Insights and Ideas

  • The "Less is More" Paradox: You might think having five experts (one for each noise type + clean) would be best. However, the researchers found that two experts (Clean vs. Noisy) often outperformed five. This is due to a trade-off: more experts mean each one sees less data during training, preventing them from becoming truly "expert."
  • Acoustic Confusion: The learned router was excellent at distinguishing "clean" from "noisy" speech, but struggled to tell the difference between a "bus" and a "street." It turns out that from a feature perspective, many noises are quite similar, and a general "noise-robust" expert is more useful than a "bus-specific" one.
  • Training Efficiency: Modular models converged significantly faster than the baseline. The modular versions reached peak performance around epoch 20, while the standard Conformer took 25-35 epochs. Specialization seems to simplify the optimization landscape.

Key Results

The modular approach consistently beat the standard Conformer baseline across the board:

  • Clean Speech: Achieved a relative Word Error Rate (WER) reduction of 36.3% to 40.4%.
  • Noisy Speech: Improvements ranged between 6.1% and 15.1% for most noise types.
  • Best Configuration: Using just two experts (one for clean, one for all noise) on the first layer of the encoder provided the best balance of performance and simplicity.
  • Learned vs. Fixed: The "Learned Routing" (using the CNN classifier) performed almost as well as "Fixed Routing," proving this method is viable for real-world applications where noise labels aren't provided.

Limitations and Future Work

The authors noted that their classifier-based router is still a bit of a bottleneck; if the classifier picks the wrong expert, performance suffers. They also pointed out that while modularity helps, the distinction between "real" and "simulated" noise remains a challenge for the models.

Future directions include exploring "fully learned routing"—where the model discovers its own categories of audio without being told what "clean" or "noisy" is—and testing whether this modularity holds up as models scale even larger.

Summary updated

Abstract

Whilst state of the art automatic speech recognition (ASR) can perform well, it still degrades when exposed to acoustic environments that differ from those used when training the model. Unfamiliar environments for a given model may well be known a-priori, but yield comparatively small amounts of adaptation data. In this experimental study, we investigate to what extent recent formalisations of modularity can aid adaptation of ASR to new acoustic environments. Using a conformer based model and fixed routing, we confirm that environment awareness can indeed lead to improved performance in known environments. However, at least on the (CHIME) datasets in the study, it is difficult for a classifier module to distinguish different noisy environments, a simpler distinction between noisy and clean speech being the optimal configuration. The results have clear implications for deploying large models in particular environments with or without a-priori knowledge of the environmental noise.
BibTeX
@misc{gibson2024investigation,
  author = {Louise Coppieters de Gibson and Philip N. Garner and Pierre-Edouard Honnet},
  title = {{An investigation of modularity for noise robustness in conformer-based ASR}},
  year = {2024},
  eprint = {2409.05589},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.05589},
}
Figure