Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models

Christopher Simic, Korbinian Riedhammer, Tobias Bocklet

Introduction

Automatic Speech Recognition (ASR) has reached impressive levels of accuracy in clean environments. However, performance often degrades significantly in "the wild" due to background noise such as music, overlapping speech, or environmental sounds. Audio-Visual Speech Recognition (AVSR) addresses this by integrating visual cues—specifically lip movements—to supplement the audio signal.

While effective, state-of-the-art AVSR models are typically trained from scratch, requiring massive computational resources and ignoring the robust pre-training available in large-scale ASR models like OpenAI’s Whisper. This research proposes a modular, parameter-efficient extension that adapts pre-trained ASR models for robust AVSR performance across diverse noise conditions.

Methodology

The proposed architecture utilizes a "multi-agent" approach to handle varying noise scenarios dynamically. The system consists of three primary components integrated with a frozen pre-trained Whisper model:

  1. AV Fusion Module: A lightweight module added upstream of the encoder that extracts visual features and fuses them with audio features using cross-attention.
  2. LoRA Adapters: Instead of full fine-tuning, Low-Rank Adaptation (LoRA) layers are inserted into the Whisper transformer blocks. Only these small adapter layers are updated during training.
  3. Multi-Agent Selection: The system maintains a bank of specialized "adapter-sets." A ResNet-based noise-scenario classifier analyzes the input audio to select the most appropriate adapter-set (e.g., specialized for babble noise or high-noise levels) for the current environment.
Architecture of the proposed multi-agent AVSR system, showing the frozen Whisper core and the dynamic selection of LoRA adapter-sets via a noise classifier.

Experimental Setup

The models were evaluated on the LRS3-TED benchmark (430 hours) after self-supervised pre-training on VoxCeleb2 (1,700 hours). To simulate real-world conditions, speech data was synthetically contaminated using the Musan corpus, covering music, natural sounds, and "babble" (multi-speaker) noise at various Signal-to-Noise Ratios (SNR).

Key Results and Discussion

The multi-agent approach demonstrates significant improvements in both efficiency and performance. By specializing adapters for specific noise levels (SNR) or categories, the model achieves results comparable to fully fine-tuned state-of-the-art models while using a fraction of the trainable parameters.

Comparative performance table showing Word Error Rate (WER) across different noise types and SNR levels, alongside trainable (TrP) and total (ToP) parameters.

Parameter Efficiency

As shown in the results table, the "small" version of the LoRA-AVSR model achieves performance competitive with AV-HuBERT and AV-Fusion while utilizing 79.3% to 89.1% fewer trainable parameters. For instance, the base LoRA-AVSR requires only 18M trainable parameters compared to the 87M required by the AV-Fusion base model.

Robustness to Noise

The specialization strategy proves particularly effective in high-noise scenarios (-10dB SNR). The data indicates that specializing adapters based on the Signal-to-Noise Ratio (SNR) is generally more effective than specializing by noise category. In the difficult "Sidespeaker" scenario—where background speech competes with the primary speaker—the multi-agent model outperformed the fully fine-tuned baseline at the lowest SNR levels.

Conclusion

This research demonstrates that a modular, adapter-based approach can effectively upgrade pre-trained ASR models for complex audio-visual tasks. By employing a bank of specialized LoRA agents and a dynamic classifier, the system achieves high robustness to environmental noise with minimal computational overhead. Future work will focus on refining the classifier to better handle overlapping speech signals and expanding the agent bank for even more granular noise specialization.

Summary updated

BibTeX
@inproceedings{simic2025adapter,
  author = {Christopher Simic and Korbinian Riedhammer and Tobias Bocklet},
  title = {{Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889566},
}
Figure