Introduction
Automatic Speech Recognition (ASR) has reached impressive levels of accuracy in clean environments. However, performance often degrades significantly in "the wild" due to background noise such as music, overlapping speech, or environmental sounds. Audio-Visual Speech Recognition (AVSR) addresses this by integrating visual cues—specifically lip movements—to supplement the audio signal.
While effective, state-of-the-art AVSR models are typically trained from scratch, requiring massive computational resources and ignoring the robust pre-training available in large-scale ASR models like OpenAI’s Whisper. This research proposes a modular, parameter-efficient extension that adapts pre-trained ASR models for robust AVSR performance across diverse noise conditions.
Methodology
The proposed architecture utilizes a "multi-agent" approach to handle varying noise scenarios dynamically. The system consists of three primary components integrated with a frozen pre-trained Whisper model:
- AV Fusion Module: A lightweight module added upstream of the encoder that extracts visual features and fuses them with audio features using cross-attention.
- LoRA Adapters: Instead of full fine-tuning, Low-Rank Adaptation (LoRA) layers are inserted into the Whisper transformer blocks. Only these small adapter layers are updated during training.
- Multi-Agent Selection: The system maintains a bank of specialized "adapter-sets." A ResNet-based noise-scenario classifier analyzes the input audio to select the most appropriate adapter-set (e.g., specialized for babble noise or high-noise levels) for the current environment.
Experimental Setup
The models were evaluated on the LRS3-TED benchmark (430 hours) after self-supervised pre-training on VoxCeleb2 (1,700 hours). To simulate real-world conditions, speech data was synthetically contaminated using the Musan corpus, covering music, natural sounds, and "babble" (multi-speaker) noise at various Signal-to-Noise Ratios (SNR).
Key Results and Discussion
The multi-agent approach demonstrates significant improvements in both efficiency and performance. By specializing adapters for specific noise levels (SNR) or categories, the model achieves results comparable to fully fine-tuned state-of-the-art models while using a fraction of the trainable parameters.
Parameter Efficiency
As shown in the results table, the "small" version of the LoRA-AVSR model achieves performance competitive with AV-HuBERT and AV-Fusion while utilizing 79.3% to 89.1% fewer trainable parameters. For instance, the base LoRA-AVSR requires only 18M trainable parameters compared to the 87M required by the AV-Fusion base model.
Robustness to Noise
The specialization strategy proves particularly effective in high-noise scenarios (-10dB SNR). The data indicates that specializing adapters based on the Signal-to-Noise Ratio (SNR) is generally more effective than specializing by noise category. In the difficult "Sidespeaker" scenario—where background speech competes with the primary speaker—the multi-agent model outperformed the fully fine-tuned baseline at the lowest SNR levels.
Conclusion
This research demonstrates that a modular, adapter-based approach can effectively upgrade pre-trained ASR models for complex audio-visual tasks. By employing a bank of specialized LoRA agents and a dynamic classifier, the system achieves high robustness to environmental noise with minimal computational overhead. Future work will focus on refining the classifier to better handle overlapping speech signals and expanding the agent bank for even more granular noise specialization.
Summary updated