Visual-Aware Speech Recognition for Noisy Scenarios

Balaji Darur, Karan SinglaEMNLP 2025

TL;DR

This paper introduces a novel audio-visual speech recognition (AVSR) system that leverages environmental visual cues to significantly improve speech transcription accuracy in noisy conditions. By creating a scalable pipeline for generating audio-visual datasets and employing a finetuning method that integrates pretrained audio and visual encoders, the system demonstrates robust performance even when only audio is available during inference.

What Problem Does This Paper Address?

Traditional Automatic Speech Recognition (ASR) systems struggle in noisy environments, where background sounds can easily corrupt the audio signal and lead to transcription errors. While Audio-Visual Speech Recognition (AVSR) models have shown promise by incorporating visual information, many existing approaches either rely heavily on clear lip movements (requiring the speaker to be visible and well-lit) or struggle with scalability due to the need for specialized, manually curated datasets. This paper addresses the challenge of improving ASR robustness in noisy scenarios by exploiting broader environmental visual cues, similar to how humans naturally use context to understand speech, and proposes a scalable method to achieve this.

Methods

The core of the proposed method is a late-fusion architecture that combines pretrained audio and visual encoders. Specifically:

  • Audio Encoder: A Conformer-based E2E ASR encoder is used to extract audio embeddings from noisy speech. This encoder is frozen during the AVSR finetuning but can be adapted with trainable adapters.
  • Visual Encoder: CLIP's ViT-L/14 image encoder is used to extract visual features. This encoder is also frozen.
  • Cross-Modal Attention: The audio and visual embeddings are projected into a shared space and then processed through a standard transformer encoder block with Multi-Head Self-Attention. This allows for cross-modal interaction, enabling the model to learn representations that are aware of both audio and visual contexts.
  • Noise Label Prediction: The model is trained to jointly predict both the speech transcription and the noise label associated with the audio. This is achieved by appending a special token representing the noise label to the end of the transcript and using a CTC loss.
  • Scalable Dataset Creation: A key innovation is a pipeline for automatically generating audio-visual datasets. This pipeline involves filtering AudioSet for videos with specific noise labels, then mixing these noisy audio segments with clean speech from the People's Speech dataset at varying noise ratios. This eliminates the need for manually collected, noise-specific datasets.

Datasets

  • AudioSet: Used for filtering videos with specific noise labels.
  • People's Speech: A large-scale English ASR dataset used for finetuning the model. The authors leverage its clean speech subset.
  • VANS (Visual-Aware Noisy Speech) Dataset: A newly introduced dataset created using the authors' proposed pipeline. It contains approximately 28K samples (75 hours of training data) with audio-visual pairs where visual cues correlate to noise sources in the audio.

Key Insights and Ideas

  • Environmental Visual Cues are Powerful: The paper demonstrates that broader environmental visual context, not just lip movements, can significantly enhance speech recognition in noisy conditions. This aligns with human cognitive abilities.
  • Scalable Dataset Generation is Crucial: The proposed pipeline for creating audio-visual datasets by mixing existing resources is a significant contribution, making AVSR research more accessible and scalable.
  • Joint Noise and Speech Prediction: Training the model to predict noise labels alongside speech transcripts provides a richer context that benefits transcription accuracy.
  • Visual Guidance for Noise Invariance: Visual cues act as a "cognition enhancer," helping the model learn noise-invariant representations, leading to improved robustness.
  • Inference Flexibility: The model can perform well even when only audio is available during inference, suggesting that the visual training helps develop a more generalized understanding of acoustic environments.

Key Results

  • Improved Transcription Accuracy: The AV-UNI-SNR model shows significant gains in transcription accuracy compared to audio-only models in noisy scenarios (e.g., achieving a WER of 20.71 at 10dB SNR compared to 26.99 for Conformer-CTC).
  • Effective Noise Label Prediction: The model achieves high accuracy in predicting noise labels, with categories like "Blender" and "Toilet flush" showing particularly good performance due to their distinct visual characteristics.
  • Robustness Across SNRs: The AV-UNI-SNR model generalizes well across various Signal-to-Noise Ratio (SNR) levels, outperforming individual models trained at fixed SNRs, especially in lower SNR conditions.
  • Parameter Efficiency: The proposed AV-UNI-SNR model achieves competitive or superior performance compared to larger, audio-only models like Whisper Medium/Large V3 and Gemini, demonstrating its parameter efficiency.
  • Out-of-Domain Performance: The model maintains its performance advantage even when tested on out-of-domain LibriSpeech data, confirming its robustness.

Limitations and Future Work

  • Limitations:
    • The scalability of the approach relies on the quality of noise-to-video correlations in datasets like AudioSet, which are manually curated and may not capture all real-world complexities.
  • Using visual inputs at inference incurs computational overhead from the CLIP encoder, though the authors note this is a trade-off for maximum accuracy.
  • Future Work:
    • Extend the approach to incorporate more than just noise labels, including other visual cues and related events as tags.
  • Scale up the dataset generation pipeline to create over 4000 hours of data.
  • Explore additional pretrained speech and visual encoder checkpoints.
  • Investigate the potential for models trained with visual cues to achieve superior performance with audio-only inputs at test time compared to audio-only trained models.

Summary updated

BibTeX
@inproceedings{darur2025visual,
  author = {Balaji Darur and Karan Singla},
  title = {{Visual-Aware Speech Recognition for Noisy Scenarios}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure