Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning

Xujian Zhao, Yixin Wang, Peiquan JinAAAI 2025 · DOI

Abstract

Audio-Visual Question Answering (AVQA) requires a model to understand dynamic scenes by integrating information from both visual and auditory streams. Current approaches often struggle with precise spatio-temporal alignment and static fusion strategies that fail to account for the varying importance of modalities across different questions. We propose the Audio-Visual Adaptive Fusion Network (AVAF-Net), which employs contrastive learning for robust alignment and a Question-Oriented Adaptive Fusion (QOAF) module for dynamic modality weighting. Our model achieves state-of-the-art performance on the MUSIC-AVQA dataset, with an average accuracy of 75.90%, representing a significant improvement over existing methods.

Introduction

Audio-Visual Question Answering (AVQA) is a complex multimodal task where a model must reason over a video's visual and auditory content to answer natural language questions. For instance, answering "Is the cello always playing?" requires identifying the cello visually, locating its corresponding sound in the audio track, and tracking both over time.

The Audio-Visual Question Answering (AVQA) task requires models to reason over both visual frames and audio waveforms to answer natural language queries.

Existing models face two primary hurdles: alignment gaps and static fusion. Alignment gaps occur when models fail to map audio signals to specific visual regions (spatial alignment) or specific moments in time (temporal alignment). Static fusion strategies treat audio and visual data as equally important, ignoring that some questions are inherently "audio-heavy" while others are "visual-heavy."

Methodology

AVAF-Net introduces three key innovations to overcome these challenges:

  1. Temporal Alignment Contrastive Learning (TACL): This module ensures that audio and visual streams are mapped into a shared semantic space where corresponding temporal segments are mathematically closer.
  2. Spatial Alignment Contrastive Learning (SACL): SACL uses contrastive loss to highlight visual regions that correlate strongly with the audio signal and the question's keywords, effectively solving the "who is making that sound?" problem.
  3. Question-Oriented Adaptive Fusion (QOAF): This module dynamically generates weights for audio and visual features based on the question. If a question pertains to sound, the audio weight increases; if it concerns an object's location, the visual weight takes precedence.
AVAF-Net utilizes contrastive learning for spatial and temporal alignment, allowing the model to focus on relevant visual regions and audio segments based on the question.

The model leverages CLIP for visual and text features and VGGish for audio features, providing strong pre-trained representations for the AVQA task.

Experimental Results

AVAF-Net was evaluated on the MUSIC-AVQA and MUSIC-AVQA-Real datasets. It achieved an overall accuracy of 75.90% on MUSIC-AVQA, outperforming the previous state-of-the-art model (APL) by 2.04%. The model showed significant improvements in specific categories, such as a 5.35% increase in "Temporal" questions and a 3.09% increase in visual "Counting" tasks. On the MUSIC-AVQA-Real dataset, AVAF-Net demonstrated a maximum improvement of 15.90% in average accuracy compared to baseline GRU-based architectures. Ablation studies confirmed that removing the QOAF module caused accuracy to drop to 72.78%, highlighting its critical role in the system's success.

Conclusion

AVAF-Net sets a new benchmark for AVQA by integrating contrastive alignment with dynamic, question-aware fusion. By allowing the question to guide the flow of multimodal information, the model avoids being distracted by irrelevant noise. Future work will explore fine-tuning the pre-trained backbones and testing the model's generalization in more complex, non-musical environments.

Summary updated

BibTeX
@inproceedings{zhao2025audio,
  author = {Xujian Zhao and Yixin Wang and Peiquan Jin},
  title = {{Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i10.33138},
}
Figure