Abstract
Audio-Visual Question Answering (AVQA) requires a model to understand dynamic scenes by integrating information from both visual and auditory streams. Current approaches often struggle with precise spatio-temporal alignment and static fusion strategies that fail to account for the varying importance of modalities across different questions. We propose the Audio-Visual Adaptive Fusion Network (AVAF-Net), which employs contrastive learning for robust alignment and a Question-Oriented Adaptive Fusion (QOAF) module for dynamic modality weighting. Our model achieves state-of-the-art performance on the MUSIC-AVQA dataset, with an average accuracy of 75.90%, representing a significant improvement over existing methods.
Introduction
Audio-Visual Question Answering (AVQA) is a complex multimodal task where a model must reason over a video's visual and auditory content to answer natural language questions. For instance, answering "Is the cello always playing?" requires identifying the cello visually, locating its corresponding sound in the audio track, and tracking both over time.
Existing models face two primary hurdles: alignment gaps and static fusion. Alignment gaps occur when models fail to map audio signals to specific visual regions (spatial alignment) or specific moments in time (temporal alignment). Static fusion strategies treat audio and visual data as equally important, ignoring that some questions are inherently "audio-heavy" while others are "visual-heavy."
Methodology
AVAF-Net introduces three key innovations to overcome these challenges:
- Temporal Alignment Contrastive Learning (TACL): This module ensures that audio and visual streams are mapped into a shared semantic space where corresponding temporal segments are mathematically closer.
- Spatial Alignment Contrastive Learning (SACL): SACL uses contrastive loss to highlight visual regions that correlate strongly with the audio signal and the question's keywords, effectively solving the "who is making that sound?" problem.
- Question-Oriented Adaptive Fusion (QOAF): This module dynamically generates weights for audio and visual features based on the question. If a question pertains to sound, the audio weight increases; if it concerns an object's location, the visual weight takes precedence.
The model leverages CLIP for visual and text features and VGGish for audio features, providing strong pre-trained representations for the AVQA task.
Experimental Results
AVAF-Net was evaluated on the MUSIC-AVQA and MUSIC-AVQA-Real datasets. It achieved an overall accuracy of 75.90% on MUSIC-AVQA, outperforming the previous state-of-the-art model (APL) by 2.04%. The model showed significant improvements in specific categories, such as a 5.35% increase in "Temporal" questions and a 3.09% increase in visual "Counting" tasks. On the MUSIC-AVQA-Real dataset, AVAF-Net demonstrated a maximum improvement of 15.90% in average accuracy compared to baseline GRU-based architectures. Ablation studies confirmed that removing the QOAF module caused accuracy to drop to 72.78%, highlighting its critical role in the system's success.
Conclusion
AVAF-Net sets a new benchmark for AVQA by integrating contrastive alignment with dynamic, question-aware fusion. By allowing the question to guide the flow of multimodal information, the model avoids being distracted by irrelevant noise. Future work will explore fine-tuning the pre-trained backbones and testing the model's generalization in more complex, non-musical environments.
Summary updated