Introduction
Audio-Visual Question Answering (AVQA) represents a significant frontier in artificial intelligence, requiring models to synthesize information from video frames, audio tracks, and natural language queries. However, the efficacy of current benchmarks, particularly the widely used MUSIC-AVQA dataset, has come under scrutiny. Researchers have identified a pervasive "cheating" phenomenon where models exploit statistical shortcuts—language priors—rather than performing genuine cross-modal reasoning. For instance, in the original dataset, the answer to "Is this sound from the instrument in the video?" is "Yes" over 90% of the time, allowing models to achieve high accuracy by simply guessing "Yes" without analyzing the audio-visual content.
Addressing Data Bias: MUSIC-AVQA v2.0
To rectify these systemic biases, the authors introduced MUSIC-AVQA v2.0. This enhanced dataset was meticulously engineered to force models to "look and listen" before responding. The team expanded the dataset with 1,230 new real-world videos and 8.1k QA pairs, specifically targeting "minority" answers to flatten the distribution.
Key balancing strategies included:
- Audio Swapping: To balance binary "Yes/No" questions, the researchers created "No" samples by replacing a video's original audio with a different instrument from the same family (e.g., swapping a violin for a cello).
- Spatial Augmentation: Techniques like horizontal flipping were employed to balance spatial questions, ensuring that answers like "Left" or "Right" were equally probable.
The LAST-Att Model
Alongside the new dataset, the authors proposed LAST-Att, a novel baseline model designed for fine-grained audio-visual grounding. Unlike previous models that often collapse audio data into a single average vector, LAST-Att preserves spatial and semantic details through two primary innovations:
- AST Branch: Integration of a pre-trained Audio-Spectrogram-Transformer to capture rich semantic audio features.
- Cross-Modal Pixel-Wise Attention: This mechanism performs direct matching between audio spectrogram pixels and visual frame regions, allowing the model to precisely correlate specific sounds with their visual sources.
Key Results and Insights
The transition to a balanced dataset revealed the limitations of "lazy" AI models. When tested on MUSIC-AVQA v2.0, the LAST-Att model outperformed existing state-of-the-art architectures (such as LAVISH and AVST) by over 2% in total accuracy. More importantly, in a "Contrasting Binary QA" test—where models must answer opposite questions for similar videos—LAST-Att achieved 58.86% accuracy, significantly surpassing the 52.4% of previous baselines. This suggests that the model is indeed relying more on actual audio-visual evidence.
Conclusion and Future Work
The development of MUSIC-AVQA v2.0 provides a more rigorous and "honest" benchmark for the field. While it successfully mitigates many biases, challenges remain in categories like counting and existential questions due to the scarcity of complex real-world footage. Future research will likely focus on deeper temporal reasoning—understanding the precise synchronization of rhythmic patterns across multiple instruments—and moving toward end-to-end training on even larger balanced datasets.
Summary updated