Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering

Xiulong Liu, Xiulong Liu 0002, Zhikang Dong, Peng Zhang

Introduction

Audio-Visual Question Answering (AVQA) represents a significant frontier in artificial intelligence, requiring models to synthesize information from video frames, audio tracks, and natural language queries. However, the efficacy of current benchmarks, particularly the widely used MUSIC-AVQA dataset, has come under scrutiny. Researchers have identified a pervasive "cheating" phenomenon where models exploit statistical shortcuts—language priors—rather than performing genuine cross-modal reasoning. For instance, in the original dataset, the answer to "Is this sound from the instrument in the video?" is "Yes" over 90% of the time, allowing models to achieve high accuracy by simply guessing "Yes" without analyzing the audio-visual content.

Addressing Data Bias: MUSIC-AVQA v2.0

To rectify these systemic biases, the authors introduced MUSIC-AVQA v2.0. This enhanced dataset was meticulously engineered to force models to "look and listen" before responding. The team expanded the dataset with 1,230 new real-world videos and 8.1k QA pairs, specifically targeting "minority" answers to flatten the distribution.

Examples of balanced question-answering pairs in MUSIC-AVQA v2.0, illustrating binary and spatial reasoning tasks.

Key balancing strategies included:

  • Audio Swapping: To balance binary "Yes/No" questions, the researchers created "No" samples by replacing a video's original audio with a different instrument from the same family (e.g., swapping a violin for a cello).
  • Spatial Augmentation: Techniques like horizontal flipping were employed to balance spatial questions, ensuring that answers like "Left" or "Right" were equally probable.

The LAST-Att Model

Alongside the new dataset, the authors proposed LAST-Att, a novel baseline model designed for fine-grained audio-visual grounding. Unlike previous models that often collapse audio data into a single average vector, LAST-Att preserves spatial and semantic details through two primary innovations:

  1. AST Branch: Integration of a pre-trained Audio-Spectrogram-Transformer to capture rich semantic audio features.
  2. Cross-Modal Pixel-Wise Attention: This mechanism performs direct matching between audio spectrogram pixels and visual frame regions, allowing the model to precisely correlate specific sounds with their visual sources.
A temporal comparison question from the dataset, requiring the model to determine which instrument plays for a longer duration.

Key Results and Insights

The transition to a balanced dataset revealed the limitations of "lazy" AI models. When tested on MUSIC-AVQA v2.0, the LAST-Att model outperformed existing state-of-the-art architectures (such as LAVISH and AVST) by over 2% in total accuracy. More importantly, in a "Contrasting Binary QA" test—where models must answer opposite questions for similar videos—LAST-Att achieved 58.86% accuracy, significantly surpassing the 52.4% of previous baselines. This suggests that the model is indeed relying more on actual audio-visual evidence.

Conclusion and Future Work

The development of MUSIC-AVQA v2.0 provides a more rigorous and "honest" benchmark for the field. While it successfully mitigates many biases, challenges remain in categories like counting and existential questions due to the scarcity of complex real-world footage. Future research will likely focus on deeper temporal reasoning—understanding the precise synchronization of rhythmic patterns across multiple instruments—and moving toward end-to-end training on even larger balanced datasets.

Summary updated

Abstract

In recent years, there has been a growing emphasis on the intersection of audio, vision, and text modalities, driving forward the advancements in multimodal research. However, strong bias that exists in any modality can lead to the model neglecting the others. Consequently, the model's ability to effectively reason across these diverse modalities is compromised, impeding further advancement. In this paper, we meticulously review each question type from the original dataset, selecting those with pronounced answer biases. To counter these biases, we gather complementary videos and questions, ensuring that no answers have outstanding skewed distribution. In particular, for binary questions, we strive to ensure that both answers are almost uniformly spread within each question category. As a result, we construct a new dataset, named MUSIC-AVQA v2.0, which is more challenging and we believe could better foster the progress of AVQA task. Furthermore, we present a novel baseline model that delves deeper into the audio-visual-text interrelation. On MUSIC-AVQA v2.0, this model surpasses all the existing benchmarks, improving accuracy by 2% on MUSIC-AVQA v2.0, setting a new state-of-the-art performance.
BibTeX
@inproceedings{liu2024tackling,
  author = {Xiulong Liu and Xiulong Liu and Zhikang Dong and Peng Zhang},
  title = {{Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00442},
}
Figure