Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering

Xiulong Liu, Zhikang Dong, Peng Zhang

Abstract

In recent years, there has been a growing emphasis on the intersection of audio, vision, and text modalities, driving forward the advancements in multimodal research. However, strong bias that exists in any modality can lead to the model neglecting the others. Consequently, the model's ability to effectively reason across these diverse modalities is compromised, impeding further advancement. In this paper, we meticulously review each question type from the original dataset, selecting those with pronounced answer biases. To counter these biases, we gather complementary videos and questions, ensuring that no answers have outstanding skewed distribution. In particular, for binary questions, we strive to ensure that both answers are almost uniformly spread within each question category. As a result, we construct a new dataset, named MUSIC-AVQA v2.0, which is more challenging and we believe could better foster the progress of AVQA task. Furthermore, we present a novel baseline model that delves deeper into the audio-visual-text interrelation. On MUSIC-AVQA v2.0, this model surpasses all the existing benchmarks, improving accuracy by 2% on MUSIC-AVQA v2.0, setting a new state-of-the-art performance.

The Problem: Statistical Shortcuts in Multimodal Learning

Audio-Visual Question Answering (AVQA) is a sophisticated task requiring a model to synthesize information from both visual and auditory streams. For instance, a model might be asked to identify which specific musician in an ensemble is responsible for a particular sound. However, researchers discovered a significant flaw in the widely used MUSIC-AVQA benchmark: "answer shortcuts."

Statistical biases in the dataset allowed models to achieve high accuracy by memorizing distribution patterns rather than performing genuine reasoning. If a binary question like "Is this sound from the instrument in the video?" results in a "Yes" answer 90% of the time, a model can ignore the video entirely and still perform well. Similar skews were found in temporal questions (favoring "Simultaneously") and counting questions (favoring "1" or "2"). This paper introduces a de-biased benchmark to force models to "look" and "listen" properly.

Methods: Data Engineering and Architectural Innovation

The researchers addressed these issues through two primary contributions:

1. Dataset Balancing

The team reviewed 33 question templates across 9 categories. To mitigate bias, they collected 1,204 new real-world videos of complex musical ensembles from sources like YouTube. They generated 8,136 new QA pairs designed to create a near-uniform distribution of answers, particularly for binary and counting categories.

2. The LAST-Att Model

To establish a more robust baseline, the authors proposed the LAVISH-AST-Spatial-Temporal Attention (LAST-Att) model. This architecture features:

  • AST Branch: Integration of a pre-trained Audio-Spectrogram-Transformer (AST) for superior semantic audio capture.
  • Cross-Modal Pixel-wise Attention: Instead of using global audio features, this module computes mutual, granular attention between audio and visual spatial maps. This allows the model to precisely correlate specific sounds with specific regions in a video frame.

The MUSIC-AVQA v2.0 Dataset

The resulting MUSIC-AVQA v2.0 dataset is significantly more balanced and challenging than its predecessor. It contains a total of 9,290 videos and 45,867 questions, covering a wide array of musical instruments and ensemble configurations.

Examples of various question types and audio-visual contexts in the balanced MUSIC-AVQA v2.0 dataset.

Key Insights: Why Granularity Matters

A core finding of this research is that models trained on biased data fail when tested on balanced sets. Existing state-of-the-art models like LAVISH saw performance drops when evaluated on v2.0, confirming their reliance on dataset skews.

Furthermore, the authors emphasize that global audio features are insufficient for complex scenes. True reasoning requires pixel-wise mutual attention—where the audio "looks" at pixels and pixels "listen" to the audio—to disambiguate which instrument is active in a multi-instrumentalist setting.

Temporal and spatial reasoning tasks require the model to distinguish which instrument produces sound first, a task that fails without proper multi-modal alignment.

Key Results

  • State-of-the-Art Accuracy: The LAST-Att model achieved 75.44% total accuracy on the balanced MUSIC-AVQA v2.0 test set, outperforming previous benchmarks by over 2%.
  • Impact of De-biasing: Training models on the balanced dataset led to significant improvements in generalization. For example, the LAVISH model's accuracy improved by nearly 3% when trained on v2.0.
  • Reasoning Capability: In "Contrasting Binary QA" tests—where two videos have the same question but different answers—LAST-Att reached 58.86% accuracy, demonstrating a superior grasp of actual audio-visual relationships compared to previous models (52.4%).

Limitations and Future Work

While bias has been significantly reduced, challenges remain. "Counting" questions still skew toward smaller numbers because videos with 10+ distinct sounding instruments are rare and difficult to annotate. Future research may explore unsupervised bias correction or extend these methods to more diverse, non-musical scenarios.

Summary updated

Abstract

In recent years, there has been a growing emphasis on the intersection of audio, vision, and text modalities, driving forward the advancements in multimodal research. However, strong bias that exists in any modality can lead to the model neglecting the others. Consequently, the model's ability to effectively reason across these diverse modalities is compromised, impeding further advancement. In this paper, we meticulously review each question type from the original dataset, selecting those with pronounced answer biases. To counter these biases, we gather complementary videos and questions, ensuring that no answers have outstanding skewed distribution. In particular, for binary questions, we strive to ensure that both answers are almost uniformly spread within each question category. As a result, we construct a new dataset, named MUSIC-AVQA v2.0, which is more challenging and we believe could better foster the progress of AVQA task. Furthermore, we present a novel baseline model that delves deeper into the audio-visual-text interrelation. On MUSIC-AVQA v2.0, this model surpasses all the existing benchmarks, improving accuracy by 2% on MUSIC-AVQA v2.0, setting a new state-of-the-art performance.
BibTeX
@inproceedings{liu2024tackling,
  author = {Xiulong Liu and Zhikang Dong and Peng Zhang},
  title = {{Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00442},
}
Figure