Abstract
We focus on the audio-visual video parsing (AVVP) problem that involves detecting audio and visual event labels with temporal boundaries. The task is especially challenging since it is weakly supervised with only event labels available as a bag of labels for each video. An existing state-of-the-art model for AVVP uses a hybrid attention network (HAN) to generate cross-modal features for both audio and visual modalities, and an attentive pooling module that aggregates predicted audio and visual segment-level event probabilities to yield video-level event probabilities. We provide a detailed analysis of modality bias in the existing HAN architecture, where a modality is completely ignored during prediction. We also propose a variant of feature aggregation in HAN that leads to an absolute gain in F-scores of about 2% and 1.6% for visual and audio-visual events at both segment-level and event-level, in comparison to the existing HAN model.
Introduction
Audio-Visual Video Parsing (AVVP) is a complex task requiring models to identify both the nature and temporal boundaries of events across two modalities: sight and sound. Unlike standard classification, AVVP must distinguish between purely audio events (e.g., hearing a car), purely visual events (e.g., seeing a car), and synchronized audio-visual events.
A significant hurdle in AVVP is the reliance on weak supervision. Training datasets typically provide only a "bag of labels" for a video clip without precise timestamps. While current state-of-the-art models like the Hybrid Attention Network (HAN) use Multiple Instance Learning (MIL) and attention to bridge this gap, they often fall victim to modality bias. This occurs when a model learns spurious correlations in one modality and effectively "shuts off" the other, leading to poor generalization.
Methods
The researchers investigated the architecture of the Hybrid Attention Network (HAN) to understand and mitigate this bias.
Asymmetric Feature Aggregation
Standard HAN models treat audio and video symmetrically, applying both self-attention and cross-attention to both modalities. The authors propose an asymmetric variant, $A_{cross} + V_{self}$:
- Audio ($A_{cross}$): Uses cross-attention to incorporate visual information. This helps the model disambiguate similar sounds (e.g., a blender vs. a vacuum) by "looking" at the scene.
- Video ($V_{self}$): Uses only self-attention. By preventing the visual branch from "listening" to the audio, the model avoids injecting noise from off-camera background sounds (like a distant siren) into visual feature representations.
Modality Bias Analysis
The study also examines how training dynamics, specifically label smoothing, influence modality preference. They found that smoothing labels for only one modality can inadvertently force the model's loss function to prioritize the "sharper," unsmoothed modality, thereby reinforcing bias.
Datasets
Experiments were conducted on the Look, Listen, and Parse (LLP) dataset:
- Scale: 11,849 YouTube clips (~32.9 hours) across 25 categories.
- Labels: 10-second clips with video-level labels for training and second-level ground truth for evaluation.
- Features: Visual features were extracted via ResNet152 and 3D ResNet; audio features were processed using VGGish.
Key Insights
- The Trap of Symmetry: While multi-modal learning often strives for equal interaction, this research suggests that because visual signals are typically more localized and "cleaner" than audio, forcing visual features to attend to audio can introduce more distractive noise than useful context.
- Label Smoothing Paradox: Label smoothing, a common regularization technique, can be a double-edged sword. If applied unevenly across modalities, it can act as a catalyst for modality bias by making one signal appear less "certain" to the optimizer.
- Background Noise as a Distractor: Cross-attention in the visual branch can lead to incorrect associations, such as linking the pixels of a visible cat to the sound of a background siren.
Key Results
The proposed asymmetric $A_{cross} + V_{self}$ model demonstrated superior performance and robustness.
- Performance Gains: The model achieved an absolute gain of 2% in F-scores for visual events and 1.6% for audio-visual events at both segment and event levels compared to the baseline.
- Stability: The asymmetric model was significantly less sensitive to hyperparameter tuning, showing much lower performance variance (0.35) compared to the baseline (0.95) across different label smoothing settings.
Conclusion
This work highlights that architectural symmetry in multi-modal models can be counterproductive. By introducing strategic asymmetry, the researchers improved AVVP performance and identified how standard techniques like label smoothing can exacerbate modality bias. Future research should focus on developing modality-aware loss functions that can automatically detect and correct these biases during training, reducing the need for manual architectural adjustments.
Summary updated