Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing

Shentong Mo, Yapeng TianNeurIPS 2022 · OpenReview

Overview

The Multi-modal Grouping Network (MGN) introduces an explicit grouping mechanism to solve the "where and in what modality" problem in weakly-supervised video parsing. By using learnable category tokens to anchor audio and visual features, MGN achieves state-of-the-art performance while being 50% more lightweight than previous baselines.

Introduction: The Challenge of Weak Supervision

In Audio-Visual Video Parsing (AVVP), the goal is to analyze a video and determine exactly when an event happens and how it is perceived (is it just heard, just seen, or both?).

The primary challenge is that most models are trained under weak supervision: they only know that a specific event (e.g., a "dog" barking) occurs somewhere in a 10-second clip, but lack specific timestamps or modality labels. Previous architectures, such as the Hybrid Attention Network (HAN), struggled with this "modality and temporal uncertainty." Lacking a mechanism to explicitly group related segments, they often produced false positives—predicting the same event category for unrelated parts of the video or failing to align the audio and visual streams correctly.

Methods: Explicit Semantic Grouping

To address these uncertainties, the authors propose MGN, which shifts away from generic attention mechanisms toward explicit semantic grouping. The architecture consists of two main stages:

Architecture of the Multi-modal Grouping Network (MGN)
  1. Class-aware Unimodal Grouping (CUG): Instead of processing all video frames uniformly, the model utilizes a set of learnable categorical embedding tokens (one for each class, such as "Dog" or "Car"). These tokens act as anchors that pull in relevant features from their respective modality (audio or visual). This forces the model to learn compact, discriminative representations for each category.
  2. Modality-aware Cross-modal Grouping (MCG): After grouping features within each modality, MGN employs a cross-attention layer to share information between the audio and visual groups. This ensures that if an event is both seen and heard, the representations are aligned to match the video-level target.

The model is trained using a combination of a standard Multiple Instance Learning (MIL) loss and a novel class-constrained loss that ensures the learnable tokens accurately represent their assigned categories.

Experimental Setup

The model was evaluated on the Look, Listen and Parse (LLP) Dataset, the standard benchmark for this task. The dataset contains 11,849 YouTube clips (10 seconds each) spanning 25 event categories. Training relies solely on video-level labels, while testing utilizes fine-grained segment-level annotations to verify temporal and modal accuracy.

Key Insights and Results

Grouping as a Regularizer

By forcing the network to group features into specific category tokens, the model naturally filters out noise. This leads to much more compact clusters in the embedding space compared to previous methods.

Reducing False Positives

One of the most significant advantages of MGN is its ability to drastically reduce false-positive predictions. By anchoring features to specific class tokens, the model is far less likely to "hallucinate" events in segments where they do not exist.

Qualitative comparison showing MGN reducing false positives compared to baselines

State-of-the-Art Performance with High Efficiency

  • Performance Gains: MGN outperformed previous state-of-the-art models (HAN and MA) across almost all metrics, achieving a 1.6% gain in Type@AV and a 1.8% gain in Event@AV scores.
  • Visual Accuracy: The model demonstrated a significant jump in visual segment-level parsing, improving the F1-score from 52.9% to 55.5%.
  • Parameter Reduction: Remarkably, MGN achieves these results using only 17 MB of parameters—just 47.2% of the size of baseline models (36 MB). This proves that a well-designed inductive bias (explicit grouping) can trump raw model size.

Limitations and Future Work

While MGN significantly improves visual and audio-visual parsing, gains in the "audio-only" modality were less dramatic, reflecting the inherent difficulty of audio parsing in this dataset. Furthermore, the authors noted that increasing the depth of transformer layers in the grouping modules decreased performance, likely due to the constraints of weak supervision.

Future directions include incorporating a small amount of segment-level supervision (semi-supervised learning) or adding more grouping stages to further refine the model's ability to handle complex, multi-event videos.

Summary updated

Abstract

The audio-visual video parsing task aims to parse a video into modality- and category-aware temporal segments. Previous work mainly focuses on weakly-supervised approaches, which learn from video-level event labels. During training, they do not know which modality perceives and meanwhile which temporal segment contains the video event. Since there is no explicit grouping in the existing frameworks, the modality and temporal uncertainties make these methods suffer from false predictions. For instance, segments in the same category could be predicted in different event classes. Learning compact and discriminative multi-modal subspaces is essential for mitigating the issue. To this end, in this paper, we propose a novel Multi-modal Grouping Network, namely MGN, for explicitly semantic-aware grouping. Specifically, MGN aggregates event-aware unimodal features through unimodal grouping in terms of learnable categorical embedding tokens. Furthermore, it leverages the cross-modal grouping for modality-aware prediction to match the video-level target. Our simple framework achieves improving results against previous baselines on weakly-supervised audio-visual video parsing. In addition, our MGN is much more lightweight, using only 47.2% of the parameters of baselines (17 MB vs. 36 MB). Code is available at https://github.com/stoneMo/MGN.
BibTeX
@inproceedings{mo2022multi,
  author = {Shentong Mo and Yapeng Tian},
  title = {{Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=zfo2LqFEVY},
}
Figure