MPN: Multimodal Parallel Network for Audio-Visual Event Localization

Jiashuo Yu, Ying Cheng, Rui Feng

Introduction

Audio-visual event localization (AVE) is the task of identifying segments in a video where an event is both audible and visible. For instance, in a video of a church bell, the model must distinguish between segments where the bell is only seen, only heard, or both. The core challenge lies in the tension between global context and local precision. While global features are necessary for classification, they often introduce "contextual noise" that blurs the exact temporal boundaries of an event.

Methods: The Multimodal Parallel Network

The authors propose the Multimodal Parallel Network (MPN), which decouples the task into two specialized, parallel subnetworks to prevent global noise from interfering with local localization.

The Multimodal Parallel Network (MPN) architecture, featuring parallel Classification and Localization subnetworks.

Parallel Architecture

  1. Classification Subnetwork (Global): This stream utilizes a Multimodal Co-attention Module (MCM). It employs self-attention to model intra-modal correlations and cross-modal attention to capture interactions between audio and video, building a high-level "gist" of the event category.
  2. Localization Subnetwork (Local): This stream uses a Multimodal Bottleneck Attention Module (MBAM). It focuses on segment-level details using Factorized Bilinear Coding (FBC) to fuse features into a joint representation, followed by channel-wise attention to highlight relevant temporal features.
  3. Local-to-Global Interaction: To integrate these streams, gating signals from the local MBAM refine the output of the global MCM, allowing the global view to benefit from sharp local details without being overwhelmed by noise.

Experimental Evaluation

The model was evaluated on the Audio-Visual Event (AVE) dataset, consisting of 4,143 videos across 28 categories. The MPN was tested in both fully supervised and weakly supervised settings.

Key Results

The MPN achieved new state-of-the-art (SOTA) results on the AVE benchmark:

  • Fully Supervised Accuracy: 77.6% (outperforming the previous best by 2.4%).
  • Weakly Supervised Accuracy: 72.0% (outperforming the previous best by 2.6%).
Qualitative comparison of event localization sharpness between MPN and the baseline AVEL method.

As shown in the qualitative results, MPN provides significantly sharper localization. While previous methods like AVEL often mislabeled the boundaries of events (such as "man speaking") as background, MPN correctly identifies the full duration by relying on unmixed local content.

Conclusion

By employing a parallel "divide and conquer" approach, MPN effectively mitigates the smearing effect of global context on local boundary detection. Future work may involve end-to-end training of the backbones and extending the model to handle multi-event scenarios.

Summary updated

Abstract

Audio-visual event localization aims to localize an event that is both audible and visible in the wild, which is a widespread audio-visual scene analysis task for unconstrained videos. To address this task, we propose a Multimodal Parallel Network (MPN), which can perceive global semantics and unmixed local information parallelly. Specifically, our MPN framework consists of a classification subnetwork to predict event categories and a localization subnetwork to predict event boundaries. The classification subnetwork is constructed by the Multimodal Co-attention Module (MCM) and obtains global contexts. The localization subnetwork consists of Multimodal Bottleneck Attention Module (MBAM), which is designed to extract fine-grained segment-level contents. Extensive experiments demonstrate that our framework achieves the state-of-the-art performance both in fully supervised and weakly supervised settings on the Audio-Visual Event (AVE) dataset.
BibTeX
@inproceedings{yu2021mpn,
  author = {Jiashuo Yu and Ying Cheng and Rui Feng},
  title = {{MPN: Multimodal Parallel Network for Audio-Visual Event Localization}},
  booktitle = {ICME},
  year = {2021},
  url = {https://openreview.net/forum?id=GGCX8nAPQU6},
}
Figure