Introduction
Audio-visual event localization (AVE) is the task of identifying segments in a video where an event is both audible and visible. For instance, in a video of a church bell, the model must distinguish between segments where the bell is only seen, only heard, or both. The core challenge lies in the tension between global context and local precision. While global features are necessary for classification, they often introduce "contextual noise" that blurs the exact temporal boundaries of an event.
Methods: The Multimodal Parallel Network
The authors propose the Multimodal Parallel Network (MPN), which decouples the task into two specialized, parallel subnetworks to prevent global noise from interfering with local localization.
Parallel Architecture
- Classification Subnetwork (Global): This stream utilizes a Multimodal Co-attention Module (MCM). It employs self-attention to model intra-modal correlations and cross-modal attention to capture interactions between audio and video, building a high-level "gist" of the event category.
- Localization Subnetwork (Local): This stream uses a Multimodal Bottleneck Attention Module (MBAM). It focuses on segment-level details using Factorized Bilinear Coding (FBC) to fuse features into a joint representation, followed by channel-wise attention to highlight relevant temporal features.
- Local-to-Global Interaction: To integrate these streams, gating signals from the local MBAM refine the output of the global MCM, allowing the global view to benefit from sharp local details without being overwhelmed by noise.
Experimental Evaluation
The model was evaluated on the Audio-Visual Event (AVE) dataset, consisting of 4,143 videos across 28 categories. The MPN was tested in both fully supervised and weakly supervised settings.
Key Results
The MPN achieved new state-of-the-art (SOTA) results on the AVE benchmark:
- Fully Supervised Accuracy: 77.6% (outperforming the previous best by 2.4%).
- Weakly Supervised Accuracy: 72.0% (outperforming the previous best by 2.6%).
As shown in the qualitative results, MPN provides significantly sharper localization. While previous methods like AVEL often mislabeled the boundaries of events (such as "man speaking") as background, MPN correctly identifies the full duration by relying on unmixed local content.
Conclusion
By employing a parallel "divide and conquer" approach, MPN effectively mitigates the smearing effect of global context on local boundary detection. Future work may involve end-to-end training of the backbones and extending the model to handle multi-event scenarios.
Summary updated