Abstract
Audio-Visual Event (AVE) localization aims to identify segments in a video where an event is both audible and visible. Current methods often struggle with weak supervision and the complex temporal dynamics of real-world events. This paper introduces EDRNet, a novel framework that decomposes events into distinct stages—Initiation, Continuation, and Termination—to improve localization accuracy. By employing a U-Net-inspired temporal architecture, a "Land-Shore-Sea" loss, and a state-machine-based data augmentation strategy, EDRNet achieves state-of-the-art performance on the AVE benchmark.
Introduction
Audio-Visual Event Localization (AVEL) is a critical task in video understanding, requiring the identification of temporal boundaries for events that are simultaneously seen and heard. This task faces two primary hurdles: weak supervision, where only video-level labels are available, and the inherent temporal structure of events. Most existing models treat video segments as independent or use sequential models like LSTMs that may fail to capture the holistic "unraveling" of an event from start to finish.
Methodology
The proposed Event Decomposition Recomposition Network (EDRNet) addresses these challenges by "atomizing" events into progress checkpoints.
EDRNet Architecture
The architecture follows a U-Net-style structure applied to the temporal dimension. It consists of two primary phases:
- Event Decomposition Phase (EDP): Stacked Temporal Convolutional Networks (TCNs) progressively downsample the video sequence into high-level Event Progress Checkpoints (EPCs), capturing global context.
- Event Recomposition Phase (ERP): Transpose convolutions upsample these features back to the original temporal resolution, enabling precise segment-level localization.
Modality fusion between audio and visual streams occurs at both the compressed global level and the refined segment level, facilitating "Audio-Guided Visual Attention" implicitly.
State Machine Based Video Fusion (SMBVF)
To mitigate data scarcity, the authors developed SMBVF. By modeling events as transitions between states (Initiation $\rightarrow$ Continuation $\rightarrow$ Termination), the system can synthesize new training samples by recombining stages from different videos of the same category, ensuring semantic consistency in the augmented data.
Land-Shore-Sea (LSS) Loss
The LSS loss uses a geographic metaphor to refine feature space. "Land" represents foreground event segments, "Sea" represents background, and "Shore" represents transition boundaries. The loss function encourages "Shore" features to align with "Land" while remaining distinct from "Sea," sharpening the model's ability to detect event boundaries.
Bag to Instance Label Correction (B2ILC)
In weakly supervised settings, B2ILC corrects misclassifications between similar categories (e.g., "Truck" vs. "Bus") using a majority-voting heuristic within high-confidence clusters, improving the reliability of segment-level predictions.
Results
EDRNet was evaluated on the AVE dataset (4,143 videos, 28 categories). It established new state-of-the-art results:
- Supervised Learning (SEL): 77.93% accuracy on the original AVE dataset, a 1.1% improvement over previous methods.
- Weakly Supervised Learning (WSEL): 73.86% accuracy, representing a 1.0% gain.
Ablation studies confirmed the effectiveness of individual components, with SMBVF alone contributing a 0.83% accuracy boost. Qualitative analysis via Class Activation Maps (CAMs) demonstrated that the model successfully focuses on relevant visual regions based on audio cues.
Conclusion
EDRNet demonstrates that modeling the temporal "atomization" of events is a powerful approach for audio-visual localization. By combining a hierarchical temporal architecture with innovative loss functions and augmentation strategies, the framework effectively captures the complex dynamics of real-world events. Future work may explore more granular event checkpoints and the optimization of TCNs for real-time deployment.
Summary updated