Decompose the Sounds and Pixels, Recompose the Events

Varshanth R. Rao, Md Ibrahim Khalil, Haoda Li, Peng Dai, Juwei Lu

Abstract

Audio-Visual Event (AVE) localization aims to identify segments in a video where an event is both audible and visible. Current methods often struggle with weak supervision and the complex temporal dynamics of real-world events. This paper introduces EDRNet, a novel framework that decomposes events into distinct stages—Initiation, Continuation, and Termination—to improve localization accuracy. By employing a U-Net-inspired temporal architecture, a "Land-Shore-Sea" loss, and a state-machine-based data augmentation strategy, EDRNet achieves state-of-the-art performance on the AVE benchmark.

Introduction

Audio-Visual Event Localization (AVEL) is a critical task in video understanding, requiring the identification of temporal boundaries for events that are simultaneously seen and heard. This task faces two primary hurdles: weak supervision, where only video-level labels are available, and the inherent temporal structure of events. Most existing models treat video segments as independent or use sequential models like LSTMs that may fail to capture the holistic "unraveling" of an event from start to finish.

Figure 1: Decomposition of an audio-visual event into Initiation, Continuation, and Termination stages.

Methodology

The proposed Event Decomposition Recomposition Network (EDRNet) addresses these challenges by "atomizing" events into progress checkpoints.

EDRNet Architecture

The architecture follows a U-Net-style structure applied to the temporal dimension. It consists of two primary phases:

  1. Event Decomposition Phase (EDP): Stacked Temporal Convolutional Networks (TCNs) progressively downsample the video sequence into high-level Event Progress Checkpoints (EPCs), capturing global context.
  2. Event Recomposition Phase (ERP): Transpose convolutions upsample these features back to the original temporal resolution, enabling precise segment-level localization.

Modality fusion between audio and visual streams occurs at both the compressed global level and the refined segment level, facilitating "Audio-Guided Visual Attention" implicitly.

Figure 2: The EDRNet architecture featuring the Event Decomposition Phase (EDP) and Event Recomposition Phase (ERP).

State Machine Based Video Fusion (SMBVF)

To mitigate data scarcity, the authors developed SMBVF. By modeling events as transitions between states (Initiation $\rightarrow$ Continuation $\rightarrow$ Termination), the system can synthesize new training samples by recombining stages from different videos of the same category, ensuring semantic consistency in the augmented data.

Land-Shore-Sea (LSS) Loss

The LSS loss uses a geographic metaphor to refine feature space. "Land" represents foreground event segments, "Sea" represents background, and "Shore" represents transition boundaries. The loss function encourages "Shore" features to align with "Land" while remaining distinct from "Sea," sharpening the model's ability to detect event boundaries.

Bag to Instance Label Correction (B2ILC)

In weakly supervised settings, B2ILC corrects misclassifications between similar categories (e.g., "Truck" vs. "Bus") using a majority-voting heuristic within high-confidence clusters, improving the reliability of segment-level predictions.

Results

EDRNet was evaluated on the AVE dataset (4,143 videos, 28 categories). It established new state-of-the-art results:

  • Supervised Learning (SEL): 77.93% accuracy on the original AVE dataset, a 1.1% improvement over previous methods.
  • Weakly Supervised Learning (WSEL): 73.86% accuracy, representing a 1.0% gain.

Ablation studies confirmed the effectiveness of individual components, with SMBVF alone contributing a 0.83% accuracy boost. Qualitative analysis via Class Activation Maps (CAMs) demonstrated that the model successfully focuses on relevant visual regions based on audio cues.

Conclusion

EDRNet demonstrates that modeling the temporal "atomization" of events is a powerful approach for audio-visual localization. By combining a hierarchical temporal architecture with innovative loss functions and augmentation strategies, the framework effectively captures the complex dynamics of real-world events. Future work may explore more granular event checkpoints and the optimization of TCNs for real-time deployment.

Summary updated

Abstract

In this paper, we propose a framework centering around a novel architecture called the Event Decomposition Recomposition Network (EDRNet) to tackle the Audio-Visual Event (AVE) localization problem in the supervised and weakly supervised settings. AVEs in the real world exhibit common unraveling patterns (termed as Event Progress Checkpoints(EPC)), which humans can perceive through the cooperation of their auditory and visual senses. Unlike earlier methods which attempt to recognize entire event sequences, the EDRNet models EPCs and inter-EPC relationships using stacked temporal convolutions. Based on the postulation that EPC representations are theoretically consistent for an event category, we introduce the State Machine Based Video Fusion, a novel augmentation technique that blends source videos using different EPC template sequences. Additionally, we design a new loss function called the Land-Shore-Sea loss to compactify continuous foreground and background representations. Lastly, to alleviate the issue of confusing events during weak supervision, we propose a prediction stabilization method called Bag to Instance Label Correction. Experiments on the AVE dataset show that our collective framework outperforms the state-of-the-art by a sizable margin.
BibTeX
@inproceedings{rao2022decompose,
  author = {Varshanth R. Rao and Md Ibrahim Khalil and Haoda Li and Peng Dai and Juwei Lu},
  title = {{Decompose the Sounds and Pixels, Recompose the Events}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i2.20111},
}
Figure