Decompose the Sounds and Pixels, Recompose the Events

Varshanth R. Rao, Md Ibrahim Khalil, Haoda Li, Peng Dai, Juwei Lu

Abstract

In this paper, we propose a framework centering around a novel architecture called the Event Decomposition Recomposition Network (EDRNet) to tackle the Audio-Visual Event (AVE) localization problem in the supervised and weakly supervised settings. AVEs in the real world exhibit common unravelling patterns (termed as Event Progress Checkpoints (EPC)), which humans can perceive through the cooperation of their auditory and visual senses. Unlike earlier methods which attempt to recognize entire event sequences, the EDRNet models EPCs and inter-EPC relationships using stacked temporal convolutions. Based on the postulation that EPC representations are theoretically consistent for an event category, we introduce the State Machine Based Video Fusion, a novel augmentation technique that blends source videos using different EPC template sequences. Additionally, we design a new loss function called the Land-Shore-Sea loss to compactify continuous foreground and background representations. Lastly, to alleviate the issue of confusing events during weak supervision, we propose a prediction stabilization method called Bag to Instance Label Correction. Experiments on the AVE dataset show that our collective framework outperforms the state-of-the-art by a sizable margin.

Introduction

Audio-Visual Event (AVE) localization is the task of identifying the temporal boundaries and categories of events that are simultaneously seen and heard in a video. This task presents significant challenges, including modality noise (where one sense is clearer than the other), temporal complexity, and the frequent lack of fine-grained segment-level labels (weak supervision). Traditional approaches often rely on Recurrent Neural Networks (RNNs) like LSTMs, which can struggle with long-term dependencies and fail to explicitly model the structured "lifecycle" of an event.

Event Atomization: Progress Checkpoints

A core contribution of this work is the concept of "event atomization." Instead of treating an event as a monolithic block, we decompose it into Event Progress Checkpoints (EPCs): Initiation, Continuation, and Termination. This mirrors human perception; for instance, the way a cat begins a meow is semantically distinct from the middle of the sound or its conclusion.

Visualizing the three Event Progress Checkpoints (EPCs): Initiation, Continuation, and Termination for a cat meowing event.

By modeling these stages, the system can better generalize across different instances of the same event category, as the underlying temporal structure remains consistent.

The EDRNet Framework

The Event Decomposition Recomposition Network (EDRNet) is designed to exploit this temporal structure. Inspired by the U-Net architecture, it consists of two primary phases:

  1. Event Decomposition Phase (EDP): Uses stacked 1D temporal convolutions to summarize the video into high-level conceptual representations.
  2. Event Recomposition Phase (ERP): Employs 1D transpose convolutions to upsample these concepts back to the segment level, allowing for precise localization.
The EDRNet architecture, featuring the Event Decomposition Phase (EDP) and Event Recomposition Phase (ERP) with dual-phase modality fusion.

Crucially, EDRNet utilizes a dual-phase modality fusion strategy. Rather than merging audio and visual features only at the end, the modalities interact during both the summary and reconstruction phases, allowing them to provide mutual guidance throughout the process.

Advanced Training Strategies

To further enhance performance, we introduced three novel techniques:

  • State Machine Based Video Fusion (SMBVF): This data augmentation method uses a state machine to "stitch" together EPCs from different videos of the same category. This creates synthetic training samples that maintain realistic temporal logic, significantly boosting the model's robustness.
  • Land-Shore-Sea (LSS) Loss: This loss function treats foreground events as "Land," background as "Sea," and transitions as "Shores." It encourages the model to group similar foreground features together while pushing them away from background noise.
  • Bag to Instance Label Correction (B2ILC): In weakly supervised settings, this heuristic stabilizes predictions by correcting "jittery" segment labels based on the consensus of neighboring segments.

Experimental Results

EDRNet was evaluated on the AVE dataset, which includes 28 event categories. It achieved state-of-the-art results in both supervised and weakly supervised settings:

  • Supervised Learning (SEL): Achieved 77.93% accuracy, a significant improvement over previous benchmarks.
  • Weakly Supervised Learning (WSEL): Reached 73.86% accuracy, demonstrating the effectiveness of the B2ILC and LSS loss components.

Ablation studies confirmed that the SMBVF augmentation alone provided a 0.83% accuracy boost, proving that remixing videos based on event stages is a powerful tool for training on smaller datasets.

Conclusion

By shifting the focus from whole-event recognition to the modeling of atomic event stages, EDRNet provides a more nuanced and accurate framework for audio-visual understanding. The combination of temporal decomposition, dual-phase fusion, and topographical loss functions allows the system to navigate the complexities of real-world video data with unprecedented precision.

Summary updated

Abstract

In this paper, we propose a framework centering around a novel architecture called the Event Decomposition Recomposition Network (EDRNet) to tackle the Audio-Visual Event (AVE) localization problem in the supervised and weakly supervised settings. AVEs in the real world exhibit common unravelling patterns (termed as Event Progress Checkpoints (EPC)), which humans can perceive through the cooperation of their auditory and visual senses. Unlike earlier methods which attempt to recognize entire event sequences, the EDRNet models EPCs and inter-EPC relationships using stacked temporal convolutions. Based on the postulation that EPC representations are theoretically consistent for an event category, we introduce the State Machine Based Video Fusion, a novel augmentation technique that blends source videos using different EPC template sequences. Additionally, we design a new loss function called the Land-Shore-Sea loss to compactify continuous foreground and background representations. Lastly, to alleviate the issue of confusing events during weak supervision, we propose a prediction stabilization method called Bag to Instance Label Correction. Experiments on the AVE dataset show that our collective framework outperforms the state-of-the-art by a sizable margin.
BibTeX
@inproceedings{rao2022decompose,
  author = {Varshanth R. Rao and Md Ibrahim Khalil and Haoda Li and Peng Dai and Juwei Lu},
  title = {{Decompose the Sounds and Pixels, Recompose the Events}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i2.20111},
}
Figure