AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization

Tanvir Mahmud, Diana Marculescu

Introduction

Audio-Visual Event (AVE) localization is the task of identifying the specific temporal segments in a video where an action is both seen and heard simultaneously. While identifying events in single modalities is well-studied, the synchronization of sight and sound presents unique challenges. AVE-CLIP introduces a novel framework that leverages multi-modal pre-training and a multi-window temporal transformer to achieve a new state-of-the-art accuracy of 83.7% on the standard AVE benchmark—a significant 5.9% improvement over previous methods.

The Challenge of Audio-Visual Event Localization

The authors identify three primary hurdles that hinder current localization models:

  1. Ineffective Feature Alignment: Standard models often use encoders trained independently on images (e.g., ImageNet) or audio (e.g., AudioSet). These "expert" models fail to capture the inherent correlation—the "glue"—between the two modalities.
  2. The Timescale Dilemma: Events vary significantly in duration. A fleeting sound like a door slam requires local precision, while a long speech requires global context. Balancing these needs is difficult for fixed-window models.
  3. Background Noise: Distinguishing the target event from complex auditory or visual background noise, particularly at the boundaries of an event, remains a major source of error.

The AVE-CLIP Framework

To address these challenges, AVE-CLIP employs a three-stage approach:

1. Contrastive Fine-tuning

Instead of generic encoders, the framework utilizes AudioCLIP, an extension of OpenAI's CLIP that incorporates audio. By fine-tuning these encoders on audio-image pairs from the target dataset, the model ensures that visual and auditory features are aligned in a shared embedding space before localization begins.

2. Multi-window Temporal Fusion (MWTF)

The core innovation is the MWTF module. Rather than analyzing the video at a single scale, the model uses a transformer that operates across multiple temporal "windows" (e.g., 2s, 5s, and 10s). This multi-scale approach allows the model to capture sharp local transitions while maintaining a broad understanding of the overall scene.

3. Event-Guided Temporal Attention (EGTA) and Refinement

To mitigate background noise, a supervised attention mechanism (EGTA) is introduced to sharpen the contrast between event segments and background frames. Finally, a temporal smoothing algorithm filters out "flickering" predictions, ensuring that localized events are continuous and physically plausible.

Key Results and Qualitative Analysis

AVE-CLIP achieves a state-of-the-art accuracy of 83.7% on the AVE Dataset, which contains 4,143 video clips across 28 categories. Ablation studies confirm that contrastive fine-tuning contributes a 2.6% accuracy boost, and the multi-window approach significantly outperforms single-window baselines.

Visualization of Audio-Visual Event (AVE) localization, distinguishing 'Man Speaking' events from background noise across a temporal sequence.

As shown in the visualization above, AVE-CLIP excels at identifying the exact boundaries of an event. While older models often "hallucinate" events into background frames or struggle with transitions, AVE-CLIP's multi-scale attention allows for precise temporal segmentation.

Conclusion

The success of AVE-CLIP underscores the importance of multi-modal pre-training and multi-scale temporal analysis in video understanding. By aligning features early and processing them across various timescales, the model effectively overcomes the traditional "timescale dilemma" and noise sensitivity of previous architectures. Future work may explore the scalability of the multi-window approach and its application to related tasks like audio-visual saliency or automated video summarization.

Summary updated

Abstract

An audio-visual event (AVE) is denoted by the correspondence of the visual and auditory signals in a video segment. Precise localization of the AVEs is very challenging since it demands effective multi-modal feature correspondence to ground the short and long range temporal interactions. Existing approaches struggle in capturing the different scales of multi-modal interaction due to ineffective multi-modal training strategies. To overcome this limitation, we introduce AVE-CLIP, a novel framework that integrates the AudioCLIP pre-trained on large-scale audio-visual data with a multi-window temporal transformer to effectively operate on different temporal scales of video frames. Our contributions are three-fold: (1) We introduce a multi-stage training framework to incorporate AudioCLIP pre-trained with audio-image pairs into the AVE localization task on video frames through contrastive fine-tuning, effective mean video feature extraction, and multi-scale training phases. (2) We propose a multi-domain attention mechanism that operates on both temporal and feature domains over varying timescales to fuse the local and global feature variations. (3) We introduce a temporal refining scheme with event-guided attention followed by a simple-yet-effective post processing step to handle significant variations of the background over diverse events. Our method achieves state-of-the-art performance on the publicly available AVE dataset with 5.9% mean accuracy improvement which proves its superiority over existing approaches.
BibTeX
@inproceedings{mahmud2023ave,
  author = {Tanvir Mahmud and Diana Marculescu},
  title = {{AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00513},
}
Figure