Abstract
An audio-visual event (AVE) is denoted by the correspondence of the visual and auditory signals in a video segment. Precise localization of the AVEs is very challenging since it demands effective multi-modal feature correspondence to ground the short and long range temporal interactions. Existing approaches struggle in capturing the different scales of multi-modal interaction due to ineffective multi-modal training strategies. To overcome this limitation, we introduce AVE-CLIP, a novel framework that integrates the AudioCLIP pre-trained on large-scale audio-visual data with a multi-window temporal transformer to effectively operate on different temporal scales of video frames. Our contributions are three-fold: (1) We introduce a multi-stage training framework to incorporate AudioCLIP pre-trained with audio-image pairs into the AVE localization task on video frames through contrastive fine-tuning, effective mean video feature extraction, and multi-scale training phases. (2) We propose a multi-domain attention mechanism that operates on both temporal and feature domains over varying timescales to fuse the local and global feature variations. (3) We introduce a temporal refining scheme with event-guided attention followed by a simple-yet-effective post processing step to handle significant variations of the background over diverse events. Our method achieves state-of-the-art performance on the publicly available AVE dataset with 5.9% mean accuracy improvement which proves its superiority over existing approaches.
Introduction
Audio-Visual Event (AVE) localization is the task of identifying segments in a video where an event is both audible and visible. For example, in a video of a person talking, the audio might persist throughout, but the "event" only occurs when the person is actually on screen. Current approaches face two major hurdles: ineffective pre-training on single modalities and a temporal scale mismatch where models fail to capture both global context and fine-grained local transitions.
Methodology
The AVE-CLIP framework operates in three distinct stages to address these challenges:
- Stage 1: Contrastive Fine-tuning: Instead of starting from unimodal encoders, we utilize AudioCLIP, a model pre-trained on massive audio-image-text datasets. We fine-tune these encoders on the target dataset using a contrastive loss to ensure tight alignment between audio and visual features, which provides a 2.6% accuracy boost over baseline pre-training.
- Stage 2: Feature Extraction: Aligned features are extracted, focusing on effective mean video feature extraction to bridge the gap between image-based pre-training and video-based localization.
- Stage 3: Multi-window Temporal Fusion (MWTF): This core innovation uses multiple transformer sub-modules that analyze the video through different "windows" (e.g., 2s, 5s, and 10s). This allows the model to capture both high-frequency local variations and low-frequency global context simultaneously. Research found that using independent weights for different window sizes performed better than shared weights.
Additionally, the model employs Event-Guided Temporal Refinement (EGTA), a supervised attention mechanism that helps distinguish between actual events and background noise, followed by a temporal filtering step to remove flickering predictions.
Datasets
We evaluate our method on the Audio-Visual Event (AVE) Dataset, which contains 4,143 video clips, each 10 seconds long. The dataset covers 28 diverse event categories, including musical instruments, animal sounds, vehicle noises, and human activities. Each second is labeled as either "background" or one of the event classes.
Key Results
AVE-CLIP achieves a new state-of-the-art accuracy of 83.7% on the AVE benchmark, outperforming the previous best co-attention method (PSP) by 5.9% and the leading transformer-based approach (AV-Transformer) by 6.9%.
Qualitative results show that AVE-CLIP is particularly robust in complex scenarios. For instance, in cases where an object is visible but silent (like a helicopter in the distance), the model correctly identifies that the event only begins when the sound starts, whereas previous models often misidentified silent visual frames as part of the event.
Conclusion
By combining multi-modal pre-training with a multi-scale temporal architecture, AVE-CLIP significantly improves the precision of audio-visual event localization. Future work will explore more sophisticated, learnable temporal smoothing techniques and optimization for real-time applications on edge devices.
Summary updated