Abstract
In recent years, audio-visual event localization has attracted much attention. Its purpose is to detect the segment containing audio-visual events and recognize the event category from untrimmed videos. Existing methods use audio-guided visual attention to lead the model to pay attention to the spatial area of the ongoing event, devoting to the correlation between audio and visual information but ignoring the correlation between audio and spatial motion. We propose a past and future motion extraction (pf-ME) module to mine the visual motion from videos, embedded into the past and future motion guided network (PFAGN), and a motion guided audio attention (MGAA) module to achieve focusing on the information related to interesting events in the audio modality through the past and future visual motion. We choose AVE as the experimental verification dataset, and the experiments show that our method outperforms the state-of-the-arts in both supervised and weakly-supervised settings.
Introduction
Audio-Visual Event (AVE) localization is a challenging task in computer vision that requires a system to identify the precise temporal boundaries of an event in an untrimmed video where the event is both audible and visible. For instance, in a video of a car race, the system must distinguish between the ambient noise of the crowd (background) and the specific sound and sight of a car driving past.
Current state-of-the-art methods often rely on audio-guided visual attention, using sound to pinpoint where to look in a frame. However, these methods frequently overlook the dynamic nature of events. They focus on static spatial features rather than the motion that characterizes the event. Furthermore, audio signals are often contaminated with background noise that does not correlate with the visual action, making localization difficult.
Methodology: The PFAGN Framework
To address these limitations, the researchers developed the Past and Future Motion Guided Network (PFAGN). This framework introduces two innovative modules designed to leverage temporal dynamics.
1. Past and Future Motion Extraction (pf-ME)
Unlike traditional methods that might use computationally heavy optical flow, the pf-ME module is a lightweight alternative. It extracts motion by calculating the differences between the first frame of a current segment and the first frames of its immediate past and future neighbors. By considering both directions, the model captures the full trajectory of an action, providing a more comprehensive context than looking at a single segment in isolation.
2. Motion Guided Audio Attention (MGAA)
The core innovation of this research is the MGAA module. It uses the extracted visual motion to "purify" the audio signal.
- Temporal Attention: By comparing visual motion with audio intensity, the model can determine if a sound is part of the event. High audio combined with high visual motion indicates an event, while high audio with low visual motion suggests irrelevant background noise.
- Channel Attention: This component enhances specific audio features that are consistent with the visual motion, effectively filtering out noise before the final classification step.
Experimental Results
The model was rigorously tested on the AVE dataset, which contains over 4,000 videos across 28 categories, including musical instruments, animals, and vehicles. The PFAGN model achieved significant improvements over existing methods:
- Supervised Learning: The model reached 78.5% accuracy, setting a new benchmark by surpassing the previous state-of-the-art (PSP) by 0.7%.
- Weakly-Supervised Learning: It achieved 73.5% accuracy, demonstrating robustness even when precise temporal labels are unavailable during training.
Ablation studies confirmed that the inclusion of "past" motion is critical. Removing this component led to a noticeable drop in performance, highlighting that understanding where an object was is just as important as knowing where it is going for accurate event localization.
Conclusion
The PFAGN framework demonstrates that visual motion is a powerful tool for refining audio features in multi-modal tasks. By moving beyond static spatial analysis and incorporating temporal context from the past and future, the model achieves superior localization accuracy. Future work may involve integrating spatial audio and multi-microphone setups to further enhance the system's ability to isolate specific sound sources in complex, crowded environments.
Summary updated