Abstract
Video Anomaly Detection (VAD) often struggles with redundant visual background noise when identifying rare abnormal events. This paper introduces GMFVAD, a weakly supervised framework that fuses fine-grained visual features with dense text captions to filter out this noise. By leveraging multi-modal data to focus on semantically important actions, the model achieves state-of-the-art performance across several major VAD benchmarks including UCSD-Ped2, ShanghaiTech, and UCF-Crime.
Introduction
Detecting anomalies in continuous surveillance video is a "needle in a haystack" problem. Due to the high cost of frame-by-frame annotation, researchers typically employ weakly supervised learning, where models are trained using video-level labels indicating the presence of an anomaly without specifying its exact timing.
Traditional VAD models rely on spatio-temporal visual features. However, surveillance footage often contains massive amounts of redundant, irrelevant background information. While recent approaches have incorporated text features (e.g., video captions) for semantic context, these integrations are often coarse and fail to align text with specific video snippets. This misalignment leaves models overwhelmed by visual redundancy, making it difficult to pinpoint actual anomalies.
Methodology
The proposed GMFVAD (Grained Multi-modal Feature for Video Anomaly Detection) architecture aims to better align text and visual data through several key components.
Grained Visual Features
Instead of using raw visual features directly, GMFVAD processes features through a Glance-Focus Network. The Glance Block learns the general representation of "normal" scenes, while the Focus Block utilizes self-attention to highlight correlations between adjacent video clips, isolating snippets likely to contain anomalies.
Text Features and Multi-modal Fusion
Dense video captions are generated using a pre-trained SwinBERT model, followed by SimCSE to create rich sentence embeddings. These text features are then fused with the grained visual features. Both the original visual features and the fused multi-modal features are processed by a Multi-scale Temporal Network (MTN) to capture dependencies across various temporal scales.
Training with Top-k MIL
The model is trained using a Multiple Instance Learning (MIL) approach. It calculates loss based on the top-$k$ snippets with the highest feature magnitudes, enabling the system to distinguish normal from abnormal behavior without requiring frame-level labels.
Experimental Evaluation
The model was evaluated on four standard VAD benchmarks:
- UCSD-Ped2: A small dataset of pedestrian walkways.
- ShanghaiTech: A medium-scale dataset of street surveillance.
- UCF-Crime: A large-scale dataset (128 hours) containing 13 types of real-world crime-related anomalies.
- XD-Violence: A massive dataset (217 hours) of violent events. For fair comparison, only the video modality was used.
Key Results
GMFVAD achieved state-of-the-art Area Under the Curve (AUC) scores: 98.85% on UCSD-Ped2, 98.23% on ShanghaiTech, and 87.22% on UCF-Crime. On the XD-Violence dataset, it reached a competitive Average Precision (AP) of 81.52%, outperforming all visual-only baselines. Ablation studies confirmed that the multi-modal approach improved performance over the baseline by 3.59% on ShanghaiTech and 4.97% on UCF-Crime.
Discussion and Insights
A primary insight is that text acts as a redundancy filter. By fusing semantic text captions with visual features, the model attenuates background noise and focuses on meaningful actions. Interestingly, modality importance varies with scene complexity: text features provide a significant boost in simpler scenes (ShanghaiTech), whereas fine-grained visual features are more critical in complex environments (UCF-Crime). Furthermore, the use of the Glance-Focus network ensures that text is aligned with the most relevant portions of the video, proving superior to coarse integration methods.
Conclusion
GMFVAD demonstrates the power of grained multi-modal fusion in video anomaly detection. While the model shows strong performance, the reliance on heavy pre-trained models like SwinBERT introduces computational overhead. Future work could involve incorporating additional modalities, such as audio, and further refining text-visual alignment in highly complex scenes.
Summary updated