Abstract
GMFVAD (Grained Multi-modal Feature for Video Anomaly Detection) improves surveillance monitoring by combining refined visual features with dense text captions. By using a "Glance-Focus" network to filter out redundant video data and aligning it with semantic text embeddings, the model achieves state-of-the-art performance across major anomaly detection benchmarks.
Introduction
Video Anomaly Detection (VAD) aims to automatically identify unusual events, such as crimes or accidents, in continuous surveillance footage. Most existing systems rely solely on visual features, which often contain significant "redundant information"—background noise or irrelevant movements that mask the actual anomaly.
While some recent multi-modal approaches have attempted to integrate text descriptions, they often do so in a "coarse" manner, applying general descriptions to noisy video snippets. This makes it difficult for models to align the rich semantic meaning of text with cluttered visual data. The GMFVAD framework addresses this by creating a "grained" approach that filters out visual redundancy before fusing it with text.
Methodology
The GMFVAD framework consists of two primary branches that eventually merge into a multi-scale temporal learner.
The Visual Branch: Glance-Focus Network
Instead of processing raw video features directly, the authors implement a Glance-Focus Network to refine the visual input:
- Glance Block: Utilizes a video clip-level transformer to scan the scene and identify snippets likely to contain anomalies.
- Focus Block: Employs self-attention convolutions to highlight correlations between nearby clips, effectively "zooming in" on important temporal details and reducing background redundancy.
The Text Branch: Dense Captioning
The model uses SwinBERT to generate dense captions for video snippets. These captions are converted into sentence embeddings using SimCSE (a contrastive learning method). This provides high-level semantic labels (e.g., "a person is running with a knife") that visual pixels alone might struggle to categorize.
Fusion and Temporal Learning
The refined visual features and text embeddings are fused and passed through a Multi-scale Temporal Network (MTN). This network uses pyramid dilated convolutions and self-attention to capture both short-term and long-term dependencies in the video. The model is trained using a Top-k Multiple Instance Learning (MIL) framework, focusing on snippets with the highest anomaly scores to learn from video-level labels without frame-by-frame annotations.
Key Insights and Results
The core finding of this research is that multi-modal fusion fails when the visual signal is too noisy. By refining visual features before fusion, text features can more accurately "anchor" the semantic meaning of the scene.
Ablation studies revealed that dataset complexity dictates modality importance. On simpler datasets like ShanghaiTech, text features contributed more to the performance boost. Conversely, on complex datasets like UCF-Crime, refined visual features played a more significant role.
Performance Benchmarks
GMFVAD achieved state-of-the-art (SOTA) or near-SOTA results across four standard benchmarks:
- UCF-Crime: Achieved 87.22% AUC, significantly beating visual-only methods like RTFM (84.30%).
- ShanghaiTech: Reached 98.23% AUC, a notable improvement over the baseline.
- XD-Violence: Achieved 81.52% AP (Average Precision).
- Ablation Success: Adding both visual refinement and text features improved the baseline by 3.59% on ShanghaiTech and 4.97% on UCF-Crime.
Limitations and Future Work
Despite its strong performance, GMFVAD faces challenges regarding computational overhead due to the use of SwinBERT and SimCSE. Future work may explore integrating audio streams into the grained fusion framework, particularly for datasets like XD-Violence. Additionally, addressing inference latency remains critical for real-world deployment in live surveillance systems.
Summary updated