GMFVAD: Using Grained Multi-modal Feature to Improve Video Anomaly Detection

Guangyu Dai, Dong Chen, Siliang Tang, Yueting ZhuangarXiv 2025 · DOI · arXiv

Abstract

GMFVAD (Grained Multi-modal Feature for Video Anomaly Detection) improves surveillance monitoring by combining refined visual features with dense text captions. By using a "Glance-Focus" network to filter out redundant video data and aligning it with semantic text embeddings, the model achieves state-of-the-art performance across major anomaly detection benchmarks.

Introduction

Video Anomaly Detection (VAD) aims to automatically identify unusual events, such as crimes or accidents, in continuous surveillance footage. Most existing systems rely solely on visual features, which often contain significant "redundant information"—background noise or irrelevant movements that mask the actual anomaly.

While some recent multi-modal approaches have attempted to integrate text descriptions, they often do so in a "coarse" manner, applying general descriptions to noisy video snippets. This makes it difficult for models to align the rich semantic meaning of text with cluttered visual data. The GMFVAD framework addresses this by creating a "grained" approach that filters out visual redundancy before fusing it with text.

Methodology

The GMFVAD framework consists of two primary branches that eventually merge into a multi-scale temporal learner.

The overall architecture of the GMFVAD framework, illustrating the visual and text branches leading into grained multi-modal fusion.

The Visual Branch: Glance-Focus Network

Instead of processing raw video features directly, the authors implement a Glance-Focus Network to refine the visual input:

  • Glance Block: Utilizes a video clip-level transformer to scan the scene and identify snippets likely to contain anomalies.
  • Focus Block: Employs self-attention convolutions to highlight correlations between nearby clips, effectively "zooming in" on important temporal details and reducing background redundancy.
Detailed architecture of the Glance-Focus Network, showing the sequential Glance and Focus blocks.

The Text Branch: Dense Captioning

The model uses SwinBERT to generate dense captions for video snippets. These captions are converted into sentence embeddings using SimCSE (a contrastive learning method). This provides high-level semantic labels (e.g., "a person is running with a knife") that visual pixels alone might struggle to categorize.

Fusion and Temporal Learning

The refined visual features and text embeddings are fused and passed through a Multi-scale Temporal Network (MTN). This network uses pyramid dilated convolutions and self-attention to capture both short-term and long-term dependencies in the video. The model is trained using a Top-k Multiple Instance Learning (MIL) framework, focusing on snippets with the highest anomaly scores to learn from video-level labels without frame-by-frame annotations.

Key Insights and Results

The core finding of this research is that multi-modal fusion fails when the visual signal is too noisy. By refining visual features before fusion, text features can more accurately "anchor" the semantic meaning of the scene.

Ablation studies revealed that dataset complexity dictates modality importance. On simpler datasets like ShanghaiTech, text features contributed more to the performance boost. Conversely, on complex datasets like UCF-Crime, refined visual features played a more significant role.

Performance Benchmarks

GMFVAD achieved state-of-the-art (SOTA) or near-SOTA results across four standard benchmarks:

  • UCF-Crime: Achieved 87.22% AUC, significantly beating visual-only methods like RTFM (84.30%).
  • ShanghaiTech: Reached 98.23% AUC, a notable improvement over the baseline.
  • XD-Violence: Achieved 81.52% AP (Average Precision).
  • Ablation Success: Adding both visual refinement and text features improved the baseline by 3.59% on ShanghaiTech and 4.97% on UCF-Crime.

Limitations and Future Work

Despite its strong performance, GMFVAD faces challenges regarding computational overhead due to the use of SwinBERT and SimCSE. Future work may explore integrating audio streams into the grained fusion framework, particularly for datasets like XD-Violence. Additionally, addressing inference latency remains critical for real-world deployment in live surveillance systems.

Summary updated

Abstract

Video anomaly detection (VAD) is a challenging task that detects anomalous frames in continuous surveillance videos. Most previous work utilizes the spatio-temporal correlation of visual features to distinguish whether there are abnormalities in video snippets. Recently, some works attempt to introduce multi-modal information, like text feature, to enhance the results of video anomaly detection. However, these works merely incorporate text features into video snippets in a coarse manner, overlooking the significant amount of redundant information that may exist within the video snippets. Therefore, we propose to leverage the diversity among multi-modal information to further refine the extracted features, reducing the redundancy in visual features, and we propose Grained Multi-modal Feature for Video Anomaly Detection (GMFVAD). Specifically, we generate more grained multi-modal feature based on the video snippet, which summarizes the main content, and text features based on the captions of original video will be introduced to further enhance the visual features of highlighted portions. Experiments show that the proposed GMFVAD achieves state-of-the-art performance on four mainly datasets. Ablation experiments also validate that the improvement of GMFVAD is due to the reduction of redundant information.
BibTeX
@misc{dai2025gmfvad,
  author = {Guangyu Dai and Dong Chen and Siliang Tang and Yueting Zhuang},
  title = {{GMFVAD: Using Grained Multi-modal Feature to Improve Video Anomaly Detection}},
  year = {2025},
  eprint = {2510.20268},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.20268},
}
Figure