Feature Prediction Diffusion Model for Video Anomaly Detection

Cheng Yan, Shiyu Zhang, Yang Liu, Guansong Pang, Wenjun Wang

Introduction

Video Anomaly Detection (VAD) involves identifying rare and unpredictable events within surveillance footage. Traditionally, this "one-class learning" task relies on Generative Adversarial Networks (GANs) or Auto-Encoders (AEs) to model "normal" behavior. However, these architectures often produce blurry reconstructions and struggle with background noise, necessitating auxiliary semantic models like object detectors or optical flow estimators to achieve high performance. These "crutches" introduce significant computational overhead and potential points of failure.

The Feature Prediction Diffusion Model (FPDM) addresses these limitations by leveraging the superior generative and denoising capabilities of Diffusion Models (DMs). FPDM is the first framework to apply DMs to VAD by predicting frame features, eliminating the need for complex auxiliary tools.

Comparison of VAD approaches: GAN-based, Auto-encoder-based, and the proposed Diffusion model-based approach.

Methodology

FPDM employs a two-stage diffusion pipeline operating on compressed feature maps extracted by an unsupervised 2D encoder. This approach avoids the high computational cost of pixel-level diffusion while maintaining rich semantic information.

Feature Prediction and Refinement

The core of FPDM consists of two specialized modules:

  1. Feature Prediction Diffusion Module (Motion): This module learns the distribution of normal motion by denoising a target feature map based on a sequence of previous "normal" features.
  2. Feature Refinement Diffusion Module (Appearance): To prevent the predicted features from becoming too generic, this module uses a cross-attention mechanism to inject appearance details from the original frame, focusing on the distribution of normal appearance.
Detailed architecture of the FPDM, showing the Feature Extraction, Feature Prediction, and Feature Refinement modules.

A critical finding in the training process was that joint training of these modules led to poor performance. Instead, the authors utilized a staged training strategy, converging the prediction module before training the refinement module.

Experimental Results

FPDM was evaluated on major benchmarks including CUHK Avenue, ShanghaiTech, UCF-Crime, and UBnormal. The model achieved state-of-the-art results for image-based one-class VAD:

  • ShanghaiTech: 78.6% AUC, a significant improvement over previous image-based methods.
  • UCF-Crime: 74.7% AUC, outperforming several methods that utilize specialized 3D features.

To address the inherent latency of diffusion models, the authors implemented Denoising Diffusion Implicit Models (DDIM) sampling, resulting in a 20x speedup during inference compared to standard DDPM sampling.

Conclusion

FPDM demonstrates that Diffusion Models can effectively replace complex multi-model pipelines in video anomaly detection. By specializing in motion prediction and appearance refinement, the model achieves competitive performance using only basic 2D features. Future work may explore end-to-end training of the encoder and diffusion modules to further optimize feature alignment and detection accuracy.

Summary updated

Abstract

Anomaly detection in the video is an important research area and a challenging task in real applications. Due to the unavailability of large-scale annotated anomaly events, most existing video anomaly detection (VAD) methods focus on learning the distribution of normal samples to detect the substantially deviated samples as anomalies. To well learn the distribution of normal motion and appearance, many auxiliary networks are employed to extract foreground object or action information. These high-level semantic features effectively filter the noise from the background to decrease its influence on detection models. However, the capability of these extra semantic models heavily affects the performance of the VAD methods. Motivated by the impressive generative and anti-noise capacity of diffusion model (DM), in this work, we introduce a novel DM-based method to predict the features of video frames for anomaly detection. We aim to learn the distribution of normal samples without any extra high-level semantic feature extraction models involved. To this end, we build two denoising diffusion implicit modules to predict and refine the features. The first module concentrates on feature motion learning, while the last focuses on feature appearance learning. To the best of our knowledge, it is the first DM-based method to predict frame features for VAD. The strong capacity of DMs also enables our method to more accurately predict the normal features than non-DM-based feature prediction-based VAD methods. Extensive experiments show that the proposed approach substantially outperforms state-of-the-art competing methods. The code is available atFPDM.
BibTeX
@inproceedings{yan2023feature,
  author = {Cheng Yan and Shiyu Zhang and Yang Liu and Guansong Pang and Wenjun Wang},
  title = {{Feature Prediction Diffusion Model for Video Anomaly Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00509},
}
Figure