Long Short-Term Dynamic Prototype Alignment Learning for Video Anomaly Detection

Chao Huang, Jie Wen, Chengliang Liu, Yabo Liu, Zexuan Zhu

Abstract

Video Anomaly Detection (VAD) typically relies on learning normal patterns to identify outliers. However, existing methods often suffer from over-generalization and a lack of long-term temporal context. We present the Prototype-guided Dynamics Matching Network (PDM-Net), a novel framework that addresses these issues by introducing a memory bank of long-term normal motion prototypes. By forcing the model to perform long-distance frame prediction guided by these prototypes, PDM-Net creates a robust bottleneck that effectively distinguishes complex normal activities from subtle anomalies. Our approach achieves state-of-the-art performance on major benchmarks, including ShanghaiTech and UCSD Ped2.

Introduction

Video Anomaly Detection (VAD) is a critical task in automated surveillance, aimed at identifying unusual events such as accidents or security breaches. Most unsupervised VAD methods utilize a prediction or reconstruction paradigm, where high errors in predicting the next frame signal an anomaly.

Despite their success, these methods face two primary hurdles. First, deep neural networks often over-generalize, reconstructing anomalous pixel patterns as effectively as normal ones. Second, they are often short-sighted, focusing on immediate frame-to-frame transitions while ignoring the broader temporal logic of a scene. PDM-Net addresses these by leveraging long-term prototypes to guide future predictions.

Proposed Methodology

Prototype-guided Dynamics Matching Network (PDM-Net)

The core of PDM-Net is the integration of long-term temporal context into a short-term prediction task. The architecture consists of two main modules: the Long-Term Dynamic Prototypes Matching (LDPM) and the Prototype-guided and Dynamic-aware Long-Distance Frame Prediction (PDLP).

The overall architecture of PDM-Net, featuring the PDLP and LDPM modules.

Key Components

  1. Long-term Dynamic Prototypes (LDPM): The system maintains a memory bank of representative features of normal motion sequences. During inference, short-term input features are matched against these stored prototypes.
  2. Long-distance Frame Prediction (PDLP): Instead of predicting the immediate next frame, the model is tasked with predicting a frame further in the future. This "information gap" forces the network to rely on the high-level motion concepts stored in the prototype memory bank rather than simple pixel copying.
  3. Feature Discrimination Module: To ensure the memory bank remains discriminative, the authors employ a Dispersion Loss to keep different prototypes distinct and a Compactness Loss to ensure input features align closely with their nearest normal prototype.

Experimental Evaluation

PDM-Net was evaluated on four standard VAD benchmarks: UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech. The model achieved state-of-the-art (SOTA) results across the board using the Area Under the Curve (AUC) metric.

Dataset PDM-Net AUC Previous SOTA
ShanghaiTech 74.2% 73.7% (AMMCN)
UCSD Ped1 85.2% 84.3%
UCSD Ped2 97.7% 97.3%
CUHK Avenue 88.1% -

Qualitative results demonstrate that the model produces sharp spikes in anomaly scores when encountering irregular events, such as a person moving in the wrong direction or fighting, while maintaining low scores for normal pedestrian flow.

Anomaly score curves and qualitative detection results on UCSD Ped1 and Avenue datasets.

Conclusion

PDM-Net introduces a powerful paradigm for VAD by combining long-distance prediction with a prototype-based memory bank. By making the prediction task intentionally difficult, the model learns to prioritize high-level semantic motion patterns over local pixel regularities. This approach significantly reduces over-generalization and improves detection accuracy in complex, real-world surveillance environments.

Summary updated

BibTeX
@inproceedings{huang2024long,
  author = {Chao Huang and Jie Wen and Chengliang Liu and Yabo Liu and Zexuan Zhu},
  title = {{Long Short-Term Dynamic Prototype Alignment Learning for Video Anomaly Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure