Abstract
Video Anomaly Detection (VAD) typically relies on learning normal patterns to identify outliers. However, existing methods often suffer from over-generalization and a lack of long-term temporal context. We present the Prototype-guided Dynamics Matching Network (PDM-Net), a novel framework that addresses these issues by introducing a memory bank of long-term normal motion prototypes. By forcing the model to perform long-distance frame prediction guided by these prototypes, PDM-Net creates a robust bottleneck that effectively distinguishes complex normal activities from subtle anomalies. Our approach achieves state-of-the-art performance on major benchmarks, including ShanghaiTech and UCSD Ped2.
Introduction
Video Anomaly Detection (VAD) is a critical task in automated surveillance, aimed at identifying unusual events such as accidents or security breaches. Most unsupervised VAD methods utilize a prediction or reconstruction paradigm, where high errors in predicting the next frame signal an anomaly.
Despite their success, these methods face two primary hurdles. First, deep neural networks often over-generalize, reconstructing anomalous pixel patterns as effectively as normal ones. Second, they are often short-sighted, focusing on immediate frame-to-frame transitions while ignoring the broader temporal logic of a scene. PDM-Net addresses these by leveraging long-term prototypes to guide future predictions.
Proposed Methodology
Prototype-guided Dynamics Matching Network (PDM-Net)
The core of PDM-Net is the integration of long-term temporal context into a short-term prediction task. The architecture consists of two main modules: the Long-Term Dynamic Prototypes Matching (LDPM) and the Prototype-guided and Dynamic-aware Long-Distance Frame Prediction (PDLP).
Key Components
- Long-term Dynamic Prototypes (LDPM): The system maintains a memory bank of representative features of normal motion sequences. During inference, short-term input features are matched against these stored prototypes.
- Long-distance Frame Prediction (PDLP): Instead of predicting the immediate next frame, the model is tasked with predicting a frame further in the future. This "information gap" forces the network to rely on the high-level motion concepts stored in the prototype memory bank rather than simple pixel copying.
- Feature Discrimination Module: To ensure the memory bank remains discriminative, the authors employ a Dispersion Loss to keep different prototypes distinct and a Compactness Loss to ensure input features align closely with their nearest normal prototype.
Experimental Evaluation
PDM-Net was evaluated on four standard VAD benchmarks: UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech. The model achieved state-of-the-art (SOTA) results across the board using the Area Under the Curve (AUC) metric.
| Dataset | PDM-Net AUC | Previous SOTA |
|---|---|---|
| ShanghaiTech | 74.2% | 73.7% (AMMCN) |
| UCSD Ped1 | 85.2% | 84.3% |
| UCSD Ped2 | 97.7% | 97.3% |
| CUHK Avenue | 88.1% | - |
Qualitative results demonstrate that the model produces sharp spikes in anomaly scores when encountering irregular events, such as a person moving in the wrong direction or fighting, while maintaining low scores for normal pedestrian flow.
Conclusion
PDM-Net introduces a powerful paradigm for VAD by combining long-distance prediction with a prototype-based memory bank. By making the prediction task intentionally difficult, the model learns to prioritize high-level semantic motion patterns over local pixel regularities. This approach significantly reduces over-generalization and improves detection accuracy in complex, real-world surveillance environments.
Summary updated