Introduction
Video Anomaly Detection (VAD) is a critical task in computer vision, typically framed as an unsupervised learning problem where models are trained exclusively on "normal" data. The goal is to identify deviations from these learned patterns during inference. However, existing Deep Neural Network (DNN) approaches face significant challenges. First, the Generalization Paradox suggests that DNNs often generalize too well, inadvertently learning to reconstruct abnormal events with low error. Second, many models suffer from Short-Sightedness, focusing only on the immediate next frame ($t+1$), which allows them to "cheat" by copying pixels rather than learning motion dynamics. Finally, representing the Pattern Diversity of normal behavior (e.g., varying crowd densities) remains difficult without blurring the boundary between normal and abnormal.
To address these issues, we present PDM-Net (Prototype-guided Dynamics Matching Network), a framework that leverages a memory bank of normal motion prototypes to perform long-distance frame prediction, thereby magnifying the error signal for anomalies.
Methodology
Prototype-guided Dynamics Matching Network (PDM-Net)
The PDM-Net architecture is built around two core innovations designed to force the model to rely on high-level temporal understanding rather than low-level pixel correlations.
1. Long-Distance Frame Prediction (PDLP)
Unlike standard models that predict the next consecutive frame, PDM-Net is tasked with predicting a frame significantly further in the future. This long-distance prediction is intentionally difficult, preventing the model from relying on simple pixel-copying and forcing it to extract deeper spatio-temporal features from the input sequence.
2. Long-Term Dynamic Prototypes Matching (LDPM)
To bridge the temporal gap in long-distance prediction, the model utilizes a Memory Bank of "normal" motion prototypes. During training, motion features from long sequences are clustered into representative prototypes. During inference, the model takes a short input clip, identifies the most relevant normal prototypes, and uses them to "hallucinate" the missing motion information required to predict the distant target frame. This ensures that the model's predictive capability is strictly constrained to the "vocabulary" of normal motion.
3. Feature Discrimination
To maintain a clear definition of normality, the framework employs two constraints: Prototype Dispersion, which ensures prototypes are distinct and cover diverse normal behaviors, and Feature Compactness, which forces extracted video features to align closely with their nearest prototype.
Experiments and Results
Datasets and Evaluation
The model was evaluated on four benchmark datasets: UCSD Ped1 & Ped2, CUHK Avenue, and ShanghaiTech. These datasets encompass a wide range of environments and anomaly types, from simple pedestrian walkway violations to complex campus scenes involving fighting or objects being thrown.
Performance Analysis
PDM-Net achieved state-of-the-art (SOTA) results across all benchmarks. Notably, it reached 97.7% AUC on UCSD Ped2 and 74.2% AUC on the challenging ShanghaiTech dataset. Ablation studies confirmed that the combination of long-distance prediction and prototype matching is essential; long-distance prediction alone actually degrades performance, but when guided by the prototype memory bank, accuracy improves significantly.
As shown in the anomaly score curves, the model produces sharp spikes when encountering abnormal dynamics. Because the model can only "imagine" the future using normal prototypes, its prediction error increases dramatically when the actual video contains abnormal motion (e.g., fighting), making detection highly effective.
Discussion and Conclusion
PDM-Net demonstrates that "forced failure" can be a powerful feature in anomaly detection. By making the prediction task harder and restricting the model's internal representations to a set of normal prototypes, the system guarantees a high error rate for unseen abnormal behaviors.
Despite its success, some limitations remain. The use of a fixed number of prototypes ($B=100$) may be insufficient for extremely complex urban environments. Additionally, the heavy reliance on motion dynamics (residual sequences) makes the model less sensitive to static anomalies, such as abandoned objects. Future research will focus on integrating appearance-based features to better handle static threats and exploring dynamic prototype allocation to adapt to more complex scenes.
Summary updated