Abstract
While classic video anomaly detection (VAD) requires labeled normal videos for training, emerging unsupervised VAD (UVAD) aims to discover anomalies directly from fully unlabeled videos. However, existing UVAD methods often rely on shallow models and underperform compared to their semi-supervised counterparts. This paper proposes a full deep neural network (DNN) based solution for highly effective UVAD. We identify a key property called the "normality advantage," where normal events naturally achieve lower reconstruction loss as a DNN learns from unlabeled data. Building on this, we introduce Localization based Reconstruction (LBR) and a novel Self-Paced Refinement (SPR) scheme. SPR proactively filters out suspicious anomalies during training, consolidating the normality advantage. Our framework significantly outperforms existing UVAD methods by 5% to 9% AUROC and rivals the performance of classic VAD methods.
What Problem Does This Paper Address?
Video Anomaly Detection (VAD) is essential for automated surveillance, yet it is traditionally hampered by the need for extensive manual labeling.
Classic VAD is semi-supervised: it requires a "clean" dataset containing only normal activities to train a model. Collecting and verifying such pure data is labor-intensive and often impractical in real-world scenarios. In contrast, Unsupervised VAD (UVAD) seeks to learn directly from raw, unlabeled video streams that may contain both normal and abnormal events.
The primary challenge in UVAD is "pollution": if a model learns from a dataset containing anomalies, it might learn to reconstruct those anomalies as normal, making them undetectable. Previous UVAD attempts often used shallow models or hand-crafted features that lacked the representational power of deep learning. This research introduces a "full deep" solution designed to thrive in the presence of noisy, unlabeled data.
Methods: The LBR-SPR Framework
The proposed solution rests on three pillars:
1. Localization-based Reconstruction (LBR)
Instead of processing entire video frames, the model extracts "Spatio-Temporal Cubes" (STCs)—small 3D patches centered on moving objects. By focusing on localized actions (e.g., a person jumping a fence) rather than static backgrounds, the model amplifies the signal of abnormal behavior. This localization makes the difference in reconstruction loss between normal and abnormal events much more distinct.
2. The "Normality Advantage"
A core insight of this work is that Deep Neural Networks (DNNs) possess an inherent bias: they prioritize learning frequent patterns over rare ones. Because anomalies are, by definition, infrequent, the DNN naturally learns to reconstruct normal patterns first.
As shown in the figure above, even when trained on a mixture of data, normal events consistently exhibit a lower reconstruction error than anomalies. This "normality advantage" provides a solid foundation for distinguishing between the two without explicit labels.
3. Self-Paced Refinement (SPR)
To further enhance this advantage, the authors introduce Self-Paced Refinement (SPR). Traditional self-paced learning starts with "easy" samples and gradually adds "harder" ones. SPR flips this logic: it starts with the entire unlabeled dataset and gradually drops samples with high reconstruction loss—the "suspicious" samples. By proactively filtering out potential anomalies during the learning process, SPR prevents the model from eventually "learning" the anomalies, thereby purifying the normality model on the fly.
4. Motion Enhancement
To capture temporal anomalies (such as a person running in a pedestrian zone), the framework includes an optical flow-based branch. This allows the model to reconstruct motion patterns alongside visual ones, ensuring that both appearance-based and behavior-based anomalies are detected.
Key Results and Benchmarks
The LBR-SPR framework was evaluated on four major VAD benchmarks: UCSDped1, UCSDped2, CUHK Avenue, and ShanghaiTech (SHTech). The results demonstrate a significant leap in UVAD performance:
- Performance Gain: The method outperformed previous state-of-the-art UVAD techniques by a wide margin of 5% to 9% AUROC.
- Competitive Accuracy: On the UCSDped2 dataset, it achieved an AUROC of 97.2%, rivaling or exceeding many classic methods that require clean training data.
- Robustness: It reached 92.8% on the Avenue dataset and 72.6% on the complex ShanghaiTech dataset, proving its effectiveness in diverse and challenging environments.
Conclusion and Future Directions
This research demonstrates that the inherent biases of deep learning can be turned into a powerful asset for unsupervised learning. By exploiting the "normality advantage" and employing a proactive refinement strategy like SPR, it is possible to detect anomalies with high accuracy without any manual labeling.
However, challenges remain. The authors noted that "Merge mode" (combining training and test sets) can sometimes be less effective than "Partial mode" due to distribution shifts in what constitutes "normal" behavior across different scenes. Additionally, the method's reliance on object localization means its performance is tied to the quality of the foreground detection. Future work may explore other self-supervised tasks, such as frame prediction or shuffling detection, to further bolster the robustness of unsupervised anomaly detection.
Summary updated