Sensing Anomalies like Humans: A Hominine Framework to Detect Abnormal Events from Unlabeled Videos

Siqi Wang, Guang Yu, Zhiping Cai, En Zhu, Xinwang Liu, Jianping Yin, Chengzhang Zhu

Abstract

While classic video anomaly detection (VAD) requires labeled normal videos for training, emerging unsupervised VAD (UVAD) aims to discover anomalies directly from fully unlabeled videos. However, existing UVAD methods often rely on shallow models and underperform compared to their semi-supervised counterparts. This paper proposes a full deep neural network (DNN) based solution for highly effective UVAD. We identify a key property called the "normality advantage," where normal events naturally achieve lower reconstruction loss as a DNN learns from unlabeled data. Building on this, we introduce Localization based Reconstruction (LBR) and a novel Self-Paced Refinement (SPR) scheme. SPR proactively filters out suspicious anomalies during training, consolidating the normality advantage. Our framework significantly outperforms existing UVAD methods by 5% to 9% AUROC and rivals the performance of classic VAD methods.

What Problem Does This Paper Address?

Video Anomaly Detection (VAD) is essential for automated surveillance, yet it is traditionally hampered by the need for extensive manual labeling.

Classic VAD is semi-supervised: it requires a "clean" dataset containing only normal activities to train a model. Collecting and verifying such pure data is labor-intensive and often impractical in real-world scenarios. In contrast, Unsupervised VAD (UVAD) seeks to learn directly from raw, unlabeled video streams that may contain both normal and abnormal events.

Comparison between classic semi-supervised VAD and the proposed unsupervised UVAD framework.

The primary challenge in UVAD is "pollution": if a model learns from a dataset containing anomalies, it might learn to reconstruct those anomalies as normal, making them undetectable. Previous UVAD attempts often used shallow models or hand-crafted features that lacked the representational power of deep learning. This research introduces a "full deep" solution designed to thrive in the presence of noisy, unlabeled data.

Methods: The LBR-SPR Framework

The proposed solution rests on three pillars:

1. Localization-based Reconstruction (LBR)

Instead of processing entire video frames, the model extracts "Spatio-Temporal Cubes" (STCs)—small 3D patches centered on moving objects. By focusing on localized actions (e.g., a person jumping a fence) rather than static backgrounds, the model amplifies the signal of abnormal behavior. This localization makes the difference in reconstruction loss between normal and abnormal events much more distinct.

2. The "Normality Advantage"

A core insight of this work is that Deep Neural Networks (DNNs) possess an inherent bias: they prioritize learning frequent patterns over rare ones. Because anomalies are, by definition, infrequent, the DNN naturally learns to reconstruct normal patterns first.

The 'normality advantage' demonstrated by the gap in reconstruction loss (RL) between normal and abnormal events during training.

As shown in the figure above, even when trained on a mixture of data, normal events consistently exhibit a lower reconstruction error than anomalies. This "normality advantage" provides a solid foundation for distinguishing between the two without explicit labels.

3. Self-Paced Refinement (SPR)

To further enhance this advantage, the authors introduce Self-Paced Refinement (SPR). Traditional self-paced learning starts with "easy" samples and gradually adds "harder" ones. SPR flips this logic: it starts with the entire unlabeled dataset and gradually drops samples with high reconstruction loss—the "suspicious" samples. By proactively filtering out potential anomalies during the learning process, SPR prevents the model from eventually "learning" the anomalies, thereby purifying the normality model on the fly.

4. Motion Enhancement

To capture temporal anomalies (such as a person running in a pedestrian zone), the framework includes an optical flow-based branch. This allows the model to reconstruct motion patterns alongside visual ones, ensuring that both appearance-based and behavior-based anomalies are detected.

Key Results and Benchmarks

The LBR-SPR framework was evaluated on four major VAD benchmarks: UCSDped1, UCSDped2, CUHK Avenue, and ShanghaiTech (SHTech). The results demonstrate a significant leap in UVAD performance:

  • Performance Gain: The method outperformed previous state-of-the-art UVAD techniques by a wide margin of 5% to 9% AUROC.
  • Competitive Accuracy: On the UCSDped2 dataset, it achieved an AUROC of 97.2%, rivaling or exceeding many classic methods that require clean training data.
  • Robustness: It reached 92.8% on the Avenue dataset and 72.6% on the complex ShanghaiTech dataset, proving its effectiveness in diverse and challenging environments.

Conclusion and Future Directions

This research demonstrates that the inherent biases of deep learning can be turned into a powerful asset for unsupervised learning. By exploiting the "normality advantage" and employing a proactive refinement strategy like SPR, it is possible to detect anomalies with high accuracy without any manual labeling.

However, challenges remain. The authors noted that "Merge mode" (combining training and test sets) can sometimes be less effective than "Partial mode" due to distribution shifts in what constitutes "normal" behavior across different scenes. Additionally, the method's reliance on object localization means its performance is tied to the quality of the foreground detection. Future work may explore other self-supervised tasks, such as frame prediction or shuffling detection, to further bolster the robustness of unsupervised anomaly detection.

Summary updated

Abstract

While classic video anomaly detection (VAD) requires labeled normal videos for training, emerging unsupervised VAD (UVAD) aims to discover anomalies directly from fully unlabeled videos. However, existing UVAD methods still rely on shallow models to perform detection or initialization, and they are evidently inferior to classic VAD methods. This paper proposes a full deep neural network (DNN) based solution that can realize highly effective UVAD. First, we, for the first time, point out that deep reconstruction can be surprisingly effective for UVAD, which inspires us to unveil a property named "normality advantage", i.e., normal events will enjoy lower reconstruction loss when DNN learns to reconstruct unlabeled videos. With this property, we propose Localization based Reconstruction (LBR) as a strong UVAD baseline and a solid foundation of our solution. Second, we propose a novel self-paced refinement (SPR) scheme, which is synthesized into LBR to conduct UVAD. Unlike ordinary self-paced learning that injects more samples in an easy-to-hard manner, the proposed SPR scheme gradually drops samples so that suspicious anomalies can be removed from the learning process. In this way, SPR consolidates normality advantage and enables better UVAD in a more proactive way. Finally, we further design a variant solution that explicitly takes the motion cues into account. The solution evidently enhances the UVAD performance, and it sometimes even surpasses the best classic VAD methods. Experiments show that our solution not only significantly outperforms existing UVAD methods by a wide margin (5% to 9% AUROC), but also enables UVAD to catch up with the mainstream performance of classic VAD.
BibTeX
@inproceedings{wang2021sensing,
  author = {Siqi Wang and Guang Yu and Zhiping Cai and En Zhu and Xinwang Liu and Jianping Yin and Chengzhang Zhu},
  title = {{Sensing Anomalies like Humans: A Hominine Framework to Detect Abnormal Events from Unlabeled Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.48550/arxiv.2108.01975},
}
Figure