Synthetic Pseudo Anomalies for Unsupervised Video Anomaly Detection: A Simple Yet Efficient Framework Based on Masked Autoencoder

Zhiqiang Wu, Xiangyu Huang, Caidan Zhao, Chenxing Gao, Lvdong Chen, Zhiqiang Wu 0001

TL;DR

This paper introduces a novel, efficient framework for unsupervised video anomaly detection by synthesizing "pseudo anomalies" from normal video data using masked autoencoders. This approach, combined with a "normalcy consistency training" strategy, helps models learn more robust boundaries between normal and abnormal events, leading to improved detection performance without requiring real anomaly samples.

What Problem Does This Paper Address?

Video anomaly detection (VAD) is crucial for identifying unusual events in surveillance footage, but it's hampered by the scarcity of anomalous data for training. Traditional methods often rely on autoencoders (AEs) trained on normal data, assuming they'll reconstruct normal scenes well and abnormal ones poorly. However, AEs can sometimes reconstruct anomalies effectively, blurring the lines for detection. While some methods use synthetic anomalies, they often require complex data augmentation or separate training phases, leading to instability and increased training time. This paper aims to address these limitations by proposing a simpler, more efficient way to generate and utilize synthetic anomalies for better VAD.

Methods

The core of the proposed framework is a masked autoencoder (MAE) architecture, inspired by recent advances in masked image modeling.

  • Pseudo Anomaly Synthesis: Instead of complex augmentation, the authors generate pseudo anomalies by randomly masking patches in normal video frames and feeding these masked frames into the MAE. This is a simple yet effective way to create variations that deviate from purely normal data.
  • Future Frame Prediction: The MAE is trained to predict the next frame in a video sequence. The encoder extracts features from both normal and pseudo-anomaly frames. A decoder then attempts to reconstruct the original frame.
  • Normalcy Consistency Training (NCT): This is a key innovation. Instead of treating pseudo anomalies as rare occurrences, the framework simultaneously trains on normal and corresponding pseudo-anomaly samples. It minimizes the bidirectional KL-divergence between the encoded features of normal and pseudo-anomaly pairs. This encourages the model to learn consistent "normalcy" representations, forcing it to better distinguish between what is truly normal and what is a synthetic deviation.
  • Loss Function: The overall loss combines intensity and gradient losses for accurate frame reconstruction of normal data, and a similar loss for pseudo-anomaly frames. The NCT loss ensures consistent feature representations.

Datasets

The framework was evaluated on three standard VAD datasets:

  • UCSD Ped2: A dataset of pedestrian walkways with anomalies like groups of people deviating from the norm.
  • CUHK Avenue: A dataset of street scenes with anomalies such as abandoned objects or unusual vehicle movements.
  • ShanghaiTech: A large-scale dataset of crowded scenes with various anomalies.

Key Insights and Ideas

  • Masked Autoencoders for VAD: Leveraging MAEs, which excel at learning robust representations from masked data, proves effective for VAD by forcing the model to learn underlying patterns rather than just memorizing.
  • "Normalcy Consistency" is Key: The novel NCT strategy is crucial. By explicitly encouraging the model to maintain consistent representations for normal and synthetically altered (pseudo-anomaly) data, it learns sharper boundaries, making anomalies more detectable. This is more effective than simply feeding pseudo anomalies with low probability.
  • Simplicity and Efficiency: The proposed method avoids complex data augmentation pipelines or separate synthesizer training, making it simpler to implement and significantly faster to train, as demonstrated by the comparison with LNRA.
  • Pseudo Anomalies as a Regularizer: Synthetically generated anomalies, when used with NCT, act as a powerful regularizer, pushing the model to learn a more generalized understanding of normal behavior.

Key Results

  • State-of-the-Art Performance: The proposed framework achieves competitive or superior performance compared to several state-of-the-art VAD methods across the three benchmark datasets, particularly showing significant improvement on the challenging ShanghaiTech dataset.
  • Improved AUROC: The paper reports AUROC scores that are comparable to or better than existing methods. For instance, on ShanghaiTech, their method achieved an AUROC of 76.57%, outperforming many others.
  • Significant Speedup: Compared to a similar ViT-based method (LNRA), the proposed framework drastically reduces training time per epoch, making it much more efficient. For example, on the SHTech dataset, their training time per epoch was 2.9 hours, compared to 25.2 hours for LNRA.
  • Ablation Study Success: An ablation study confirmed that both the pseudo anomaly synthesis (PASRM) and the normalcy consistency training (NCT) contribute significantly to the performance improvement.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method is efficient, it still relies on the general AE architecture for future frame prediction. The effectiveness might be further enhanced by exploring more advanced AE designs. The paper also doesn't discuss the impact of different masking strategies or patch sizes in detail, which could be areas for future exploration.
  • Future Work: The authors suggest that their framework can be extended to more complex VAD scenarios. Further research could explore integrating appearance and motion information more deeply within the MAE framework or investigating adaptive masking strategies for anomaly synthesis. The simplicity and efficiency of the approach also make it a good candidate for real-time VAD applications.

Summary updated

Abstract

Due to the limited availability of anomalous samples for training, video anomaly detection is commonly viewed as a one-class classification problem. Many prevalent methods investigate the reconstruction difference produced by AutoEncoders (AEs) under the assumption that the AEs would reconstruct the normal data well while reconstructing anomalies poorly. However, even with only normal data training, the AEs often reconstruct anomalies well, which depletes their anomaly detection performance. To alleviate this issue, we propose a simple yet efficient framework for video anomaly detection. The pseudo anomaly samples are introduced, which are synthesized from only normal data by embedding random mask tokens without extra data processing. We also propose a normalcy consistency training strategy that encourages the AEs to better learn the regular knowledge from normal and corresponding pseudo anomaly data. This way, the AEs learn more distinct reconstruction boundaries between normal and abnormal data, resulting in superior anomaly discrimination capability. Experimental results demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{wu2023synthetic,
  author = {Zhiqiang Wu and Xiangyu Huang and Caidan Zhao and Chenxing Gao and Lvdong Chen and Zhiqiang Wu},
  title = {{Synthetic Pseudo Anomalies for Unsupervised Video Anomaly Detection: A Simple Yet Efficient Framework Based on Masked Autoencoder}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10094296},
}
Figure