Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors

Nicolae-Catalin Ristea, Florinel-Alin Croitoru, Radu Tudor Ionescu, Marius Popescu, Fahad Shahbaz Khan, Mubarak Shah

TL;DR

This paper introduces a lightweight, frame-level masked auto-encoder (MAE) for video anomaly detection that achieves an unprecedented processing speed of 1655 FPS—roughly 8 to 70 times faster than current state-of-the-art methods. By combining motion-aware loss weighting, self-distillation between two decoders, and training on synthetic anomalies, the model maintains competitive accuracy while being efficient enough to monitor dozens of video streams on a single GPU.

What Problem Does This Paper Address?

Video Anomaly Detection (VAD) is a critical task for automated surveillance, yet it faces two major hurdles:

  1. Computational Cost: Most high-accuracy models are "object-centric," meaning they rely on heavy pre-processing steps like object detection. This limits them to processing only one or two video streams per GPU (20-30 FPS), which is prohibitively expensive for city-wide surveillance.
  2. The Generalization Paradox: Traditional auto-encoders (AEs) are trained to reconstruct "normal" video frames. The idea is that they will fail to reconstruct "abnormal" events, creating a high error signal. However, modern AEs often generalize too well, reconstructing anomalies with high precision and thus failing to detect them.

The authors aim to bridge the gap between the speed of frame-level methods and the accuracy of object-centric ones.

Methods

The proposed architecture is a lightweight Masked Auto-Encoder based on Convolutional Vision Transformer (CvT) blocks. The framework introduces three key innovations:

  • Motion Gradient Weighting: Standard MAEs treat all image patches (tokens) equally. In surveillance, backgrounds are mostly static and trivial to reconstruct. The authors weight the reconstruction loss based on motion gradients, forcing the model to focus its learning capacity on moving foreground objects where anomalies typically occur.
  • Self-Distillation Pipeline: The architecture uses a shared encoder but two different decoders: a deep "teacher" and a shallower "student." During inference, the discrepancy between the teacher’s and student’s outputs serves as a strong indicator of an anomaly. Because they share an encoder, this adds negligible computational overhead.
  • Open-Set Supervision with Synthetic Anomalies: To prevent the model from reconstructing anomalies too well, the authors augment normal training data with synthetic anomalies cropped from the UBnormal dataset. The model is tasked to "overlook" these anomalies (reconstruct the original normal frame) and simultaneously predict a pixel-level anomaly map.
The proposed architecture featuring a shared encoder, dual decoders, and motion-based weighting.

Datasets

The model was evaluated on four major benchmarks:

  • Avenue: A popular dataset with 15K training and 15K testing frames.
  • ShanghaiTech: The largest real-world VAD dataset (270K training frames).
  • UCSD Ped2: A smaller, classic benchmark.
  • UBnormal: A synthetic dataset used here primarily to provide the "virtual" anomalies for data augmentation during training.

Key Insights and Ideas

  • Efficiency is a First-Class Citizen: By replacing standard fully connected layers with pointwise convolutions and using a shallow 3-block encoder/decoder, the authors prove that "less is more" for real-time applications.
  • Static Backgrounds are Noise: In the context of VAD, a perfectly reconstructed wall is useless information. The motion-gradient weighting is a clever way to inject domain knowledge (that anomalies involve movement) into a self-supervised transformer.
  • Discrepancy as a Signal: The "aha moment" here is using self-distillation not just for model compression, but as a detection tool. A student model trained to mimic a teacher will struggle more on out-of-distribution (abnormal) data than the teacher does, making their disagreement a clear red flag.

Key Results

The most striking result is the speed-to-accuracy trade-off. As shown in Figure 2, the model sits in the top-right corner of the chart, offering state-of-the-art speed with competitive AUC (Area Under the Curve) scores.

  • Speed: 1655 FPS on an NVIDIA RTX 3090. For context, the next fastest competitive method (Park et al.) runs at 195 FPS.
  • Accuracy: On the Avenue dataset, it achieved a 91.3% Micro AUC, outperforming all other frame-level methods and coming within 2% of the much slower object-centric SOTA.
  • Qualitative Success: Figure 5 demonstrates that the model successfully "erases" anomalies (like a cyclist in a pedestrian zone) in its reconstruction, leading to high error values and accurate anomaly maps.
Figure 2. Performance versus speed trade-offs for our self-distilled masked AE and several state-of-the-art methods [26–28, 47, 49, 60, 61, 69, 84] (with open-sourced code), on the Avenue data set. The running times of all methods are measured on a computer with one Nvidia GeForce GTX 3090 GPU with 24 GB of VRAM. Best viewed in color.

Limitations and Future Work

While the model is exceptionally fast, there remains a small performance gap compared to the best object-centric models, which benefit from the explicit spatial priors of a dedicated object detector.

The authors suggest that future work could involve further refining the synthetic anomaly generation to better match the lighting and texture of specific real-world scenes. Additionally, while the model handles spatial anomalies well, exploring how this lightweight transformer architecture could better capture long-term temporal anomalies (events that are only strange because of their duration or sequence) remains an open path.

Summary updated

Abstract

We propose an efficient abnormal event detection model based on a lightweight masked auto-encoder (AE) applied at the video frame level. The novelty of the proposed model is threefold. First, we introduce an approach to weight tokens based on motion gradients, thus shifting the focus from the static background scene to the foreground objects. Second, we integrate a teacher decoder and a student decoder into our architecture, leveraging the discrepancy between the outputs given by the two decoders to improve anomaly detection. Third, we generate synthetic abnormal events to augment the training videos, and task the masked AE model to jointly reconstruct the original frames (without anomalies) and the corresponding pixel-level anomaly maps. Our design leads to an efficient and effective model, as demonstrated by the extensive experiments carried out on four benchmarks: Avenue, ShanghaiTech, UBnormal and UCSD Ped2. The empirical results show that our model achieves an excellent trade-off between speed and accuracy, obtaining competitive AUC scores, while processing 1655 FPS. Hence, our model is between 8 and 70 times faster than competing methods. We also conduct an ablation study to justify our design. Our code is freely available at: https://github.com/ristea/aed-mae.
BibTeX
@inproceedings{ristea2023self,
  author = {Nicolae-Catalin Ristea and Florinel-Alin Croitoru and Radu Tudor Ionescu and Marius Popescu and Fahad Shahbaz Khan and Mubarak Shah},
  title = {{Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure