Robust Unsupervised Video Anomaly Detection by Multi-Path Frame Prediction

Xuanzhao Wang, Zhengping Che, Ke Yang, Bo Jiang, Jian Tang, Jieping Ye, Jingyu Wang, Qi Qi

Introduction

Video anomaly detection (VAD) is a critical component of modern surveillance systems, yet it remains a challenging task due to the contextual nature of anomalies and the inherent noise in real-world footage. Traditional methods often struggle with unpredictable background fluctuations and varying object scales. This paper introduces ROADMAP (Robust unsupervised videO Anomaly Detection by Multi-path frAme Prediction), a framework designed to robustly predict normal video frames by ignoring random camera noise and focusing on semantic motion. By leveraging a multi-path architecture and a novel noise tolerance loss, ROADMAP achieves state-of-the-art performance on major VAD benchmarks.

The Challenge: Noise and Scale

In unsupervised video anomaly detection, models are typically trained on "normal" sequences to learn expected patterns. Deviations from these patterns during inference are flagged as anomalies. A common approach is frame prediction, where the model predicts the next frame and calculates the error relative to the ground truth.

However, two major hurdles persist:

  1. Pixel-Level Noise: Surveillance videos contain random fluctuations (sensor noise, lighting shifts) that are unpredictable. Standard models often waste capacity trying to reconstruct this meaningless noise rather than learning semantic motion.
  2. Varying Scales: Objects of interest (e.g., pedestrians vs. vehicles) appear at different scales depending on their distance from the camera, requiring a multi-resolution approach to capture dynamics effectively.

The ROADMAP Framework

To address these challenges, the authors propose a multi-path frame prediction network. Unlike standard U-Net architectures that rely on a single bottleneck, ROADMAP utilizes parallel prediction modules operating at different spatial resolutions.

The ROADMAP architecture featuring a multi-path frame prediction network and a noise tolerance loss module.

As shown in the architecture diagram above, the network consists of three main components:

  • Encoder: A CNN-based backbone that extracts hierarchical features.
  • Predictor: Parallel paths containing Non-local blocks (for global spatial dependencies) and ConvGRU units (for local temporal dynamics). Shallow paths handle static backgrounds, while deeper paths focus on complex, multi-scale semantic motions.
  • Decoder: Reconstructs the predicted frame $\hat{I}_t$ from the aggregated multi-path features.

Noise Tolerance Loss

A key innovation of ROADMAP is the Noise Tolerance Loss. Instead of relying solely on pixel-level Mean Squared Error (MSE), which forces the model to fit random noise, the authors introduce a perceptual loss. Both the predicted and ground-truth frames are passed through a pre-trained VGG16 network. The loss is calculated as the $L_1$ distance between intermediate feature maps.

Comparison between pixel-level frame differences and the noise-tolerant feature differences used by ROADMAP.

As illustrated in the comparison above, raw pixel differences ($|I_t - I_{t-1}|$) are dominated by high-frequency background noise. In contrast, the feature-level differences ($|f_{nt}(I_t) - f_{nt}(I_{t-1})|$) effectively filter out this static, highlighting only the semantically meaningful motion of pedestrians. This ensures the model learns to predict actual content changes rather than sensor artifacts.

Experimental Evaluation

The model was evaluated on three standard benchmarks: CUHK Avenue, ShanghaiTech Campus, and UCSD Pedestrian (Ped1 & Ped2). Anomalies are detected by calculating the Peak Signal-to-Noise Ratio (PSNR) between the predicted and actual frames; a low PSNR indicates a high likelihood of an anomaly.

Key Results:

  • State-of-the-Art Accuracy: ROADMAP achieved frame-level AUROC scores of 88.3% on Avenue, 76.6% on ShanghaiTech, and 96.3% on Ped2.
  • Ablation Insights: The noise tolerance loss provided a significant boost on high-resolution datasets (e.g., +2.1% AUROC on ShanghaiTech). The multi-path design improved performance by 3.7% on the Avenue dataset compared to a single-path baseline.
  • Efficiency: The system processes frames at approximately 10 FPS, making it suitable for near-real-time surveillance applications.

Conclusion and Limitations

ROADMAP demonstrates that filtering out irrelevant camera noise through perceptual loss and multi-scale temporal modeling significantly improves anomaly detection robustness. However, the authors note a limitation: the Noise Tolerance Loss relies on a VGG16 network pre-trained on ImageNet. This makes it less effective for low-resolution or grayscale videos, such as those in the UCSD Pedestrian datasets, where the loss weight had to be reduced to zero. Future research may focus on developing domain-specific noise-tolerant mechanisms for low-resolution environments.

Summary updated

Abstract

Video anomaly detection is commonly used in many applications, such as security surveillance, and is very challenging. A majority of recent video anomaly detection approaches utilize deep reconstruction models, but their performance is often suboptimal because of insufficient reconstruction error differences between normal and abnormal video frames in practice. Meanwhile, frame prediction-based anomaly detection methods have shown promising performance. In this article, we propose a novel and robust unsupervised video anomaly detection method by frame prediction with a proper design which is more in line with the characteristics of surveillance videos. The proposed method is equipped with a multipath ConvGRU-based frame prediction network that can better handle semantically informative objects and areas of different scales and capture spatial-temporal dependencies in normal videos. A noise tolerance loss is introduced during training to mitigate the interference caused by background noise. Extensive experiments have been conducted on the CUHK Avenue, ShanghaiTech Campus, and UCSD Pedestrian datasets, and the results show that our proposed method outperforms existing state-of-the-art approaches. Remarkably, our proposed method obtains the frame-level AUROC score of 88.3% on the CUHK Avenue dataset.
BibTeX
@article{wang2022robust,
  author = {Xuanzhao Wang and Zhengping Che and Ke Yang and Bo Jiang and Jian Tang and Jieping Ye and Jingyu Wang and Qi Qi},
  title = {{Robust Unsupervised Video Anomaly Detection by Multi-Path Frame Prediction}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2022},
  doi = {10.1109/tnnls.2021.3083152},
}
Figure