Video Anomaly Detection Based on Convolutional Recurrent AutoEncoder

Bokun Wang, Caiqian Yang2022 · DOI

TL;DR

This paper introduces the Convolutional Recurrent AutoEncoder (CR-AE), a novel deep learning framework for video anomaly detection. By effectively combining convolutional layers for spatial feature extraction and attention-based ConvLSTM for temporal pattern analysis, CR-AE achieves state-of-the-art performance on benchmark datasets, demonstrating its ability to identify unusual events in surveillance footage.

What Problem Does This Paper Address?

Video surveillance systems are ubiquitous, aiming to enhance public safety by detecting abnormal events. However, manually monitoring these vast amounts of video data is impractical and inefficient. Video anomaly detection aims to automatically identify unusual or infrequent events within video streams, such as fights, accidents, or suspicious activities. The core challenges in this field include the ambiguous definition of "anomaly," the scarcity of anomalous data for training, and the complexity of real-world backgrounds and dynamic scenes. Existing methods often struggle to simultaneously capture both spatial and temporal irregularities effectively, leading to suboptimal performance.

Methods

The proposed CR-AE model is an end-to-end deep learning framework designed to address the limitations of previous approaches. It integrates two key components:

  • Convolutional AutoEncoder (ConvAE): This part is responsible for capturing spatial irregularities within video frames. It uses convolutional layers to encode the spatial features of the input video clip.
  • Attention-based Convolutional Long-Short-Term Memory (ConvLSTM): This component focuses on modeling temporal patterns and irregularities. By spanning different time steps, the attention mechanism within ConvLSTM allows the model to selectively focus on relevant hidden states from previous frames, mitigating the issue of long-term dependency deterioration.

The CR-AE works by training on normal video sequences. During training, the model learns to reconstruct normal video clips with low error. When presented with a test video clip, if it contains an anomaly, the model will struggle to reconstruct it accurately, resulting in a higher reconstruction error. This error is then used to classify frames as either normal or anomalous. The authors highlight that their hybrid architecture avoids the need for separate optical flow extraction, which is common in some two-stream networks, simplifying the process.

Datasets

The proposed CR-AE model was evaluated on two widely used public datasets for video anomaly detection:

  • UCSD Ped2 Dataset: This dataset consists of footage from a stationary camera overlooking a sidewalk. Anomalies are typically caused by non-pedestrian objects or unusual pedestrian movements. It comprises 4560 frames with a resolution of 320x240, split into 16 training and 14 testing clips.
  • ShanghaiTech Dataset: This dataset offers greater scene diversity, with 13 different scenes and a large number of anomaly types. It contains approximately 316,000 video frames with a resolution of 856x480, split into 330 training and 107 testing video clips. It includes anomalies like sudden movements, chases, and quarrels.

Key Insights and Ideas

  • Synergistic Spatial-Temporal Modeling: The core innovation lies in the effective fusion of convolutional networks for spatial understanding and ConvLSTM with attention for temporal dynamics. This combined approach allows for a more comprehensive capture of anomalies that manifest in both space and time.
  • Attention for Temporal Relevance: The use of an attention mechanism within ConvLSTM is crucial for focusing on the most relevant past frames, improving the model's ability to discern subtle temporal deviations from normal behavior.
  • End-to-End Learning: The CR-AE framework is designed as an end-to-end system, simplifying the pipeline by integrating feature extraction and anomaly detection within a single network, thus avoiding the need for separate pre-processing steps like optical flow estimation.
  • Reconstruction Error as Anomaly Indicator: The fundamental principle that normal patterns can be reconstructed with low error, while anomalies lead to high reconstruction error, is a robust and widely adopted strategy in autoencoder-based anomaly detection.

Key Results

The CR-AE model demonstrated strong performance on both datasets:

  • UCSD Ped2 Dataset: The proposed method achieved a frame-level AUC of 95.6%, outperforming most state-of-the-art methods.
  • ShanghaiTech Dataset: The model achieved a frame-level AUC of 73.1%. While lower than on UCSD Ped2, this is still competitive given the increased complexity and diversity of the ShanghaiTech dataset.
  • Computational Efficiency: The CR-AE achieved a detection speed of 249 fps, significantly exceeding real-time requirements and other compared methods, making it suitable for practical surveillance applications.

Qualitative results showed that the CR-AE could accurately predict anomaly scores close to zero for normal videos and higher scores for anomalous frames. The model was able to detect events like cars and bicycles on sidewalks, as well as intense movements.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Frame-Level Detection: The current method only provides frame-level anomaly detection and does not offer localization of anomaly events within the frame.
  • Challenging Cases: Anomalous objects that are occluded, poorly illuminated, or exhibit very little movement (e.g., a lost package) can still be challenging to detect.

Future work suggested by the authors includes:

  • Online and Adaptive Model Updating: Developing methods for continuous learning and adaptation to changing surveillance environments.
  • Object-Level and Pixel-Level Anomaly Detection: Extending the framework to provide more granular anomaly detection, pinpointing specific objects or pixels involved in anomalous events.

Summary updated

BibTeX
@misc{wang2022video,
  author = {Bokun Wang and Caiqian Yang},
  title = {{Video Anomaly Detection Based on Convolutional Recurrent AutoEncoder}},
  year = {2022},
  doi = {10.3390/s22124647},
}
Figure