STAN: Spatio-Temporal Adversarial Networks for Abnormal Event Detection

Sangmin Lee, Hak Gu Kim, Yong Man Ro

Abstract

In this paper, we propose a novel abnormal event detection method with spatio-temporal adversarial networks (STAN). We devise a spatio-temporal generator which synthesizes an inter-frame by considering spatio-temporal characteristics with bidirectional ConvLSTM. A proposed spatio-temporal discriminator determines whether an input sequence is real-normal or not with 3D convolutional layers. These two networks are trained in an adversarial way to effectively encode spatio-temporal features of normal patterns. After the learning, the generator and the discriminator can be independently used as detectors, and deviations from the learned normal patterns are detected as abnormalities.

Introduction

Automating video surveillance is a critical yet challenging task. While humans can easily identify anomalies like a vehicle on a pedestrian path, computational models often struggle with the complexity of high-dimensional video data. Traditional methods typically suffer from two main issues: brittleness in crowded scenes due to occlusion and a heavy reliance on computationally expensive motion descriptors like optical flow.

STAN addresses these challenges by learning "normal" behavior directly from raw pixels. By capturing both spatial (appearance) and temporal (motion) characteristics without external descriptors, the model provides a robust framework for anomaly detection.

The STAN Architecture

The core of the proposed method is an adversarial framework consisting of a generator and a discriminator designed to capture spatio-temporal features.

The architecture of STAN, featuring a bidirectional ConvLSTM generator and a 3D-CNN discriminator.

Spatio-Temporal Generator

The generator acts as an "artist" that learns to synthesize a missing middle frame ($\hat{X}_t$) given a sequence of preceding ($X_{t-5} \dots X_{t-1}$) and succeeding ($X_{t+1} \dots X_{t+5}$) frames. It utilizes a bidirectional ConvLSTM architecture. By looking both forward and backward in time, the generator captures complex motion transitions more effectively than unidirectional models.

Spatio-Temporal Discriminator

The discriminator serves as the "critic." It employs a 3D Convolutional Neural Network (3D-CNN) to analyze a sequence of frames and distinguish between "real" normal sequences and "fake" sequences produced by the generator.

During training, both networks are exposed only to normal video data. The generator learns to reconstruct normal frames with high fidelity, while the discriminator learns the specific spatio-temporal signatures of normal motion.

Detection and Interpretability

When presented with a test video, anomalies are identified based on deviations from the learned normal patterns. If a frame contains an unseen event (e.g., a bicycle in a pedestrian zone), the generator produces a high reconstruction error, and the discriminator flags the sequence as "fake." The final abnormality score is a weighted combination of these two metrics.

Qualitative results showing (a) original frames, (b) reconstructions, (c) anomaly heatmaps, and (d) discriminator gradient visualizations.

A key advantage of STAN is its interpretability. By visualizing the gradients of the discriminator using guided backpropagation, the system can highlight the specific spatial regions that triggered the anomaly detection, as shown in the figure above.

Experimental Results

The authors evaluated STAN on three benchmark datasets: UCSD Ped1, UCSD Ped2, and the Avenue Dataset.

  • UCSD Ped2: STAN achieved a state-of-the-art 96.5% AUC, demonstrating superior performance in identifying varied anomalies like skaters and carts.
  • Avenue Dataset: The model reached 87.2% AUC, proving effective even in complex scenes with frequent occlusions.
  • Precision: Notably, the model achieved 100% precision on the Ped2 dataset at the event level, indicating zero false alarms.

On the Ped1 dataset, the AUC was slightly lower (82.1%). The authors attributed this to the model's high sensitivity; it correctly flagged video artifacts and noise as "abnormal," even though these were not labeled as anomalies in the ground truth.

Conclusion

STAN provides a powerful, descriptor-free approach to video anomaly detection. By leveraging bidirectional ConvLSTMs and 3D-CNNs in an adversarial framework, it achieves competitive performance and offers visual interpretability. Future work will focus on optimizing the computational efficiency of these heavy architectures for real-time deployment and improving robustness against highly dynamic backgrounds.

Summary updated

Abstract

In this paper, we propose a novel abnormal event detection method with spatio-temporal adversarial networks (STAN). We devise a spatio-temporal generator which synthesizes an inter-frame by considering spatio-temporal characteristics with bidirectional ConvLSTM. A proposed spatio-temporal discriminator determines whether an input sequence is real-normal or not with 3D convolutional layers. These two networks are trained in an adversarial way to effectively encode spatio-temporal features of normal patterns. After the learning, the generator and the discriminator can be independently used as detectors, and deviations from the learned normal patterns are detected as abnormalities. Experimental results show that the proposed method achieved competitive performance compared to the state-of-the-art methods. Further, for the interpretation, we visualize the location of abnormal events detected by the proposed networks using a generator loss and discriminator gradients.
BibTeX
@inproceedings{lee2018stan,
  author = {Sangmin Lee and Hak Gu Kim and Yong Man Ro},
  title = {{STAN: Spatio-Temporal Adversarial Networks for Abnormal Event Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=weKjMtQcmA},
}
Figure