Robust Anomaly Detection in Videos Using Multilevel Representations

Hung Vu, Tu Dinh Nguyen, Trung Le, Wei Luo, Dinh Q. Phung

Introduction

Detecting anomalies in surveillance video—such as unauthorized vehicles or unusual pedestrian behavior—remains a significant challenge in computer vision. Traditional detection methods often struggle with two primary issues: high sensitivity to environmental noise (shadows, lighting changes) and fragmented detections where a single anomalous object is perceived as disconnected parts. This paper proposes the MultiLevel Anomaly Detector (MLAD), a framework designed to mitigate these issues by integrating low-level visual features with high-level semantic representations.

Methodology: The MLAD Framework

The MLAD system processes video through a multilevel approach, leveraging both appearance and motion data across various layers of abstraction.

Overview of the MLAD training and detection phases.

The framework operates in three main phases:

  1. Representation Learning: Denoising Autoencoders (DAEs) are trained on raw video frames (intensity) and optical flow maps (motion). By extracting activations from different layers of these DAEs, the system captures a range of features from simple edges (low-level) to complex object shapes and spatial relationships (high-level).
  2. Cross-Domain Generation with cGANs: For every representation level, the system trains a pair of Conditional Generative Adversarial Networks (cGANs). One GAN predicts the motion map given the appearance frame, while the other predicts the appearance frame given the motion map. In "normal" scenes, the relationship between appearance and motion is predictable; if the GANs fail to accurately reconstruct one from the other, it indicates an anomaly.
  3. Multilevel Consolidation: Finally, the system generates error maps at each level and merges them. High-level detections, which are more robust but less precise, help validate and complete low-level detections, which are precise but often noisy.

Key Insights and Results

A core strength of MLAD is its ability to leverage "objectness." While low-level features might only detect high-contrast edges, higher-level layers encode semantic identities, allowing the system to identify an anomaly as a single continuous entity rather than a collection of moving pixels.

Comparison of anomaly heatmaps between AlexNet and MLAD (Layer 3).

The authors found that DAEs trained specifically on surveillance scenes outperformed general-purpose features from models like AlexNet. As shown in the comparison above, MLAD provides a more coherent and localized detection map. This is attributed to the fact that surveillance anomalies are often small and local, whereas models like AlexNet are optimized for large, global object classification.

Performance Benchmarks

The model was evaluated on the UCSD Ped 1, Ped 2, and CUHK Avenue datasets. Notably, the authors provided a re-annotated ground truth for UCSD Ped 1 to correct previous labeling errors involving missing anomalies and mislabeled occlusions. MLAD achieved state-of-the-art results, specifically improving the pixel-level Equal Error Rate (EER) by:

  • UCSD Ped 1: 11.35% improvement.
  • UCSD Ped 2: 12.32% improvement.
  • Avenue: 4.31% improvement.

Conclusion and Future Work

MLAD effectively reduces false positives and fragmented detections by combining multilevel representations with cross-modal validation. While the computational cost of running multiple GANs is a consideration, the authors suggest that a simplified version using only low and high-level layers (MLAD_0+3) offers an optimal balance of speed and accuracy. Future research directions include integrating temporal consistency across longer sequences of frames to further refine detection stability and reduce noise.

Summary updated

Abstract

Detecting anomalies in surveillance videos has long been an important but unsolved problem. In particular, many existing solutions are overly sensitive to (often ephemeral) visual artifacts in the raw video data, resulting in false positives and fragmented detection regions. To overcome such sensitivity and to capture true anomalies with semantic significance, one natural idea is to seek validation from abstract representations of the videos. This paper introduces a framework of robust anomaly detection using multilevel representations of both intensity and motion data. The framework consists of three main components: 1) representation learning using Denoising Autoencoders, 2) level-wise representation generation using Conditional Generative Adversarial Networks, and 3) consolidating anomalous regions detected at each representation level. Our proposed multilevel detector shows a significant improvement in pixel-level Equal Error Rate, namely 11.35%, 12.32% and 4.31% improvement in UCSD Ped 1, UCSD Ped 2 and Avenue datasets respectively. In addition, the model allowed us to detect mislabeled anomalies in the UCDS Ped 1.
BibTeX
@inproceedings{vu2019robust,
  author = {Hung Vu and Tu Dinh Nguyen and Trung Le and Wei Luo and Dinh Q. Phung},
  title = {{Robust Anomaly Detection in Videos Using Multilevel Representations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33015216},
}
Figure