Real-Time Anomalous Behavior Detection and Localization in Crowded Scenes

Mohammad Sabokrou, Mahmood Fathy, Mojtaba Hosseini

TL;DR

This paper introduces a novel real-time system for detecting and localizing anomalous behavior in crowded scenes by representing videos as cubic patches. It leverages both global and local feature descriptors, fused through a novel strategy, and utilizes unsupervised feature learning to achieve state-of-the-art performance with improved efficiency.

What Problem Does This Paper Address?

Detecting and localizing unusual events in crowded environments, such as public spaces or surveillance footage, is a significant challenge in computer vision. Traditional methods often struggle with the high dimensionality of video data, limited training samples, and the need for real-time processing. Anomalies are defined by their low likelihood of occurrence and deviation from normal patterns. Existing approaches often fall short in accurately pinpointing the anomaly's location or are computationally too expensive for practical applications.

Methods

The proposed system tackles anomaly detection by treating videos as sequences of cubic patches. It employs a two-pronged approach using two distinct "views" of the data:

  • Global Descriptor: This view captures the overall appearance of patches. Instead of relying on traditional low-level features, the authors use an unsupervised feature learning method based on an auto-encoder. This auto-encoder learns sparse features from normal video patches, aiming to reconstruct them. The learned features are then modeled using Gaussian distributions.
  • Local Descriptor: This view focuses on the spatial and temporal relationships between a patch and its neighbors. It utilizes the Structural Similarity Index Measure (SSIM) to compute similarity between patches across frames and adjacent patches. This descriptor captures sudden changes in spatial-temporal domains.

These two descriptors are then fused. The system uses two Gaussian classifiers, one for each descriptor, to classify patches into "strongly normal," "partly normal/anomaly," and "strongly anomaly." A final decision is made by combining the outputs of these classifiers. To address the computational cost of processing large patches, they propose a method where features learned from small patches (10x10x5) are convolved and pooled across larger test patches (40x40x5), making the learned classifier applicable to larger regions without retraining.

Datasets

The proposed method is evaluated on two widely used datasets for anomaly detection:

  • UCSD: This dataset includes two subsets, Ped1 and Ped2, featuring outdoor scenes with pedestrians. Anomalies include objects like cars, skateboarders, or wheelchairs deviating from normal pedestrian flow.
  • UMN: This dataset consists of three scenes where groups of people suddenly run away, which is considered the anomaly.

Key Insights and Ideas

  • Dual-View Representation: Using both global (overall appearance) and local (neighboring relationships) descriptors provides a more comprehensive understanding of video content, leading to improved anomaly detection.
  • Unsupervised Feature Learning: Employing auto-encoders for feature learning allows the system to discover discriminative features from normal data without explicit labeling, overcoming the limitations of handcrafted features.
  • Patch-Based Approach with Feature Reuse: The strategy of learning features from small patches and then applying them to larger patches via convolution and pooling is a clever way to balance representational power and computational efficiency.
  • Fusion for Robustness: Combining the outputs of two partially independent classifiers (global and local) helps to mitigate the weaknesses of each individual descriptor and improve overall accuracy.
  • Real-time Capability: The design choices, particularly the feature learning and fusion strategy, enable the system to operate in real-time, a crucial requirement for surveillance applications.

Key Results

The proposed method achieves state-of-the-art performance on the UCSD and UMN datasets.

  • UCSD Ped1: The method achieves an EER of 8.4% for frame-level detection and 16.7% for pixel-level detection, outperforming most existing methods.
  • UCSD Ped2: It achieves an EER of 7.5% for frame-level and 16% for pixel-level detection, comparable to or better than other leading approaches.
  • UMN: The system achieves an EER of 2.5% and an AUC of 99.6%, demonstrating strong performance on this dataset as well.
  • Time Efficiency: The method is real-time, achieving up to 25 fps (0.04 seconds per frame) with overlapping patches and even faster (0.11-0.13 seconds per frame) without overlapping, making it suitable for practical deployment.

Limitations and Future Work

  • Patch Size and Overlapping: While the paper explores the influence of patch size and overlapping, finding the absolute optimal parameters might still be an area for further refinement.
  • Computational Cost of Feature Learning: The auto-encoder training phase is noted as time-consuming, though the inference phase is efficient.
  • Generalizability: While tested on standard datasets, the performance on highly diverse or extremely complex real-world scenarios might require further investigation.
  • Future Work: The authors suggest exploring more sophisticated fusion strategies and potentially incorporating more advanced feature learning techniques. They also highlight the potential for further optimization of the real-time performance.

Summary updated

Abstract

In this paper, we propose an accurate and real-time anomaly detection and localization in crowded scenes, and two descriptors for representing anomalous behavior in video are proposed. We consider a video as being a set of cubic patches. Based on the low likelihood of an anomaly occurrence, and the redundancy of structures in normal patches in videos, two (global and local) views are considered for modeling the video. Our algorithm has two components, for (1) representing the patches using local and global descriptors, and for (2) modeling the training patches using a new representation. We have two Gaussian models for all training patches respect to global and local descriptors. The local and global features are based on structure similarity between adjacent patches and the features that are learned in an unsupervised way. We propose a fusion strategy to combine the two descriptors as the output of our system. Experimental results show that our algorithm performs like a state-of-the-art method on several standard datasets, but even is more time-efficient.
BibTeX
@misc{sabokrou2015real,
  author = {Mohammad Sabokrou and Mahmood Fathy and Mojtaba Hosseini},
  title = {{Real-Time Anomalous Behavior Detection and Localization in Crowded Scenes}},
  year = {2015},
  eprint = {1511.07425},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.07425},
}
Figure