Anomaly Detection and Localization in Crowded Scenes by Motion-field Shape Description and Similarity-based Statistical Learning

Xinfeng Zhang, Yang Su, Xinjian Zhang, Weishan Zhang, Jiulong Zhang

Abstract

In crowded scenes, detection and localization of abnormal behaviors is challenging because high-density crowds make object segmentation and tracking extremely difficult. This paper presents a method that associates the optical flows of multiple frames to capture short-term trajectories and introduces a histogram-based shape descriptor, referred to as shape contexts, to describe these trajectories. Furthermore, we propose a K-NN similarity-based statistical model to detect anomalies over time and space. This unsupervised one-class learning algorithm requires no clustering or prior assumptions. By retrieving K-NN samples from the training set and constructing a Gaussian model based on their intra-class similarities, we calculate the joint probability of a test sample's similarity. Abnormal events are detected when this probability falls below predefined thresholds. This scheme adapts to the entire scene, as the probability is not affected by motion distortions arising from perspective distortion. Experiments on real-world surveillance videos demonstrate that the proposed method reliably detects and locates abnormal events, outperforming state-of-the-art approaches.

Introduction: The Challenge of Crowded Scenes

Detecting anomalies in dense crowds is a significant hurdle in computer vision. Traditional methods typically rely on segmenting and tracking individual objects. However, in a dense crowd, individuals constantly overlap and obscure one another, causing standard trackers to fail.

Beyond density, unsupervised models often struggle with "perspective distortion." In a typical surveillance shot, objects far from the camera appear smaller and move fewer pixels per second than those close to the camera. Using a fixed distance threshold to define "normal" behavior often leads to false positives, where a normal person walking near the camera is flagged as an anomaly simply because their pixel-velocity is higher than the training data from the background. This research seeks a method robust to these challenges without requiring labeled "abnormal" data for training.

Methodology

The proposed pipeline consists of three primary stages designed for efficiency and robustness:

  1. Short-term Trajectory Capture: Rather than attempting long-term tracking, the system uses optical flow to track "particles" over a window of 10 frames. This captures the immediate "shape" of motion without the complexity and failure points of long-term object tracking.
  2. Shape Context Descriptors: These short-term trajectories are converted into histogram-based descriptors called "shape contexts." This summarizes the distribution of motion within a local patch in a computationally efficient manner.
  3. K-NN Similarity-based Statistical Learning: This core innovation involves finding the $K$ nearest neighbors in the normal training data for any new test patch. Instead of measuring raw distance, the system builds a Gaussian model based on the intra-class similarity of those $K$ neighbors. It then calculates the joint probability that the test patch fits into that similarity distribution. If the probability is too low, the patch is flagged as an anomaly.

This "one-class" learning approach is advantageous because it only requires "normal" video for training and bypasses the need for pre-defined clusters, which are often difficult to maintain in dynamic real-world environments.

Experimental Evaluation

The researchers evaluated the model on two standard benchmarks: the UCSDped1 dataset, which features significant perspective distortion and anomalies like bikers and carts on pedestrian paths, and the Subway Dataset, which includes anomalies like loitering and wrong-way walking.

Quantitative Results

The proposed method outperformed several state-of-the-art (SOTA) approaches, including Social Force Models (SFM) and Mixture of Dynamic Textures (MDT). On the UCSDped1 dataset, the method achieved an Equal Error Rate (EER) of 25.6% at the pixel level, a marked improvement over the 37.3% achieved by Gaussian Process Regression (GPR).

ROC curves comparing the proposed method against state-of-the-art approaches on the UCSDped1 dataset (a) and Subway dataset (b).

Localization and Efficiency

Qualitatively, the model proved highly effective at localizing anomalies. Rather than flagging an entire frame, the system draws specific grids around the anomalous objects. The system is also designed for real-time application, processing frames in approximately 175 milliseconds—significantly faster than competing spatio-temporal composition methods.

Qualitative results of anomaly detection and localization. Red boxes indicate detected anomalies like bikers and carts across various test frames.

Conclusion

By modeling the distribution of similarities rather than raw feature distances, this method naturally scales with perspective. If a distant, small biker appears, the model compares it to other distant, small normal pedestrians, maintaining local consistency. While optical flow can be noisy in extremely dense crowds, the current results show a robust ability to identify and locate "weird" behaviors in unsupervised settings. Future work may explore multi-scale analysis to improve object coverage and investigate how similarity-based logic handles contextual anomalies where the location itself defines the abnormality of an action.

Summary updated

Abstract

In crowded scenes, detection and localization of abnormal behaviors is challenging in that high-density people make object segmentation and tracking extremely difficult. We associate the optical flows of multiple frames to capture short-term trajectories and introduce the histogram-based shape descriptor referred to as shape contexts to describe such short-term trajectories. Furthermore, we propose a K-NN similarity-based statistical model to detect anomalies over time and space, which is an unsupervised one-class learning algorithm requiring no clustering nor any prior assumption. Firstly, we retrieve the K-NN samples from the training set in regard to the testing sample, and then use the similarities between every pair of the K-NN samples to construct a Gaussian model. Finally, the probabilities of the similarities from the testing sample to the K-NN samples under the Gaussian model are calculated in the form of a joint probability. Abnormal events can be detected by judging whether the joint probability is below predefined thresholds in terms of time and space, separately. Such a scheme can adapt to the whole scene, since the probability computed as such is not affected by motion distortions arising from perspective distortion. We conduct experiments on real-world surveillance videos, and the results demonstrate that the proposed method can reliably detect and locate the abnormal events in the video sequences, outperforming the state-of-the-art approaches.
BibTeX
@misc{zhang2018anomaly,
  author = {Xinfeng Zhang and Yang Su and Xinjian Zhang and Weishan Zhang and Jiulong Zhang},
  title = {{Anomaly Detection and Localization in Crowded Scenes by Motion-field Shape Description and Similarity-based Statistical Learning}},
  year = {2018},
  eprint = {1805.10620},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.10620},
}
Figure