Abnormal Behavior Detection Based on Target Analysis

Luchuan Song, Bin Liu, Huihui Zhu, Qi Chu, Nenghai YuarXiv 2021 · arXiv

Introduction

In the context of "intelligent cities," automated surveillance systems are increasingly tasked with identifying abnormal behaviors. However, "abnormality" is highly context-dependent; for instance, a bicycle is normal on a dedicated path but represents an anomaly in a crowded pedestrian square. Traditional AI models often treat anomaly detection as a binary classification problem, flagging frames without explaining the underlying cause. This research addresses two primary gaps: explainability (providing specific reasons like "unauthorized action") and granularity (analyzing individual targets rather than entire video clips).

Methodology

The proposed multivariate fusion method analyzes video through three parallel branches to generate a final anomaly score. This approach allows the system to understand what an object is, what it is doing, and how it is moving.

The multivariate fusion framework architecture, illustrating the Object, Action, and Motion branches leading to a fused anomaly score.
  1. Object Branch: Utilizes the YOLO v3 detector to identify entities (e.g., persons, cars) and compares them against a pre-defined "normal list."
  2. Action Branch: Tracks individual targets using the Kernelized Correlation Filter (KCF) and classifies their behavior (e.g., "riding," "running") via a two-stream ConvNet (BNInception) that analyzes both visual appearance and optical flow.
  3. Motion Branch: Focuses on the physics of movement by extracting Histogram of Magnitude Optical Flow (HMOF) features, which are then processed through an autoencoder and a Gaussian Mixture Model (GMM) to detect deviations from normal motion patterns.

The Fusion Module combines these weighted scores, selecting the maximum value to determine the final anomaly status, thereby ensuring that an anomaly detected by any single specialized branch is captured.

Key Results

The model was evaluated on the UMN and UCSD Ped2 datasets. The fusion approach demonstrated significant improvements over single-branch analysis and existing state-of-the-art (SOTA) methods.

ROC curves for the UMN (left) and UCSD Ped2 (right) datasets, showing the performance of the fused model (Ours-Merge) against individual branches and other methods.

On the UMN dataset, which focuses on panic scenarios, the system achieved an AUC of 99.8%. For the UCSD Ped2 dataset, which features pedestrian walkways with non-pedestrian intruders, the pixel-level Equal Error Rate (EER) was reduced to 5.5%, a substantial improvement from the previous SOTA of 17.1%. These results confirm that the three branches provide complementary, non-redundant information that enhances overall detection robustness.

Discussion and Conclusion

By integrating object, action, and motion analysis, the system provides human-readable explanations (e.g., [person, riding, abnormal motion]), which is crucial for real-world security operators to quickly validate alerts. While the parallel deep learning branches introduce computational complexity, the method effectively solves multi-target scenarios by tracking individuals before classification. Future work will focus on optimizing real-time performance and refining the action recognition module to handle more complex, multi-person interactions in diverse environments.

Summary updated

Abstract

Abnormal behavior detection in surveillance video is a pivotal part of the intelligent city. Most existing methods only consider how to detect anomalies, with less considering to explain the reason of the anomalies. We investigate an orthogonal perspective based on the reason of these abnormal behaviors. To this end, we propose a multivariate fusion method that analyzes each target through three branches: object, action and motion. The object branch focuses on the appearance information, the motion branch focuses on the distribution of the motion features, and the action branch focuses on the action category of the target. The information that these branches focus on is different, and they can complement each other and jointly detect abnormal behavior. The final abnormal score can then be obtained by combining the abnormal scores of the three branches.
BibTeX
@misc{song2021abnormal,
  author = {Luchuan Song and Bin Liu and Huihui Zhu and Qi Chu and Nenghai Yu},
  title = {{Abnormal Behavior Detection Based on Target Analysis}},
  year = {2021},
  eprint = {2107.13706},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.13706},
}
Figure