Introduction
In the context of "intelligent cities," automated surveillance systems are increasingly tasked with identifying abnormal behaviors. However, "abnormality" is highly context-dependent; for instance, a bicycle is normal on a dedicated path but represents an anomaly in a crowded pedestrian square. Traditional AI models often treat anomaly detection as a binary classification problem, flagging frames without explaining the underlying cause. This research addresses two primary gaps: explainability (providing specific reasons like "unauthorized action") and granularity (analyzing individual targets rather than entire video clips).
Methodology
The proposed multivariate fusion method analyzes video through three parallel branches to generate a final anomaly score. This approach allows the system to understand what an object is, what it is doing, and how it is moving.
- Object Branch: Utilizes the YOLO v3 detector to identify entities (e.g., persons, cars) and compares them against a pre-defined "normal list."
- Action Branch: Tracks individual targets using the Kernelized Correlation Filter (KCF) and classifies their behavior (e.g., "riding," "running") via a two-stream ConvNet (BNInception) that analyzes both visual appearance and optical flow.
- Motion Branch: Focuses on the physics of movement by extracting Histogram of Magnitude Optical Flow (HMOF) features, which are then processed through an autoencoder and a Gaussian Mixture Model (GMM) to detect deviations from normal motion patterns.
The Fusion Module combines these weighted scores, selecting the maximum value to determine the final anomaly status, thereby ensuring that an anomaly detected by any single specialized branch is captured.
Key Results
The model was evaluated on the UMN and UCSD Ped2 datasets. The fusion approach demonstrated significant improvements over single-branch analysis and existing state-of-the-art (SOTA) methods.
On the UMN dataset, which focuses on panic scenarios, the system achieved an AUC of 99.8%. For the UCSD Ped2 dataset, which features pedestrian walkways with non-pedestrian intruders, the pixel-level Equal Error Rate (EER) was reduced to 5.5%, a substantial improvement from the previous SOTA of 17.1%. These results confirm that the three branches provide complementary, non-redundant information that enhances overall detection robustness.
Discussion and Conclusion
By integrating object, action, and motion analysis, the system provides human-readable explanations (e.g., [person, riding, abnormal motion]), which is crucial for real-world security operators to quickly validate alerts. While the parallel deep learning branches introduce computational complexity, the method effectively solves multi-target scenarios by tracking individuals before classification. Future work will focus on optimizing real-time performance and refining the action recognition module to handle more complex, multi-person interactions in diverse environments.
Summary updated