TL;DR
This paper introduces DyAnNet, an unsupervised video anomaly detection framework that leverages iterative learning and a novel combination of spatial (RGB) and temporal (optical flow) features. By dynamically refining pseudo-labels and integrating scene "dynamicity" with anomaly scores, DyAnNet achieves competitive performance without requiring labeled anomaly data, a significant advantage in real-world surveillance scenarios.
What Problem Does This Paper Address?
Video Anomaly Detection (VAD) is crucial for automated surveillance, but a major hurdle is the sheer volume of data and the prohibitive cost of manual annotation. While supervised methods offer high accuracy, they struggle with novel, unseen anomaly types. Unsupervised methods aim to overcome this by learning from unlabeled data, but often face challenges in accurately distinguishing normal from anomalous behavior, especially when anomalies are subtle or context-dependent. Existing unsupervised approaches often rely on either appearance-based features or deep features, but struggle to capture the full spectrum of anomalies. This paper addresses the need for a practical, unsupervised VAD system that can effectively learn from unlabeled data and robustly detect diverse anomalies by integrating both spatial and temporal information.
Methods
DyAnNet employs a three-stage iterative learning framework:
- Pseudo Label Assignment: In the absence of ground truth, the system first generates "pseudo" anomaly scores ($\hat{y}_s$) and "dynamicity" scores ($\hat{y}_d$) for each video segment.
- Anomaly Score ($\hat{y}_s$): This is derived using unsupervised clustering methods like OneClassSVM and iForest, applied to deep features extracted from I3D pre-trained on the Kinetics dataset. These methods aim to identify segments that deviate from the "normal" data distribution.
- Dynamicity Score ($\hat{y}_d$): This captures the rate of change in pixel displacement over time, calculated using optical flow (e.g., SelFlow or Farneback algorithm). High dynamicity suggests rapid scene changes, which can be indicative of anomalies.
- Pseudo Label ($\hat{y}$): A segment is assigned a pseudo-label of 1 (anomalous) if both its anomaly and dynamicity scores exceed a predefined threshold ($\tau$), otherwise it's labeled 0 (normal). These pseudo-labels form initial "bags" of positive (A) and negative (N) segments.
- Score Learning: Two separate regressor networks, $\Omega$ (for anomaly) and $\psi$ (for dynamicity), are trained using the pseudo-labeled bags. These regressors are implemented using a two-stream I3D network (RGB and optical flow) followed by a 3-layer FCN. They are trained with Mean Squared Error (MSE) loss, treating the task as regression.
- Segment Re-mapping via Iterative Learning: This is the core of the framework. The process iterates:
- The trained regressors ($\Omega$ and $\psi$) are used to re-estimate anomaly and dynamicity scores for all segments.
- These refined scores are used to update the membership of bags A and N.
- The regressors are retrained with the newly refined bags. This iterative refinement allows the model to gradually learn more accurate representations and improve its ability to distinguish anomalies, effectively "self-training" with increasingly confident labels. The final anomaly score ($y_s$) and dynamicity score ($y_d$) are an aggregation of the outputs from the optimized regressors across all passes.
Datasets
The proposed DyAnNet framework was evaluated on three widely used real-world video anomaly detection datasets:
- UCF-Crime [38]: Contains 1900 videos of normal and criminal activities (robbery, vandalism, shooting, etc.) recorded by CCTV.
- CCTV-Fights [31]: Comprises 1000 videos of fighting scenarios recorded in real-world settings.
- UBI-Fights [6]: Includes 1000 videos, with a majority being normal and a smaller portion depicting fighting scenarios.
Key Insights and Ideas
- Dynamicity as a Complementary Signal: Integrating "scene dynamicity" (derived from optical flow) alongside anomaly scores significantly enhances detection performance. This is crucial because some events might have high motion but low semantic anomaly (e.g., smoke from a fire vs. smoke from a controlled effect), or vice-versa.
- Iterative Pseudo-Label Refinement: The core innovation is the iterative self-training loop. By repeatedly re-labeling segments based on current model predictions and retraining, the system progressively refines its understanding of normal and anomalous patterns without needing ground truth.
- Unsupervised Advantage: The framework successfully avoids the need for manual annotation, making it highly practical for large-scale surveillance systems where labeling is a bottleneck.
- Low-Level Features Matter: The paper emphasizes that combining low-level motion features (dynamicity) with high-level deep features is essential for robust anomaly detection.
Key Results
- State-of-the-Art Performance: DyAnNet achieves competitive, and in some cases superior, performance compared to existing unsupervised and weakly-supervised methods on the UCF-Crime, CCTV-Fights, and UBI-Fights datasets, as measured by AUC scores. For instance, the SelFlow-based version of DyAnNet achieved AUC scores of 81.01, 86.31, and 84.50 on CCTV-Fights, UBI-Fights, and UCF-Crime, respectively.
- Improved Detection: Qualitative results show DyAnNet accurately detects anomalous patterns and abrupt scene changes, often a few frames earlier than other methods. It also demonstrates the ability to detect multiple anomaly occurrences within a single video.
- Reduced False Alarms: The iterative learning process helps reduce the False Alarm Rate (FAR), leading to more reliable anomaly detection.
- Backbone Flexibility: The framework shows good performance across different backbone architectures (P3D, TSN, C3D, I3D), with I3D yielding the best results.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while the framework avoids supervision, the accuracy of pseudo-labels is crucial. Inaccurate initial pseudo-labels could potentially lead to suboptimal performance.
- The performance is dependent on the quality of the underlying backbone architecture and the number of training iterations/passes.
- Future Work:
- Explore more advanced techniques to integrate low-level motion features and deep features.
- Investigate more sophisticated methods for pseudo-label generation and refinement.
- The authors note that while AI-assisted surveillance is powerful, it cannot be a complete replacement for manual oversight, highlighting the importance of human judgment in critical decision-making.
Summary updated