TL;DR
This paper tackles the challenge of video anomaly detection (VAD) by proposing a novel continual learning framework. It introduces a new, more realistic dataset and evaluation metric that considers temporal aspects and the need for systems to adapt to evolving "normal" behavior over time, outperforming existing methods in this challenging setting.
What Problem Does This Paper Address?
Video anomaly detection (VAD) aims to identify unusual events in video surveillance footage, a critical task for security and monitoring. However, current VAD research faces several limitations:
- Flawed Problem Formulation: Most methods assume a static definition of "normal" behavior, which is unrealistic. In real-world scenarios, normal patterns evolve, and systems need to adapt without forgetting past knowledge. This is known as the "open set" nature of nominal classes.
- Inadequate Evaluation: Existing benchmarks and metrics (like frame-level AUC) focus on detecting anomalous frames rather than anomalous activities, and they don't account for the crucial temporal aspect or the delay in detection.
- Lack of Realistic Datasets: Current datasets are often small, contain staged anomalies, and don't reflect the continuous learning required in real-world applications. They also don't capture the contextual nature of anomalies (e.g., loitering being normal during the day but anomalous at night).
- Catastrophic Forgetting: Deep learning models, prevalent in VAD, are prone to forgetting previously learned information when trained incrementally on new data, making them unsuitable for continual learning.
This paper addresses these issues by proposing a VAD approach that can learn continually from new data, adapting to evolving nominal behaviors and providing a more realistic evaluation framework.
Methods
The authors propose a two-stage continual learning approach for VAD:
- Feature Extraction: They first use a pre-trained object detector (YOLO-v4) to identify objects in each frame. Then, they construct a feature embedding that captures:
- Spatio-temporal object features: Derived from object paths tracked by DeepSORT.
- Contextual features: Including the number of objects per class, observed object classes, day of the week, and time of day (discretized into weekday/weekend and active/inactive times).
- Prediction errors: From an RNN that predicts future object paths.
- Anomaly Detection with kNN and RNN:
- kNN Distance: The feature vector's distance to a memory module of nominal feature vectors is computed. This kNN distance serves as an indicator of anomaly, as anomalous frames are expected to be farther from nominal ones.
- RNN for Temporal Dependency: To leverage temporal continuity, an RNN is used to process the sequence of kNN distances, capturing temporal dependencies for anomaly decision-making.
- Continual Learning Strategies: * Incremental Update: For smaller datasets, the memory module is incrementally updated with new nominal features.
- Experience Replay (k-DNN): For scalability, a k-Nearest-Neighbors Deep Neural Network (k-DNN) is trained to estimate kNN distances. Experience replay is used to update the k-DNN with both recent and past data, mitigating catastrophic forgetting.
Datasets
- Existing Datasets: UCSD Ped 2, CUHK Avenue, and ShanghaiTech. These are discussed as being insufficient for continual learning due to their static nature and limited scope.
- New Dataset: NOLA (New Orleans): A novel, comprehensive dataset is introduced for continual VAD. It consists of 110 training and 50 testing video segments captured over a week from a single moving camera in New Orleans. Key features include:
- Temporal Richness: Videos captured day and night, across different days of the week.
- Contextual Anomalies: Anomalies are contextual (e.g., loitering at night) and require deeper understanding.
- Scale: Significantly larger than existing datasets (990,000 training frames, 450,000 testing frames).
- Metadata: Includes day/time information crucial for contextual anomaly detection.
Key Insights and Ideas
- Real-world VAD requires continual learning: The static definition of "normal" in current VAD is a major bottleneck. Systems must adapt to evolving behaviors.
- Context is crucial for anomaly detection: Simply detecting unusual motion or objects isn't enough. The time of day, day of the week, and scene context are vital for correctly identifying anomalies.
- Frame-level metrics are insufficient: Evaluating VAD based on individual frames misses the point of detecting anomalous activities and doesn't account for detection delay.
- kNN distances can be effective features for continual learning: Using distances to a memory of nominal features, combined with an RNN, provides a robust way to detect anomalies while allowing for incremental updates.
- Existing datasets are "skewed": Simple optical flow methods perform surprisingly well on current benchmarks, indicating that these datasets don't pose a sufficiently challenging problem for advanced VAD techniques.
Key Results
- Outperformance on NOLA: The proposed continual learning algorithm significantly outperforms two state-of-the-art methods (Future Frame Prediction and MNAD) on the new NOLA dataset across multiple continual learning splits, achieving higher Average Precision Delay (APD) scores.
- Adaptation to Evolving Baselines: The proposed method shows improved performance over splits as it learns new nominal behaviors, while benchmark methods exhibit performance degradation, indicating catastrophic forgetting.
- Competitive on Existing Datasets: Even when evaluated using the traditional frame-level AUC metric on smaller benchmark datasets (UCSD Ped 2, ShanghaiTech), the proposed algorithm shows superior performance, particularly in scenarios where a new activity type is introduced as nominal.
- Real-time Capability: The pipeline can run at approximately 18 fps on an RTX 2070 GPU, with potential for further improvement.
Limitations and Future Work
- Limitations of Existing Datasets: The paper strongly emphasizes that existing datasets are not suitable for evaluating continual learning in VAD.
- Fail Cases: The authors acknowledge that their model can miss subtle, context-dependent anomalies that require a deeper understanding of intricate object relationships (e.g., a person carrying a snake, a staged car stop).
- Future Work:
- Multi-modal Learning: Leveraging both audio and video data for improved detection performance.
- Further Dataset Development: The NOLA dataset is intended to guide future research towards more practical and reproducible VAD solutions.
- Improving Contextual Understanding: Developing algorithms that can better grasp the intricate relationships between objects and their environment to detect more complex, context-dependent anomalies.
Summary updated