TL;DR
This paper introduces REWARD, the first end-to-end trained algorithm for real-time weakly supervised video anomaly detection (wVAD). By leveraging self-supervised learning and a novel kNN-based approach, REWARD significantly improves detection speed and accuracy compared to existing methods, achieving high performance with a drastically reduced decision period.
What Problem Does This Paper Address?
Video anomaly detection is crucial for applications like surveillance, where identifying unusual events in real-time is essential for timely response. Weakly supervised video anomaly detection (wVAD) presents a specific challenge: training data includes videos labeled as either "normal" or "anomalous" at the video level, but without precise frame-level annotations indicating when the anomaly occurs. Existing state-of-the-art wVAD methods often rely on pre-trained feature extractors and complex feature aggregation techniques. While effective for offline detection (analyzing the entire video after it has been recorded), these methods struggle with real-time performance because their decision-making process requires processing a significant number of frames, leading to long decision periods. This paper aims to bridge this gap by developing an end-to-end trainable model that can achieve both high accuracy and real-time detection capabilities in wVAD.
Methods
The core contribution of this paper is the REWARD (Real-Time End-to-End Weakly Supervised Video Anomaly Relevance Detector) algorithm. REWARD's novelty lies in its end-to-end training approach, which is made possible by a novel self-supervised learning mechanism.
Here's a breakdown of the key components:
- End-to-End Training: Unlike previous methods that train only parts of the network or rely on pre-trained feature extractors, REWARD trains a large video model (specifically, a Transformer-based model like Uniformer) from scratch for the anomaly detection task. This allows the model to learn task-specific representations directly from raw video data, leading to improved efficiency and performance.
- Self-Supervised Learning with kNN: To overcome the limitations of metric learning losses (which require large batch sizes and are difficult to train end-to-end), REWARD uses a three-step self-supervised process to generate pseudo-labels for training:
- kNN Distance Calculation: Features from anomalous videos are compared to features from normal videos using Euclidean distance. Segments with smaller distances are considered more similar to normal behavior.
- Smoothing and Anomaly Evidence: A smoothing step is applied to account for temporal continuity, followed by a cumulative sum and ReLU activation to generate "anomaly evidence" for each segment. Segments with higher evidence are potential anomalies.
- MLP and Final Selection: An MLP classifier is trained on these initial anomaly evidences to refine the selection of anomalous segments. This refined set of pseudo-labels is then used to train the main video model end-to-end.
- Simplified Pipeline: By training end-to-end, REWARD eliminates the need for complex feature aggregation modules, simplifying the inference pipeline and enabling faster processing.
Datasets
The paper evaluates REWARD on two widely used wVAD datasets:
- UCF-Crime: Contains 1900 surveillance videos covering 13 types of real-world anomalies. The training set has 1610 videos (810 nominal, 800 anomalous), and the test set has 290 videos.
- XD-Violence: A more recent and larger dataset with 4754 untrimmed videos (including audio, though only video information was used). It features diverse sources like surveillance footage, movies, and dash-cam recordings. The training set includes over 1900 anomalous and over 2000 nominal videos.
Key Insights and Ideas
- End-to-End is Key for Real-Time wVAD: The paper strongly demonstrates that achieving real-time performance in wVAD is fundamentally linked to end-to-end training. By avoiding feature aggregation and refinement steps, the model can process information much faster.
- Self-Supervision as a Bridge: The novel kNN-based self-supervised learning approach is crucial. It effectively generates reliable pseudo-labels from video-level annotations, enabling the end-to-end training of powerful Transformer models without requiring frame-level ground truth.
- Transformer Models Shine in End-to-End wVAD: The use of modern Transformer architectures (like Uniformer) in an end-to-end fashion proves highly effective, outperforming methods that rely on older 3D-CNNs or only fine-tune parts of the network.
- The Trade-off Between Decision Period and Accuracy: The paper visually and quantitatively illustrates how significantly performance degrades for existing methods as the decision period (number of frames processed at once) decreases. REWARD breaks this trend.
- Handling Data Heterogeneity: The paper acknowledges that datasets like XD-Violence, with their dynamic camera views and diverse sources, pose greater challenges than more static datasets like UCF-Crime. REWARD's end-to-end approach seems robust to this.
Key Results
REWARD achieves state-of-the-art performance in real-time wVAD:
- UCF-Crime: Achieves 86.94% AUC with a decision period of 6.4 seconds. This is a significant improvement over competing methods that achieve at most 85.92% AUC with a decision period of 273 seconds.
- XD-Violence: Achieves 77.71% AP with a decision period of 6.4 seconds, outperforming existing methods by a notable margin.
- Computational Efficiency: REWARD processes at 63.3 fps on a Nvidia RTX-2070 GPU, demonstrating its real-time inference capability.
Limitations and Future Work
- Feature Extractor Dependency: The paper notes that REWARD's performance is highly dependent on the chosen feature extractor. While Uniformer-32 was used, exploring other advanced feature extractors could yield further improvements.
- Banner Artifacts: During self-supervision on UCF-Crime, the presence of "banners" in some anomaly videos led to artificially high anomaly scores. The authors mitigated this by dropping initial and final segments during training, but it highlights a potential challenge in real-world data.
- Longer Videos and Decision Periods: While REWARD excels at a fixed short decision period, the paper also briefly explores offline performance with longer segments (32 segments, ~1024 frames). This suggests that further research could explore adaptive decision periods or methods that better leverage longer temporal contexts for even more robust detection.
- Impact of Feature Aggregation: The paper shows that omitting feature aggregation significantly degrades performance for existing methods, reinforcing the benefit of REWARD's end-to-end approach. Future work could explore how to integrate some form of aggregation or refinement within an end-to-end framework.
Summary updated