TL;DR
This paper introduces CADE, a novel approach to video anomaly detection (VAD) that addresses the challenge of evolving data distributions (domain shift) by combining continual learning (CL) with weakly-supervised VAD (WVAD). CADE uses a dual-generator system to handle imbalanced and uncertain data, and a multi-discriminator ensemble to combat "forgetting" and improve detection accuracy, outperforming existing methods on challenging multi-scene datasets.
What Problem Does This Paper Address?
Video anomaly detection (VAD) is crucial for public security, aiming to identify unusual events in surveillance footage. While weakly-supervised VAD (WVAD) methods have gained traction due to reduced annotation costs, they often assume static datasets. In real-world scenarios, surveillance data can change significantly over time due to factors like varying environments, locations, or even seasons. This "domain shift" poses a major challenge: simply retraining models on new data can lead to "catastrophic forgetting," where the model loses its ability to detect anomalies from previous data distributions. Furthermore, WVAD methods can suffer from "incompleteness," where models become biased towards certain anomaly types and miss others. This paper tackles the problem of continual WVAD in the face of evolving data domains, aiming to maintain high detection performance without forgetting past knowledge and to address the incompleteness issue.
Methods
CADE is built upon three key components designed to address the challenges of continual WVAD:
- Dual-Generator (DG): Existing generative replay (GR) methods for CL struggle with the imbalanced and noisy nature of WVAD data. CADE's DG, inspired by DMVAE, uses separate generators for normal and abnormal data. This allows for better handling of data imbalance and captures richer latent representations, improving the quality of replayed samples for the discriminator.
- Multi-Discriminator (MD) Ensemble: To combat the "incompleteness" issue where models might miss certain anomaly types due to forgetting, CADE employs an ensemble of multiple discriminators. These discriminators are trained to be diverse, helping to capture a wider range of anomalies.
- Inference-Time Ensemble: The final anomaly score for a video segment is an average of the scores from the multiple discriminators. This ensemble approach leverages the strengths of individual discriminators to provide a more robust and comprehensive anomaly detection.
CADE integrates these components within a continual learning framework, specifically adapting them to the Multiple Instance Learning (MIL) setup commonly used in WVAD.
Datasets
The proposed CADE method is evaluated on three multi-scene datasets suitable for realistic VAD applications with changing scenes:
- ShanghaiTech (SHT): A medium-sized dataset with videos from 13 different scenes, used to simulate domain shifts by treating each scene as a domain.
- Charlotte Anomaly Dataset (CHAD): Contains videos from four different scenes, with each scene treated as a domain.
- UCF-Crime: A larger dataset with 1,610 training videos. For DIL evaluation, video-level CLIP embeddings are clustered into 10 scene-like domains, and these domains are introduced sequentially.
Key Insights and Ideas
- Continual Learning is Essential for Real-World VAD: The paper strongly argues that static datasets are insufficient for VAD in dynamic surveillance environments. The ability to learn continuously without forgetting is paramount.
- Generative Replay Needs Adaptation for WVAD: Standard GR methods are not directly suitable for WVAD due to data imbalance and label uncertainty. CADE's Dual-Generator approach is a crucial adaptation to make GR effective in this context.
- Ensembling Discriminators Solves Incompleteness: The "incompleteness" problem, where models become biased and miss anomalies, is effectively addressed by ensembling multiple discriminators. This highlights the power of diversity in anomaly detection.
- CADE is a General Framework: The authors emphasize that CADE can be integrated with various existing WVAD methods, suggesting its broad applicability.
Key Results
CADE demonstrates significant improvements across all tested datasets:
- Superior Performance: CADE consistently outperforms fine-tuning (FT) and other continual learning baselines (EWC, SI, iCaRL) on SHT, CHAD, and UCF-Crime datasets.
- Effectiveness on SHT: CADE achieves an AUC of 0.8490±0.0057 on SHT, significantly outperforming FT.
- Effectiveness on CHAD: CADE achieves an AUC of 0.7674±0.0058 on CHAD, showing its robustness across different datasets.
- Effectiveness on UCF-Crime: CADE achieves higher AUROC than representative CL baselines, with RTFM+CADE reaching 0.800±0.003.
- Forgetting Prevention: Qualitative evaluations show that CADE's sensitivity to anomalies in past scenes decreases much less than fine-tuned methods, indicating effective forgetting prevention.
- Ablation Studies: Combining all components of CADE (DG, MD, and distance loss) yields the best results, with DG alone providing a substantial improvement. The MD and distance loss further enhance performance, demonstrating their individual contributions.
Limitations and Future Work
- Comparison to MTL: While CADE performs well in a Domain Incremental Learning (DIL) setup, it underperforms Multi-Task Learning (MTL) on SHT. The authors acknowledge that MTL, which trains on all domains simultaneously, acts as an oracle upper bound and does not follow the DIL setup.
- Replay Size Optimization: While experiments on replay size were conducted (Table 4), further exploration into optimal replay strategies for different datasets and scenarios could be beneficial.
- Future Work: The paper suggests that CADE's architecture can be applied to other WVAD methods, implying future work could involve integrating CADE with a wider range of WVAD techniques. Further research into more sophisticated ensemble strategies or exploring different generative replay mechanisms could also be promising directions.