TL;DR
This paper introduces PANDA, an "agentic AI engineer" for video anomaly detection (VAD) that can adapt to new scenarios and anomalies without any prior training or manual intervention. By combining scene-aware planning, goal-driven reasoning, tool-augmented reflection, and a memory system, PANDA achieves state-of-the-art performance on complex, open-set VAD tasks, paving the way for more generalizable and robust surveillance systems.
What Problem Does This Paper Address?
Video anomaly detection (VAD) is crucial for applications like surveillance, traffic monitoring, and autonomous driving. However, existing VAD methods are largely "specialist-oriented." This means they either require extensive domain-specific training data and manual adjustments for new scenarios and anomaly types, leading to high costs and poor generalization, or they rely on pre-trained models but still demand significant manual engineering for deployment. This "brittle" nature limits their effectiveness in the diverse and dynamic real world. The authors aim to overcome these limitations by developing a "generalist VAD" system that can autonomously adapt and perform effectively without human input or retraining.
Methods
PANDA is designed as an "agentic AI engineer" with four core capabilities:
- Self-adaptive Scene-Aware Strategy Planning: PANDA first perceives the input video to understand the scene context. It then uses a Retrieval Augmented Generation (RAG) mechanism to retrieve relevant anomaly rules from a knowledge base and constructs a tailored detection strategy.
- Goal-Driven Heuristic Reasoning: PANDA employs a multimodal large language model (MLLM) to reason about the video content, guided by task-specific prompts and anomaly rules. It outputs a score, status (normal/abnormal/insufficient), and a reason.
- Tool-Augmented Self-Reflection: When the reasoning is "insufficient," PANDA enters a reflection phase. It analyzes the cause of uncertainty and invokes a suite of tools (e.g., image enhancement, object detection, web search) to gather more information and refine its decision.
- Self-Improving Chain-of-Memory (CoM): PANDA maintains both short-term and long-term memory of its reasoning and reflection processes. This allows it to learn from past experiences, improving its decision-making consistency and accuracy over time.
Datasets
PANDA was evaluated on four benchmarks:
- UCF-Crime: A large-scale dataset with 13 types of abnormal events.
- XD-Violence: Focuses on violence detection with 6 anomaly categories.
- UBnormal: A synthetic open-set dataset with 22 anomaly types, including 12 unseen during training, to test generalization.
- CSAD: A new complex-scene anomaly detection benchmark introduced by the authors, featuring challenging conditions like low resolution and noise.
Key Insights and Ideas
- The "Agentic AI Engineer" Paradigm: The core idea is to mimic how a human engineer would approach a problem: perceive, plan, reason, reflect, and learn. This agentic approach allows for dynamic adaptation rather than static, pre-programmed responses.
- RAG for Anomaly Knowledge: Using Retrieval Augmented Generation to dynamically pull relevant anomaly rules based on scene context is crucial for adapting to new scenarios without retraining.
- Tool Use for Ambiguity Resolution: The self-reflection mechanism, empowered by a diverse set of tools, is key to handling the inherent uncertainty in VAD, especially in complex or low-quality videos.
- Chain-of-Memory for Continuous Improvement: The CoM mechanism is vital for PANDA to become "smarter" over time, leveraging past experiences to inform current decisions and ensure consistency.
Key Results
PANDA achieved state-of-the-art performance across all four datasets, significantly outperforming existing training-free baselines.
- On UCF-Crime, PANDA achieved an AUC of 84.89%.
- On XD-Violence, it achieved an AP of 70.16%.
- On UBnormal, it demonstrated strong generalization with an AUC of 75.78%.
- On the challenging CSAD benchmark, it achieved an AUC of 73.12%. Crucially, PANDA achieved these results without any training or manual involvement, highlighting its generalizability and robustness. Ablation studies showed that each of PANDA's four core capabilities contributed significantly to its overall performance.
Limitations and Future Work
The authors acknowledge that PANDA currently integrates a "curated set of commonly used tools." They suggest expanding this toolkit to include domain-specific modalities (e.g., thermal imaging) to broaden applicability. Future work will focus on:
- Enhancing Real-Time Adaptability: Reducing inference latency and optimizing tool invocation paths for practical deployment.
- Improving Spatial Reasoning and Localization: Moving beyond frame/clip-level detection to pinpoint the precise regions or objects involved in anomalies.
Summary updated