Abstract
Understanding anomalous human behaviors at a fine-grained level remains a major challenge in complex scenarios. Existing video anomaly understanding (VAU) methods often rely on coarse frame-level cues or overlook structured modeling of individual actions, limiting their capacity for reasoning about human interactions and accountability. To address these challenges, we propose TargetVAU, a multimodal anomaly-aware reasoning framework designed for individual-level anomaly recognition and explanation. TargetVAU first extracts both global-level and human-centric visual features using a frozen Vision Transformer (ViT) encoder. An Anomaly-focused Temporal Sampler is then employed to select behaviorally informative frames via a density-aware strategy guided by predicted anomaly scores. A Spatio-Temporal Interaction Graph is constructed to explicitly model interactions among individuals across time and space. These structured representations are fused with prompt embeddings via a frozen Q-Former to form a unified semantic representation. Finally, a large language model fine-tuned with low-rank adaptation (LoRA) performs instruction-guided reasoning to identify anomalous individuals and generate natural language explanations. Extensive experiments on UCCD and HIVAU-70K demonstrate that TargetVAU significantly outperforms existing methods in both accuracy and interpretability, advancing the state of individual-level anomaly understanding in surveillance videos.
Introduction: Beyond Simple Detection
In the domain of AI-driven surveillance, Video Anomaly Detection (VAD) has matured significantly, enabling models to flag unusual events. However, traditional VAD systems often operate as "black boxes," providing a single anomaly score without context. In complex scenarios, such as a crowded public space, simply knowing an anomaly has occurred is insufficient; security applications require knowing who is involved and what exactly is happening.
Existing Video Anomaly Understanding (VAU) methods attempt to bridge this gap but often suffer from a granularity problem, treating videos as global events and struggling with individual accountability and complex interpersonal interactions. To address this, we introduce the Anomalous Target Behavior Understanding (ATBU) task, which requires identifying specific individuals and providing semantic explanations for their actions. TargetVAU is our proposed framework to solve this challenge.
The TargetVAU Framework
TargetVAU utilizes a multi-stage pipeline designed to extract, sample, and reason over spatio-temporal data. The architecture is built upon several key components that allow it to move from raw pixels to structured reasoning.
Dual-Level Feature Extraction and Sampling
The model begins by extracting two types of features using a frozen Vision Transformer (ViT-L/14): global scene context and human-centric features derived from individual person detections. To manage computational costs and reduce noise, we employ an Anomaly-focused Temporal Sampler (AFTS). This module uses a lightweight scorer and a density-aware strategy to prioritize frames with a high likelihood of containing anomalous activity.
Spatio-Temporal Interaction Graph (STIG)
The core innovation of TargetVAU is the STIG, which explicitly models the relationships between individuals. By constructing a graph where nodes represent people and edges represent spatial proximity, temporal continuity, and semantic relationships (derived from video captions), the model can understand interactions—such as one person striking another—rather than just isolated movements.
Multimodal Reasoning
These structured graph representations are fused with textual prompts and fed into a Vicuna-7B large language model. Using Low-Rank Adaptation (LoRA), the LLM is fine-tuned to perform instruction-guided reasoning. This allows the system to output the specific ID of an offender and generate a natural language description of the event, answering the "who, what, and why" of the anomaly.
Experimental Results
TargetVAU was evaluated against several benchmarks, including UCCD, HIVAU-70K, UCF-Crime, and XD-Violence. The results demonstrate a significant leap in both detection and understanding capabilities.
- Superior Understanding: On the UCCD dataset, TargetVAU achieved a BLEU score of 0.582 and a CIDEr score of 0.665, substantially outperforming previous state-of-the-art models like Holmes-VAU.
- Robust Detection: Despite its focus on explainability, TargetVAU remains highly competitive in traditional detection tasks, achieving an 87.82% AP on XD-Violence and an 89.57% AUC on UCF-Crime.
- Qualitative Performance: Qualitative comparisons show that TargetVAU is significantly better at distinguishing between aggressors and bystanders in complex scenes compared to general-purpose models like Video-ChatGPT.
Conclusion
TargetVAU represents a significant advancement in video surveillance by shifting the focus from global anomaly detection to fine-grained individual behavior reasoning. By integrating spatio-temporal interaction graphs with the reasoning power of Large Language Models, the framework provides the accountability and explainability required for real-world security applications. Future work will focus on reducing the computational load of the LLM and improving the robustness of the initial person-detection phase.
Summary updated