Abstract
Video Anomaly Understanding (VAU) has traditionally focused on frame-level anomaly scores, lacking the ability to provide hierarchical reasoning or explain the "why" behind detections. We present Holmes-VAU, a framework designed for long-term video anomaly understanding across multiple temporal scales. By introducing the HIVAU-70k dataset—a large-scale benchmark with over 70,000 hierarchical annotations—and an Anomaly-focused Temporal Sampler (ATS), our approach enables multimodal large language models (MLLMs) to efficiently process long surveillance footage. Holmes-VAU achieves state-of-the-art performance in both anomaly detection and reasoning quality, bridging the gap between low-level perception and high-level causal analysis.
Introduction
Traditional Video Anomaly Understanding (VAU) has largely operated as a "black box," where models output frame-level anomaly scores without providing interpretable reasoning. This approach suffers from two primary limitations. First, there is a granularity gap: anomalies can range from brief, clip-level incidents (e.g., a single punch) to complex, video-level events (e.g., a prolonged riot). Existing datasets lack the hierarchical labels necessary to connect these scales. Second, an efficiency bottleneck exists when processing long-duration surveillance footage. Uniformly sampling frames often misses critical, brief anomalous moments—the "needle in the haystack."
To address these challenges, we propose Holmes-VAU, a system that shifts the focus toward a deep, hierarchical understanding of what is happening in a video and why.
Methodology
The Holmes-VAU framework introduces two key innovations to enable efficient and nuanced anomaly understanding:
Anomaly-focused Temporal Sampler (ATS)
Rather than employing fixed-interval sampling, the ATS module utilizes a lightweight anomaly scorer to identify "suspicious" temporal regions. It then applies a density-aware sampling strategy, concentrating more frames on these anomaly-rich segments while retaining sufficient global context to maintain the overall narrative flow of the video. This allows the model to capture critical details without the computational overhead of processing every frame.
Hierarchical Instruction Tuning
The model, built on the InternVL2-2B architecture, is fine-tuned using a specialized semi-automated "Data Engine." This pipeline involves human segmentation of videos, followed by clip caption generation using off-the-shelf models (e.g., LLaVA-Next-Video). A powerful LLM (Llama 3) then summarizes these into event-level and video-level descriptions, which are finally verified by human annotators to ensure high quality and factual accuracy.
The HIVAU-70k Dataset
A core contribution of this work is HIVAU-70k, a massive benchmark constructed from the UCF-Crime and XD-Violence datasets. It contains over 70,000 multi-granular annotations organized into three levels:
- Clip-level: Focuses on short-term visual perception (e.g., "A man is running").
- Event-level: Involves reasoning about specific incidents (e.g., "A collision between a car and a bus").
- Video-level: Provides a global summary and causal analysis of all events within the footage.
The dataset explicitly separates tasks into Judgment (detection), Description (narrative), and Analysis (causal reasoning), helping the model avoid hallucinations and improve explainability.
Key Results
Holmes-VAU establishes new state-of-the-art (SOTA) benchmarks in both detection and reasoning tasks:
- Detection Performance: The model achieved 87.68% Average Precision (AP) on the XD-Violence dataset and 88.96% AUC on UCF-Crime, outperforming both explainable and non-explainable predecessors.
- Reasoning Quality: In comparisons against models like Video-ChatGPT and InternVL2, Holmes-VAU demonstrated significant gains in text metrics. For instance, its CIDEr score at the event level reached 1.519, a vast improvement over InternVL2’s 0.022.
- Sampling Efficiency: The ATS module proved essential for long-video performance. Holmes-VAU maintained high accuracy even when sampling as few as 8 or 16 frames, whereas uniform sampling performance degraded significantly at these low frame counts.
Conclusion
Holmes-VAU demonstrates that training on multiple temporal granularities is synergistic: clip-level data enhances visual perception, while event and video-level data empower complex causal reasoning. While the system is not yet optimized for real-time streaming and currently lacks audio integration, it represents a significant step toward transparent and comprehensive video anomaly understanding. Future work will explore multi-modal cues (e.g., audio) and expansion into diverse domains such as industrial and medical monitoring.
Summary updated