Imagine an automated surveillance system flags a video clip with a high anomaly score. For a human operator, this numerical score is practically useless. Is the system alerting to a medical emergency, a security breach, or simply a camera glitch?
Traditional video anomaly detection (VAD) models operate as black boxes, outputting binary flags or continuous scores without context. To make these systems trustworthy and actionable, a major research direction has shifted from mere detection to explanation—a paradigm known as explainable video anomaly detection and event recounting. By forcing models to justify their decisions through semantic concepts, visual saliency, or natural language, researchers are bridging the gap between machine learning outputs and human reasoning.
Bridging the Semantic Gap with Event Recounting
Early video analysis models struggled with the "semantic gap"—the disconnect between low-level pixel features (like gradients or optical flow) and high-level human concepts (like "an accident" or "a robbery"). When a model relies purely on low-level features, it becomes impossible to trace why a specific frame triggered an alert.
To address this, early frameworks introduced the task of "multimedia event recounting." The core intuition is to introduce an intermediate layer of human-understandable concepts between the raw video data and the final classification score.
In the VISER system, Natarajan et al. (2012) and Natarajan et al. (2013) leveraged natural language processing to automatically discover semantic concepts from textual descriptions, fusing them with visual and audio features. Similarly, Liu et al. (2013) utilized a massive library of pre-trained visual and auditory concepts to map video segments to explicit semantic labels.
When the system detects an event, it does not just output a probability; it lists the specific high-level concepts (e.g., "people running," "glass shattering") that contributed to the score. This allows human operators to instantly verify the system's accuracy and understand the context of the event without manually reviewing hours of footage (Aly et al., 2013; Ishikawa et al., 2013).
Localizing Evidence in Space and Time
Identifying what concept triggered an alert is only half the solution; an operator also needs to know where and when the evidence occurred. Generating these spatiotemporal justifications can be computationally expensive if treated as a separate post-processing step.
To solve this efficiently, researchers developed methods to extract explanations directly from the internal representations of pre-trained networks. Bargal et al. (2018) proposed contrastive Excitation Backprop for RNNs (cEB-R), a method that utilizes a single top-down backward pass through a recurrent neural network to generate spatiotemporal saliency maps. By calculating which hidden units are excited by a specific output class, the model highlights the exact frames and spatial regions responsible for its decision without requiring explicit localization training.
[Raw Video Input] ---> [Deep Network Layers] ---> [Anomaly Score / Label]
^
| (Top-Down Saliency Pass)
[Spatiotemporal Heatmap]
(Shows exactly WHERE and WHEN)
Other approaches integrate this localization directly into the architecture. Hua et al. (2017) introduced the ER3 framework, which compresses redundant video frames into a "video imprint." This imprint naturally preserves spatial and temporal boundaries, allowing the model to generate visual heatmaps as an integrated byproduct of the recognition process. For structured, multi-step activities, frameworks like SVGraph (2022) map these spatiotemporal relationships into self-supervised semantic graphs, making the sequential logic of a video explicit and readable.
Deconstructing Anomalies via Explicit Attributes
As deep learning architectures grew more complex, their implicit priors became harder to debug, increasing the risk that a model might flag an anomaly based on spurious correlations (such as a change in lighting rather than a suspicious action). One powerful countermeasure is to completely bypass black-box feature representations in favor of explicit, human-defined attributes.
Hoshen et al. (2025) demonstrated this intuition by breaking down video objects into three interpretable attributes: velocity, pose, and deep visual features. Instead of feeding raw video into an end-to-end neural network, their method extracts these specific attributes and applies basic density estimation.
If an object is flagged as anomalous, the operator can look directly at the underlying density metrics: was it moving too fast (velocity anomaly), or was it in an unusual position (pose anomaly)? This makes the system trivial to debug and refine for real-world deployment.
When deep architectures are necessary to capture complex scenes, temporal constraints can be used to isolate anomalies. Shao et al. (2023) utilized a Temporal Convolutional Network within a Multi-Instance Learning framework to explicitly disentangle rare abnormal events from dominant normal patterns. By forcing the network to model long-range temporal correlations, the architecture prevents the model from conflating background noise with actual anomalies, allowing it to isolate and explain multiple distinct sources of anomalous behavior simultaneously (Szymanowicz et al., 2021).
The Language-Driven Era: Causation and Reasoning
The latest evolution in explainable VAD leverages the cognitive and reasoning capabilities of Large Language Models (LLMs) and Vision-Language Models (VLMs). Instead of relying on predefined concept checklists or abstract heatmaps, modern systems generate full natural language descriptions and logical justifications.
In safety-critical domains like autonomous driving, providing a text description of an anomaly is vital for understanding vehicle behavior. Tiwari et al. (2024) demonstrated that weakly supervised frameworks can successfully generate natural language descriptions of driving accidents from ego-centric cameras, moving beyond binary collision alerts.
To move from mere description to actual logical deduction, Lee et al. (2024) introduced the AnomalyRuler framework. This system uses an LLM to induce explicit rules from a few normal video frames (e.g., "vehicles should only move forward on this road"). When an anomaly occurs, the system detects it by identifying a rule violation and explains the event by citing the specific rule that was broken, mimicking human logical deduction.
[Normal Frames] ---> [LLM Rule Induction] ---> "Rule: Vehicles must move forward"
|
[Test Video] ---> [Violation Detection] ---> "Alert: Vehicle moving backward"
To evaluate this new generation of reasoning models, Du et al. (2024) established the CUVA (Causation Understanding of Video Anomaly) benchmark. CUVA explicitly tests a model's ability to explain the "why" and "how" of unusual events, moving the field toward true causal understanding.
To handle these complex reasoning tasks across long video sequences without overwhelming computational budgets, Zhang et al. (2025) developed Holmes-VAU. This framework uses an Anomaly-focused Temporal Sampler to efficiently zero in on critical frames, allowing a VLM to generate detailed, multi-granularity textual explanations of anomalies across extended timelines.
Key Foundational Work
BBN VISER TRECVID 2013 Multimedia Event Detection and Multimedia Event Recounting Systems (2013) — contributed 1 concept elements to this direction.
SRI-Sarnoff AURORA System at TRECVID 2013 Multimedia Event Detection and Recounting (2013) — contributed 1 concept elements to this direction.
BBNVISER : BBN VISER TRECVID 2012 Multimedia Event Detection and Multimedia Event Recounting Systems (2012) — contributed 1 concept elements to this direction.