In computer vision, detecting anomalies in video streams is fundamentally different from traditional classification. In a standard classification task, models learn to distinguish between well-defined classes. But an "anomaly" is defined by what it is not: it is anything that deviates from normal behavior. Because anomalies are rare, unpredictable, and infinitely varied, collecting a dataset that covers every possible abnormal event is practically impossible.
To solve this, researchers focus on normalcy modeling. The core intuition is that if a model can deeply understand the patterns, structures, and dynamics of a "normal" environment, it can flag any deviation as an anomaly. Early data-driven approaches, such as the real-time incremental learning framework proposed by René Schuster et al. in Real-time detection of unusual regions in image streams, laid the groundwork for this paradigm by learning normal appearances on the fly. Modern methods have evolved this intuition across three main frontiers: disentangling spatio-temporal features, optimizing supervision efficiency, and leveraging multimodal language models.
Disentangling Appearance and Motion
Why do standard neural networks struggle with video normalcy? A naive autoencoder trained on normal videos will minimize reconstruction error. However, deep networks are highly expressive; they often generalize so well that they accidentally reconstruct anomalous events perfectly, causing the anomaly to go unnoticed.
To prevent this over-generalization, modern frameworks explicitly separate and constrain how the model processes appearance (what things look like) and motion (how things move).
[Input Video] ---> [Appearance Encoder] ---> [Memory Bank Query] ---> [Reconstruction]
---> [Motion Encoder] ---> [Memory Bank Query] ---> [Reconstruction]
|
High Error = Anomaly
For instance, Zhiqiang Wu et al. introduced a memory-augmented approach in Multi-Level Memory-Augmented Appearance-Motion Correspondence Framework for Video Anomaly Detection. Instead of letting the network generate arbitrary outputs, the framework forces the model to query a fixed-capacity memory bank containing prototypical normal appearance and motion features. If an anomalous event occurs, the model cannot find a matching prototype in its memory, resulting in a high reconstruction error that exposes the anomaly.
Similarly, Wenhao Shao et al. leveraged a co-teaching strategy in Consistency-constrained unsupervised video anomaly detection framework based on Co-teaching. By enforcing consistency constraints between static appearance features and dynamic optical flow features, the model ensures that a normal event must make sense both structurally and temporally. If a fast-moving object appears in a zone where only slow movement is normal, the cross-feature inconsistency triggers a detection.
Beyond pixel-level features, understanding normalcy requires modeling interactions. Xianlin Zeng et al. proposed the Hierarchical Spatio-Temporal Graph Convolutional Neural Network (HSTGCNN) in A Hierarchical Spatio-Temporal Graph Convolutional Neural Network for Anomaly Detection in Videos. By treating moving entities as nodes in a graph, the model learns the normal grammar of human movements and social interactions, allowing it to detect behavioral anomalies that standard convolutional layers miss.
Drawing Boundaries with Limited Supervision
While unsupervised models learn purely from normal data, they can suffer from loose decision boundaries because they never see what an error looks like. Conversely, fully supervised models require tedious, frame-by-frame annotations. To bridge this gap, researchers design clever ways to sharpen the decision boundary using minimal or synthetic supervision.
One powerful approach is the synthesis of pseudo-anomalies. In Synthetic Pseudo Anomalies for Unsupervised Video Anomaly Detection: A Simple Yet Efficient Framework Based on Masked Autoencoder, Zhiqiang Wu et al. showed that by intentionally masking or corrupting normal video patches, they could train a model on "normalcy consistency." This forces the network to learn exactly what a realistic, uncorrupted video looks like, drawing a tight boundary around normal data without ever needing real anomaly samples.
When video-level labels are available but frame-level labels are missing (weak supervision), models must handle anomalies of varying durations. The framework in OE-CTST: Outlier-Embedded Cross Temporal Scale Transformer for Weakly-Supervised Video Anomaly Detection addresses this by using multi-scale temporal transformers and anomaly-aware position embeddings. This design allows the network to capture both sudden, short-lived anomalies (like a swift physical altercation) and prolonged abnormal events (like a vehicle loitering in a restricted zone).
To achieve an optimal balance between annotation cost and performance, Junxi Chen et al. introduced single-frame supervision in Video Anomaly Detection via Single Frame Supervision. Through Frame-guided Probabilistic Contrary Learning (FPCL), the model uses just one labeled abnormal frame per video to anchor its understanding, effectively pulling normal and abnormal feature distributions apart with minimal human effort.
The Multimodal Shift and Semantic Normalcy
Traditional video anomaly detection models output a binary anomaly score based on low-level feature deviations. However, the field is undergoing a paradigm shift toward semantic anomaly detection by integrating Large Language Models (LLMs). This allows models to not only detect where an anomaly occurs, but also explain what it is.
In Aligning Effective Tokens with Video Anomaly in Large Language Models, the authors introduced VA-GPT to solve the challenge of token explosion. Processing every video frame through an LLM is computationally prohibitive. VA-GPT solves this by selecting spatially and temporally effective visual tokens that align directly with the LLM's semantic space, achieving high-accuracy localization while processing fewer tokens.
[Raw Video Frames] ---> [Spatial/Temporal Token Selection] ---> [Aligned Visual Tokens] ---> [LLM Reasoner] ---> "Anomalous behavior detected: Loitering near exit"
Similarly, JunhoLee et al. proposed SIAD-LLM in Self-Improvement Anomaly Detection via Large Language Model for Unsupervised Zero-shot Anomaly Detection. By leveraging the zero-shot reasoning capabilities of LLMs, the framework generates interpretable anomaly maps without any prior training on anomalous examples.
This semantic grounding is further supported by datasets like PV-VTT, introduced by Ryozo Masukwa et al. in PV-VTT: A Privacy-Centric Dataset for Mission-Specific Anomaly Detection and Natural Language Interpretation, which pairs hierarchical graph networks with LLMs to generate natural language interpretations of video events at a fraction of the computational cost.
By shifting the focus from classifying rare anomalies to deeply understanding normalcy—whether through constrained spatio-temporal memories, self-supervised pseudo-anomalies, or multimodal token alignment—modern video anomaly detection frameworks achieve robust, real-time performance that scales efficiently to real-world surveillance and industrial monitoring.
Key Foundational Work
Real-Time Anomaly Detection With HMOF Feature (2018) — contributed 1 concept elements to this direction.
Real-time detection of unusual regions in image streams (2010) — contributed 1 concept elements to this direction.