TL;DR
Most video anomaly detection (AD) research focuses on surveillance—spotting a bike on a pedestrian path. This paper shifts the focus toward physical and mechanical anomalies (like a malfunctioning pendulum or an irregular candle flicker) by introducing the PHANTOM dataset. The authors demonstrate that while specialized surveillance models fail on these physical tasks, simple baselines using pretrained image and video features can effectively bridge the gap.
What Problem Does This Paper Address?
Anomaly detection in video is a mature field, but it suffers from a "surveillance bias." Current benchmarks almost exclusively feature security camera footage where "normal" is walking and "anomalous" is any other activity (skating, biking, etc.).
The authors argue that this narrow focus ignores a massive range of practical applications, such as detecting mechanical malfunctions in manufacturing or identifying subtle physical irregularities in real-world settings. These "physical anomalies" often involve changes in trajectory, motion, or procedural steps rather than just the presence of an unusual object. The challenge is that models trained for surveillance don't necessarily understand the "laws of physics" or the specific dynamics required to spot these subtle mechanical failures.
Methods
The researchers evaluate two straightforward baseline approaches based on the assumption that "normal" data forms a dense cluster in a high-dimensional feature space:
- Image-features-based AD: They use networks pretrained for image classification (like ResNet and Vision Transformer (ViT)). They extract features from individual frames, apply temporal pooling (average or max) to create a single vector for the video, and then use k-Nearest Neighbors (kNN) to calculate an "abnormality score" based on distance from the training set.
- Video-features-based AD: They use networks pretrained on video datasets (like TimeSformer or bLVNet-TAM). These models are designed to capture temporal dynamics directly. Again, they use kNN on the resulting embeddings to detect outliers.
They compare these simple baselines against MNAD (Memory-guided Normality for Anomaly Detection), a state-of-the-art method specifically designed for surveillance videos.
Datasets
The paper utilizes several benchmarks to test generality:
- PHANTOM (Physical Anomalous Trajectory or Motion): A new dataset introduced in this paper. it contains six classes: Candle, Keyboard, Magnets, Pendulum, Sushi, and Window. It features anomalies like a candle flickering unnaturally, a pendulum being manipulated, or a window shutter stuttering.
- Something-Something-V2 (SSv2): A large-scale dataset of humans performing basic actions. The authors adapted 10 classes to create a "General Activity" AD benchmark to test performance on highly variable scenes.
- UCSD Pedestrian 1 & 2: Standard surveillance benchmarks used to show how the new methods compare to traditional AD tasks.
Key Insights and Ideas
- The Domain Gap is Real: Models that dominate surveillance benchmarks (like MNAD) perform poorly on physical anomalies. On the PHANTOM dataset, MNAD achieved an average ROCAUC of only 0.55 (barely better than a coin flip), while the simple ViT+kNN baseline reached 0.78.
- Images vs. Video Features: There is a trade-off between spatial and temporal modeling. Image-based features (ViT) performed better on anomalies involving subtle local changes (like the Candle or Magnets classes). Video-based features (TimeSformer) excelled when the anomaly was defined by a change in trajectory (like the Pendulum class).
- Simplicity Wins (For Now): The fact that a basic kNN approach on pretrained features outperforms specialized AD architectures suggests that current "state-of-the-art" AD models are over-fitted to the specific characteristics of surveillance footage.
Key Results
- PHANTOM Dataset: The ViT (Vision Transformer) with average pooling was the top performer with an average ROCAUC of 0.91 across the six classes, significantly outperforming the surveillance-oriented MNAD (0.55).
- Surveillance Performance: Conversely, on the UCSD Pedestrian 2 dataset, MNAD scored 97.0, while the authors' best baseline only reached 75.0. This confirms that surveillance AD and physical AD are distinct tasks requiring different approaches.
- General Activity (SSv2): This proved to be the hardest task. The best performing method (ViT) only achieved a 0.58 ROCAUC, highlighting that high scene variability (different backgrounds, lighting, and objects) remains a major hurdle for general video AD.
Limitations and Future Work
The primary limitation is the kNN baseline's struggle with high variability. While it works well on the relatively controlled environments of the PHANTOM dataset, it falters on the SSv2 dataset where the background and objects change constantly.
The authors suggest that future work should focus on developing "General Video AD" methods that can handle both the subtle physical dynamics of PHANTOM and the high variability of real-world activities. They hope the PHANTOM dataset serves as a stepping stone toward models that truly understand "normality" in a physical sense, rather than just memorizing surveillance patterns.
Summary updated