Approaches Toward Physical and General Video Anomaly Detection

Laura Kart, Niv Cohen

Abstract

In recent years, many works have addressed the problem of finding never-seen-before anomalies in videos. Yet, most work has been focused on detecting anomalous frames in surveillance videos taken from security cameras. Meanwhile, the task of anomaly detection (AD) in videos exhibiting anomalous mechanical behavior, has been mostly overlooked. Anomaly detection in such videos is both of academic and practical interest, as they may enable automatic detection of malfunctions in many manufacturing, maintenance, and real-life settings. To assess the potential of the different approaches to detect such anomalies, we evaluate two simple baseline approaches: (i) Temporal-pooled image AD techniques. (ii) Density estimation of videos represented with features pretrained for video-classification. Development of such methods calls for new benchmarks to allow evaluation of different possible approaches. We introduce the Physical Anomalous Trajectory or Motion (PHANTOM) dataset, which contains six different video classes. Each class consists of normal and anomalous videos. The classes differ in the presented phenomena, the normal class variability, and the kind of anomalies in the videos. We also suggest an even harder benchmark where anomalous activities should be spotted on highly variable scenes.

Introduction

Video anomaly detection (AD) has traditionally been synonymous with surveillance—identifying unexpected objects or actions, such as a bicycle on a pedestrian path, within a static background. While this focus has led to significant progress, it creates a "surveillance bias" that neglects a broad spectrum of physical and mechanical anomalies. These include malfunctioning machinery, irregular physical processes, or procedural errors in industrial settings. Such anomalies are often defined by subtle changes in trajectory, motion, or temporal dynamics rather than the mere presence of an unusual object.

This paper addresses this gap by shifting the focus toward physical and mechanical anomalies. The authors argue that models trained for surveillance do not necessarily understand the "laws of physics" or the specific dynamics required to spot subtle mechanical failures. By introducing the Physical Anomalous Trajectory or Motion (PHANTOM) dataset, they demonstrate that specialized surveillance models often fail on these tasks, whereas simple baselines leveraging pretrained image and video features can effectively bridge the performance gap.

Methodology

The researchers evaluate two straightforward baseline approaches grounded in the assumption that "normal" data forms a dense cluster in a high-dimensional feature space.

  1. Image-Features-Based AD: This approach utilizes networks pretrained for image classification, such as ResNet and Vision Transformer (ViT). Features are extracted from individual frames and aggregated using temporal pooling (average or max) to form a single video-level vector. A k-Nearest Neighbors (kNN) algorithm then calculates an abnormality score based on the distance to the training set.
  2. Video-Features-Based AD: This method employs networks specifically pretrained on video datasets, such as TimeSformer or bLVNet-TAM. These models are inherently designed to capture temporal dynamics directly. Similar to the image-based approach, kNN is applied to the resulting embeddings to identify outliers.

These baselines are compared against MNAD (Memory-guided Normality for Anomaly Detection), a state-of-the-art model specifically designed for surveillance benchmarks.

The PHANTOM Dataset

To facilitate research in this new domain, the authors introduce the PHANTOM dataset. It comprises six distinct classes representing various physical phenomena: Candle, Keyboard, Magnets, Pendulum, Sushi, and Window.

The six classes of the PHANTOM dataset: Candle, Keyboard, Sushi, Magnets, Pendulum, and Window.

The dataset captures anomalies such as unnatural candle flickering, irregular pendulum motion, or stuttering window shutters. Unlike surveillance datasets where the background is typically static and the "anomaly" is an external object, PHANTOM focuses on the intrinsic dynamics of the objects themselves. Additionally, the authors utilize the Something-Something-V2 (SSv2) dataset to create a "General Activity" benchmark, testing performance in highly variable scenes with diverse backgrounds and lighting.

Experimental Results and Insights

The experiments reveal a significant domain gap between surveillance-based and physical anomaly detection. MNAD, which excels on surveillance benchmarks like UCSD Pedestrian 2 (scoring 97.0 ROCAUC), performed poorly on PHANTOM with an average ROCAUC of only 0.55—barely better than a coin flip. In contrast, the simple ViT+kNN baseline reached an average ROCAUC of 0.91 across the six classes.

Key insights from the evaluation include:

  • Spatial vs. Temporal Modeling: There is a trade-off between spatial and temporal modeling. Image-based features (ViT) were superior for anomalies involving subtle local changes, such as in the Candle or Magnets classes. Video-based features (TimeSformer) excelled when the anomaly was defined by a change in trajectory, as seen in the Pendulum class.
  • Overfitting in Surveillance Models: The failure of specialized AD architectures on PHANTOM suggests they are heavily over-fitted to the specific characteristics of surveillance footage.
  • The Challenge of Variability: The General Activity benchmark (SSv2) proved the most difficult task. The best performing method (ViT) achieved only a 0.58 ROCAUC, highlighting that high scene variability remains a major hurdle for general-purpose video AD.

Conclusion

The PHANTOM dataset serves as a critical stepping stone toward models that truly understand "normality" in a physical sense. While simple kNN baselines on pretrained features offer a strong starting point, they struggle with the high variability found in general real-world activities. Future work should focus on developing "General Video AD" methods that can handle both the subtle physical dynamics of PHANTOM and the high variability of real-world activities.

Summary updated

Abstract

In recent years, many works have addressed the problem of finding never-seen-before anomalies in videos. Yet, most work has been focused on detecting anomalous frames in surveillance videos taken from security cameras. Meanwhile, the task of anomaly detection (AD) in videos exhibiting anomalous mechanical behavior, has been mostly overlooked. Anomaly detection in such videos is both of academic and practical interest, as they may enable automatic detection of malfunctions in many manufacturing, maintenance, and real-life settings. To assess the potential of the different approaches to detect such anomalies, we evaluate two simple baseline approaches: (i) Temporal-pooled image AD techniques. (ii) Density estimation of videos represented with features pretrained for video-classification. Development of such methods calls for new benchmarks to allow evaluation of different possible approaches. We introduce the Physical Anomalous Trajectory or Motion (PHANTOM) dataset, which contains six different video classes. Each class consists of normal and anomalous videos. The classes differ in the presented phenomena, the normal class variability, and the kind of anomalies in the videos. We also suggest an even harder benchmark where anomalous activities should be spotted on highly variable scenes.
BibTeX
@inproceedings{kart2022approaches,
  author = {Laura Kart and Niv Cohen},
  title = {{Approaches Toward Physical and General Video Anomaly Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=rsth9AyoFY},
}
Figure