Approaches Toward Physical and General Video Anomaly Detection

Laura Kart, Niv Cohen

TL;DR

Most video anomaly detection (AD) research focuses on surveillance—spotting a bike on a pedestrian path. This paper shifts the focus toward physical and mechanical anomalies (like a malfunctioning pendulum or an irregular candle flicker) by introducing the PHANTOM dataset. The authors demonstrate that while specialized surveillance models fail on these physical tasks, simple baselines using pretrained image and video features can effectively bridge the gap.

What Problem Does This Paper Address?

Anomaly detection in video is a mature field, but it suffers from a "surveillance bias." Current benchmarks almost exclusively feature security camera footage where "normal" is walking and "anomalous" is any other activity (skating, biking, etc.).

The authors argue that this narrow focus ignores a massive range of practical applications, such as detecting mechanical malfunctions in manufacturing or identifying subtle physical irregularities in real-world settings. These "physical anomalies" often involve changes in trajectory, motion, or procedural steps rather than just the presence of an unusual object. The challenge is that models trained for surveillance don't necessarily understand the "laws of physics" or the specific dynamics required to spot these subtle mechanical failures.

Methods

The researchers evaluate two straightforward baseline approaches based on the assumption that "normal" data forms a dense cluster in a high-dimensional feature space:

  1. Image-features-based AD: They use networks pretrained for image classification (like ResNet and Vision Transformer (ViT)). They extract features from individual frames, apply temporal pooling (average or max) to create a single vector for the video, and then use k-Nearest Neighbors (kNN) to calculate an "abnormality score" based on distance from the training set.
  2. Video-features-based AD: They use networks pretrained on video datasets (like TimeSformer or bLVNet-TAM). These models are designed to capture temporal dynamics directly. Again, they use kNN on the resulting embeddings to detect outliers.

They compare these simple baselines against MNAD (Memory-guided Normality for Anomaly Detection), a state-of-the-art method specifically designed for surveillance videos.

Datasets

The paper utilizes several benchmarks to test generality:

  • PHANTOM (Physical Anomalous Trajectory or Motion): A new dataset introduced in this paper. it contains six classes: Candle, Keyboard, Magnets, Pendulum, Sushi, and Window. It features anomalies like a candle flickering unnaturally, a pendulum being manipulated, or a window shutter stuttering.
  • Something-Something-V2 (SSv2): A large-scale dataset of humans performing basic actions. The authors adapted 10 classes to create a "General Activity" AD benchmark to test performance on highly variable scenes.
  • UCSD Pedestrian 1 & 2: Standard surveillance benchmarks used to show how the new methods compare to traditional AD tasks.

Key Insights and Ideas

  • The Domain Gap is Real: Models that dominate surveillance benchmarks (like MNAD) perform poorly on physical anomalies. On the PHANTOM dataset, MNAD achieved an average ROCAUC of only 0.55 (barely better than a coin flip), while the simple ViT+kNN baseline reached 0.78.
  • Images vs. Video Features: There is a trade-off between spatial and temporal modeling. Image-based features (ViT) performed better on anomalies involving subtle local changes (like the Candle or Magnets classes). Video-based features (TimeSformer) excelled when the anomaly was defined by a change in trajectory (like the Pendulum class).
  • Simplicity Wins (For Now): The fact that a basic kNN approach on pretrained features outperforms specialized AD architectures suggests that current "state-of-the-art" AD models are over-fitted to the specific characteristics of surveillance footage.

Key Results

  • PHANTOM Dataset: The ViT (Vision Transformer) with average pooling was the top performer with an average ROCAUC of 0.91 across the six classes, significantly outperforming the surveillance-oriented MNAD (0.55).
  • Surveillance Performance: Conversely, on the UCSD Pedestrian 2 dataset, MNAD scored 97.0, while the authors' best baseline only reached 75.0. This confirms that surveillance AD and physical AD are distinct tasks requiring different approaches.
  • General Activity (SSv2): This proved to be the hardest task. The best performing method (ViT) only achieved a 0.58 ROCAUC, highlighting that high scene variability (different backgrounds, lighting, and objects) remains a major hurdle for general video AD.

Limitations and Future Work

The primary limitation is the kNN baseline's struggle with high variability. While it works well on the relatively controlled environments of the PHANTOM dataset, it falters on the SSv2 dataset where the background and objects change constantly.

The authors suggest that future work should focus on developing "General Video AD" methods that can handle both the subtle physical dynamics of PHANTOM and the high variability of real-world activities. They hope the PHANTOM dataset serves as a stepping stone toward models that truly understand "normality" in a physical sense, rather than just memorizing surveillance patterns.

Summary updated

Abstract

In recent years, many works have addressed the problem of finding never-seen-before anomalies in videos. Yet, most work has been focused on detecting anomalous frames in surveillance videos taken from security cameras. Meanwhile, the task of anomaly detection (AD) in videos exhibiting anomalous mechanical behavior, has been mostly overlooked. Anomaly detection in such videos is both of academic and practical interest, as they may enable automatic detection of malfunctions in many manufacturing, maintenance, and real-life settings. To assess the potential of the different approaches to detect such anomalies, we evaluate two simple baseline approaches: (i) Temporal-pooled image AD techniques. (ii) Density estimation of videos represented with features pretrained for video-classification.Development of such methods calls for new benchmarks to al-low evaluation of different possible approaches. We introduce the Physical Anomalous Trajectory or Motion (PHANTOM) dataset <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , which contains six different video classes. Each class consists of normal and anomalous videos. The classes differ in the presented phenomena, the normal class variability, and the kind of anomalies in the videos. We also suggest an even harder benchmark where anomalous activities should be spotted on highly variable scenes.
BibTeX
@inproceedings{kart2022approaches,
  author = {Laura Kart and Niv Cohen},
  title = {{Approaches Toward Physical and General Video Anomaly Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=rsth9AyoFY},
}

Loading related content...

Figure