Object-Centric Auto-Encoders and Dummy Anomalies for Abnormal Event Detection in Video

Radu Tudor Ionescu, Fahad Shahbaz Khan, Mariana-Iuliana Georgescu, Ling Shao

Abstract

Abnormal event detection in video is a challenging vision problem. Most existing approaches formulate abnormal event detection as an outlier detection task, due to the scarcity of anomalous data during training. Because of the lack of prior information regarding abnormal events, these methods are not fully-equipped to differentiate between normal and abnormal events. In this work, we formalize abnormal event detection as a one-versus-rest binary classification problem. Our contribution is two-fold. First, we introduce an unsupervised feature learning framework based on object-centric convolutional auto-encoders to encode both motion and appearance information. Second, we propose a supervised classification approach based on clustering the training samples into normality clusters. A one-versus-rest abnormal event classifier is then employed to separate each normality cluster from the rest. For the purpose of training the classifier, the other clusters act as dummy anomalies. During inference, an object is labeled as abnormal if the highest classification score assigned by the one-versus-rest classifiers is negative. Comprehensive experiments are performed on four benchmarks: Avenue, ShanghaiTech, UCSD and UMN. Our approach provides superior results on all four data sets. On the large-scale ShanghaiTech data set, our method provides an absolute gain of 8.4% in terms of frame-level AUC compared to the state-of-the-art method.

Introduction

Identifying abnormal events in video—such as a car driving on a sidewalk or a person running in a library—is a classic computer vision challenge. The difficulty lies in the fact that "abnormal" is context-dependent and anomalous data is rarely available for training. Traditionally, this has been treated as an outlier detection problem: models learn a representation of "normal" behavior and flag anything that deviates. However, without examples of anomalies, these models often struggle to draw a sharp boundary between unusual normal behavior and actual threats. Furthermore, frame-level analysis often suffers from background noise that obscures the specific actors involved in an event.

Proposed Methodology

This work, presented at CVPR 2019, reframes anomaly detection as a discriminative classification task using an object-centric framework. The pipeline consists of two primary stages:

1. Object-Centric Feature Learning

Rather than analyzing the entire frame, the system uses a pre-trained Single-Shot Detector (SSD) to isolate objects. For each detected object, three Convolutional Auto-encoders (CAEs) are employed to extract latent features:

  • Appearance CAE: Encodes the visual characteristics of the object.
  • Motion CAEs: Two separate encoders capture the temporal dynamics (image gradients) immediately before and after the current frame.

2. The "Dummy Anomaly" Strategy

The core innovation is the use of normality clusters to create a supervised learning environment from unsupervised data. Normal training samples are grouped into $k$ clusters using k-means (e.g., "walking," "standing," "cycling"). The system then trains $k$ one-versus-rest SVM classifiers. To train a classifier for a specific cluster (e.g., "walking"), all other normal clusters are treated as "dummy anomalies."

Overview of the proposed framework, including object-centric feature extraction via convolutional auto-encoders and the one-versus-rest classification strategy.

During inference, an object is classified as abnormal if it fails to fit strongly into any of the $k$ normal clusters (i.e., all classifiers return a negative score).

Experimental Results

The framework was evaluated on four major benchmarks: Avenue, ShanghaiTech, UCSD Ped2, and UMN. The results demonstrate state-of-the-art performance across all datasets:

  • ShanghaiTech: Achieved a significant 84.9% AUC, representing an 8.4% absolute gain over previous methods. This is particularly notable given the dataset's complexity and scale (13 scenes, 300k+ frames).
  • Avenue & UCSD Ped2: Reached 90.4% AUC and 97.8% AUC, respectively.
  • Efficiency: The system maintains a processing speed of approximately 11 frames per second (FPS) on a single Titan Xp GPU.
Qualitative results showing true abnormal event detections (green) and false detections (red) across four benchmark datasets.

Discussion and Conclusion

The success of this approach highlights two key insights: object-centricity allows models to ignore irrelevant background noise, and discriminative training—even using "dummy" anomalies—creates more robust decision boundaries than traditional outlier detection.

While highly effective, the system's performance is tied to the accuracy of the underlying object detector. In crowded scenes, overlapping bounding boxes can occasionally lead to false positives. Future work may involve integrating object tracking and more precise segmentation to further refine detection in complex environments. Overall, the "dummy anomaly" strategy provides a powerful new paradigm for unsupervised video analysis.

Summary updated

Abstract

Abnormal event detection in video is a challenging vision problem. Most existing approaches formulate abnormal event detection as an outlier detection task, due to the scarcity of anomalous data during training. Because of the lack of prior information regarding abnormal events, these methods are not fully-equipped to differentiate between normal and abnormal events. In this work, we formalize abnormal event detection as a one-versus-rest binary classification problem. Our contribution is two-fold. First, we introduce an unsupervised feature learning framework based on object-centric convolutional auto-encoders to encode both motion and appearance information. Second, we propose a supervised classification approach based on clustering the training samples into normality clusters. A one-versus-rest abnormal event classifier is then employed to separate each normality cluster from the rest. For the purpose of training the classifier, the other clusters act as dummy anomalies. During inference, an object is labeled as abnormal if the highest classification score assigned by the one-versus-rest classifiers is negative. Comprehensive experiments are performed on four benchmarks: Avenue, ShanghaiTech, UCSD and UMN. Our approach provides superior results on all four data sets. On the large-scale ShanghaiTech data set, our method provides an absolute gain of 8.4% in terms of frame-level AUC compared to the state-of-the-art method.
BibTeX
@inproceedings{ionescu2019object,
  author = {Radu Tudor Ionescu and Fahad Shahbaz Khan and Mariana-Iuliana Georgescu and Ling Shao},
  title = {{Object-Centric Auto-Encoders and Dummy Anomalies for Abnormal Event Detection in Video}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=SOWHxeicqyc},
}
Figure