Detection of Unknown Anomalies in Streaming Videos with Generative Energy-based Boltzmann Models

Hung Vu, Tu Dinh Nguyen, Dinh Q. PhungarXiv 2018 · OpenReview

TL;DR

This paper introduces a novel energy-based framework for detecting unknown anomalies in streaming videos. By leveraging Restricted Boltzmann Machines (RBMs) and Deep Boltzmann Machines (DBMs), the system learns from raw pixel data without requiring labeled examples or hand-crafted features, effectively addressing key challenges in video surveillance. A key innovation is the DBM-based approach, which simultaneously performs scene clustering and reconstruction, offering explainable anomaly detection.

What Problem Does This Paper Address?

Video surveillance systems are increasingly deployed, generating vast amounts of data. However, effectively detecting "unknown anomalies" (events that are not explicitly defined beforehand) poses significant challenges. These include:

  • Limited Labeled Data: Manually annotating every anomalous event in a video stream is prohibitively time-consuming and expensive.
  • Ambiguous Definition of "Abnormal": What constitutes an anomaly can vary greatly depending on the context and is often difficult to define precisely.
  • Expensive Feature Engineering: Traditional methods rely on hand-crafted visual features (like HOG or optical flow), which are computationally intensive and require domain expertise to design and adapt for different applications.

This paper aims to overcome these hurdles by developing an unsupervised anomaly detection system that can learn from raw video data and identify anomalies without prior knowledge of what they might be.

Methods

The core of the proposed system is an Energy-Based Model (EBM). EBMs are probabilistic models that learn the underlying distribution of normal data by assigning an "energy" to different configurations of variables. Anomalies are then detected as data points that have unusually high energy (i.e., are poorly reconstructed by the model).

The authors propose two main variants:

  • EAD RBM (Energy-based Anomaly Detector with Restricted Boltzmann Machines): This approach uses RBMs. It first employs a clustering RBM to segment the video scene into regions and then trains separate RBMs for each region to reconstruct the image patches within that region. Anomalies are detected based on reconstruction errors.
  • EAD DBM (Energy-based Anomaly Detector with Deep Boltzmann Machines): This is a more unified approach that uses a single Deep Boltzmann Machine (DBM). DBMs, with their hierarchical structure, are capable of performing multiple tasks simultaneously. In this framework, the DBM's architecture is designed such that one part handles scene clustering (similar to the clustering RBM) and another part handles reconstruction. This allows for a single model to learn both scene structure and reconstruct normal patterns.

Both systems are trained in an unsupervised manner directly on raw pixel data. The detection phase involves reconstructing input video frames and measuring the reconstruction error. High errors indicate potential anomalies. For streaming video, an online version is developed where model parameters are incrementally updated as new frames arrive.

A key advantage highlighted is the explainability offered by the DBM-based system. The clustering layer in the DBM can generate "pattern maps" that visualize how the model represents different objects at an abstract level, providing insights into why certain detections are made and aiding in debugging.

Datasets

The authors evaluate their methods on three standard benchmark datasets for anomaly detection:

  • UCSD Ped 1
  • UCSD Ped 2
  • Avenue

These datasets consist of real-world video sequences containing various types of anomalies, such as pedestrian crowd anomalies, object intrusions, and unusual movements. The experiments are conducted in an unsupervised setting, meaning all label information is discarded during training.

Key Insights and Ideas

  • Unsupervised Learning on Raw Pixels is Viable: The paper demonstrates that energy-based models can effectively learn the distribution of normal video data directly from raw pixels, bypassing the need for manual feature engineering and labeled data.
  • Reconstruction Error as Anomaly Signal: The core idea of detecting anomalies based on how poorly a model can reconstruct them is a powerful and intuitive approach.
  • DBMs for Unified Task Learning: The DBM architecture proves adept at simultaneously handling scene segmentation (clustering) and anomaly detection (reconstruction) within a single model, simplifying the system and enhancing its capabilities.
  • Explainable AI in Anomaly Detection: The pattern maps generated by the DBM's clustering layer offer a novel way to understand the model's internal reasoning, making anomaly detection results more transparent and interpretable. This is a significant step beyond black-box detection systems.
  • Handling Scale and Size Variations: The approach of processing videos at multiple scales and aggregating results helps in detecting anomalies of varying sizes.

Key Results

  • Superior Performance: Both EAD RBM and EAD DBM systems achieve superior anomaly detection performance compared to unsupervised baselines like PCA, OC-SVM, and GMM.
  • Competitive with State-of-the-Art: The proposed methods obtain accuracy comparable to state-of-the-art deep learning approaches, particularly in pixel-level and dual-pixel level evaluations.
  • Scene Clustering Capability: The clustering modules in EAD RBM and EAD DBM can automatically identify a plausible number of scene clusters, demonstrating their ability to understand scene structure.
  • Model Explanation: The DBM-based system successfully generates abstract pattern maps that correlate with different objects and their movements, providing visual evidence for anomaly detection and aiding in understanding system mistakes. For instance, it can explain why distant cyclists might be misclassified as pedestrians due to similar visual patterns.

Limitations and Future Work

  • Computational Cost of DBMs: While EAD DBM simplifies the system by using a single model, training DBMs can still be computationally intensive, especially with a large number of hidden units for reconstruction. The paper notes that EAD DBM with 200 reconstruction units performs comparably to EAD RBM.
  • Online Update Efficiency: For streaming scenarios, the paper proposes region-based DBMs (converted to RBMs for efficiency) to handle scene changes. However, the effectiveness and efficiency of these online updates are areas of ongoing research.
  • Future Work: The authors suggest that the pattern maps generated by the clustering layer can serve as high-level features for other computer vision tasks like object tracking and recognition, opening avenues for further research. The explainability aspect is also highlighted as a valuable tool for debugging and understanding system behavior.

Summary updated

Abstract

Abnormal event detection is one of the important objectives in research and practical applications of video surveillance. However, there are still three challenging problems for most anomaly detection systems in practical setting: limited labeled data, ambiguous definition of "abnormal" and expensive feature engineering steps. This paper introduces a unified detection framework to handle these challenges using energy-based models, which are powerful tools for unsupervised representation learning. Our proposed models are firstly trained on unlabeled raw pixels of image frames from an input video rather than hand-crafted visual features; and then identify the locations of abnormal objects based on the errors between the input video and its reconstruction produced by the models. To handle video stream, we develop an online version of our framework, wherein the model parameters are updated incrementally with the image frames arriving on the fly. Our experiments show that our detectors, using Restricted Boltzmann Machines (RBMs) and Deep Boltzmann Machines (DBMs) as core modules, achieve superior anomaly detection performance to unsupervised baselines and obtain accuracy comparable with the state-of-the-art approaches when evaluating at the pixel-level. More importantly, we discover that our system trained with DBMs is able to simultaneously perform scene clustering and scene reconstruction. This capacity not only distinguishes our method from other existing detectors but also offers a unique tool to investigate and understand how the model works.
BibTeX
@misc{vu2018detection,
  author = {Hung Vu and Tu Dinh Nguyen and Dinh Q. Phung},
  title = {{Detection of Unknown Anomalies in Streaming Videos with Generative Energy-based Boltzmann Models}},
  year = {2018},
  eprint = {1805.01090},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.01090},
}
Figure