From Local Temporal Correlation to Global Anomaly Detection

Chen Change Loy, Tao Xiang, Shaogang GongINRIA 2008

TL;DR

This paper introduces a novel cascade of Hidden Markov Models (HMMs) for detecting subtle, global anomalies in complex outdoor scenes. By first modeling local temporal correlations within semantic regions and then combining these insights, the framework achieves robust anomaly detection, even for events that are ambiguous or supported by weak visual evidence.

What Problem Does This Paper Address?

Automated anomaly detection in video surveillance is crucial for applications like security and video summarization. While detecting individual object anomalies (e.g., a person running when others walk) is relatively straightforward, identifying anomalies involving the interactions and correlations between multiple objects is significantly more challenging. These "global anomalies" are often subtle, can be missed by human observers, and are difficult to quantify. Existing methods often struggle with these complex temporal relationships, especially when dealing with noisy or ambiguous visual cues. The paper aims to address this gap by developing a system that can effectively model and detect these nuanced, multi-object anomalies.

Methods

The core of the proposed framework is a cascade of Multi-Observation Hidden Markov Models (MOHMMs), represented as a Dynamic Bayesian Network (DBN). This cascade has two stages:

  1. Local Models (First Stage): For each semantically segmented region of the scene, a MOHMM is trained to model the temporal evolution of local atomic events (e.g., vehicle movements). MOHMMs are used because they can output multiple observation symbols per time step, making them suitable for capturing the complex, correlated behaviors within a region.
  2. Global Model (Second Stage): A single MOHMM takes the inferred hidden states from all local models as input. This global model learns the temporal correlations across different regions, enabling it to detect anomalies that manifest as unusual interactions between objects in different parts of the scene.

The framework also incorporates:

  • Semantic Scene Segmentation: The scene is divided into meaningful regions (e.g., lanes on a road) to group correlated atomic events.
  • Discrete Event Representation: Instead of relying on object trajectories (which can be unreliable in crowded scenes), behaviors are represented as discrete atomic events within each region.
  • Model Training: A two-step process involving Expectation-Maximization (EM) for both local and global models. K-means clustering is used for parameter initialization to mitigate convergence to poor local optima.
  • Anomaly Detection: Achieved by examining the log-likelihood of an unseen sequence with respect to the global model. A low log-likelihood suggests an anomaly.

Datasets

The experiments were conducted on a busy road traffic junction scene. The video footage was recorded at 25Hz, with a resolution of 360x288 pixels, and a total duration of approximately 22 minutes. The dataset was segmented into 112 non-overlapping clips, with 73 clips used for training and 39 for testing. The scene was segmented into six regions, and 30 local atomic events were discovered. The authors also provide a detailed ground truth by manually examining each frame to identify and categorize anomalies.

Key Insights and Ideas

  • The Power of Cascades: The cascade structure is key. By first modeling local behaviors and then integrating them, the system can leverage subtle cues from individual regions to make a more informed global decision. This is more robust than a single, monolithic model trying to learn everything at once.
  • Beyond Obvious Anomalies: The framework excels at detecting "Category-B" (rare/ambiguous) and "Category-C" (weak evidence) anomalies, which are often missed by human observers and simpler algorithms. This is a significant step forward for practical surveillance.
  • Robustness to Noise: Semantic segmentation and the cascade structure make the model less susceptible to noise and errors in behavior representation compared to single-stage models that learn directly from raw observations.
  • Exploiting Combinatory Evidence: The system effectively combines evidence from multiple temporal processes (behaviors in different regions) to build a stronger case for anomaly detection.

Key Results

  • The proposed cascade model achieved a significantly better detection performance than a single MOHMM baseline.
  • The Area Under the ROC (AUROC) for the cascade model was 0.9765, compared to 0.5794 for the single MOHMM.
  • The cascade model successfully identified all five manually labeled abnormal clips (clips 3, 4, 10, 25, and 35) as anomalous.
  • It correctly identified visually obvious anomalies (Category-A, clips 3 & 4), ambiguous anomalies (Category-B, clips 25 & 35), and crucially, subtle anomalies supported by weak evidence (Category-C, clip 10).
  • The model demonstrated robustness to imperfect blob detection, which caused false alarms in the single MOHMM baseline.

Limitations and Future Work

  • Model Choice Flexibility: The authors note that different types of HMMs (e.g., HHMM, HSMM) could be employed at different stages depending on the detection requirements and scene complexity.
  • Duration Modeling: The current work focuses on temporal ordering. Future work will investigate incorporating duration modeling capabilities to achieve more accurate correlations with long-term temporal dependencies.
  • Generalizability: While tested on a traffic scene, further evaluation on diverse datasets would be beneficial.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{loy2008from,
  author = {Chen Change Loy and Tao Xiang and Shaogang Gong},
  title = {{From Local Temporal Correlation to Global Anomaly Detection}},
  booktitle = {INRIA},
  year = {2008},
}
Figure