Ano-Graph: Learning Normal Scene Contextual Graphs to Detect Video Anomalies

Masoud Pourreza, Mohammadreza Salehi, Mohammad Sabokrou

TL;DR

This paper introduces Ano-Graph, a novel method for video anomaly detection that explicitly models the spatio-temporal interactions between objects using graphs. By leveraging self-supervised learning on these graphs, Ano-Graph achieves state-of-the-art performance on challenging datasets, demonstrating robustness to real-world variations and improved data efficiency.

What Problem Does This Paper Address?

Detecting anomalies in videos is a notoriously difficult task. The core challenge lies in the unsupervised nature of the problem: we typically only have examples of "normal" behavior during training, and anomalous events are rare and diverse. Existing methods often rely on autoencoders or generative adversarial networks (GANs) to learn normal patterns. However, these approaches tend to overlook or poorly model the crucial spatio-temporal interactions between objects within a scene. For instance, a car driving on the sidewalk is anomalous not just because of the car's appearance, but because of its interaction with the environment and other objects. Ano-Graph aims to bridge this gap by explicitly capturing these object interactions.

Methods

Ano-Graph's core innovation is the use of Spatio-Temporal Graphs (STGs) to represent normal video scenes. Here's a breakdown of their approach:

  • Object Detection: First, an off-the-shelf object detector (like Faster-RCNN) is used to identify objects in each frame. Each detected object becomes a node in the graph.
  • Graph Construction:
    • Spatial Edges: Connections between objects within the same frame are established based on their spatial proximity, specifically using the Intersection Over Union (IOU) of their bounding boxes. This captures how objects are positioned relative to each other.
  • Temporal Edges: Connections between objects across consecutive frames are created using the cosine similarity of their feature vectors. This models how objects move and interact over time.
  • Self-Supervised Learning (SSL) on Graphs: A self-supervised learning method is applied to these STGs. The goal is to learn a "semantic space" that encapsulates normal object interactions. This involves training a discriminator to distinguish between normal graph embeddings and corrupted ones, effectively learning what constitutes "normal" interaction patterns.
  • Anomaly Scoring: At test time, the STG of a video segment is generated. The learned discriminator then assigns an anomaly score based on how well the segment's graph structure conforms to the learned normal patterns. Higher scores indicate a higher probability of anomaly.

Datasets

The authors evaluated Ano-Graph on several standard video anomaly detection benchmarks:

  • Avenue: A dataset with normal and anomalous events like people running or walking in the wrong direction.
  • ShanghaiTech: A large dataset containing various anomalies such as robberies, fights, and jumps.
  • UCSD-Ped2: Features crowded scenes with anomalies like vehicles or skateboarders in pedestrian areas.
  • ADOC (A Day on Campus): A challenging dataset with a wide range of event types and imbalanced frequencies, occurring over 24 hours with varying lighting.
  • Street Scene: A dataset captured from a static USB camera, focusing on traffic anomalies.

Key Insights and Ideas

  • Explicit Interaction Modeling is Crucial: The paper strongly argues that simply looking at object features or global scene statistics is insufficient. Modeling the interactions between objects is key to understanding normal behavior and detecting deviations.
  • Graphs as a Natural Representation: Spatio-temporal graphs provide an intuitive and powerful way to represent the complex relationships between objects in a video over time.
  • Self-Supervised Learning for Unsupervised Anomaly Detection: Leveraging SSL on graphs allows the model to learn rich representations of normal interactions without requiring labeled anomalous data, which is often scarce.
  • Data Efficiency: Ano-Graph demonstrates good performance even when a significant portion of training data is discarded, suggesting its applicability in scenarios with limited training samples.
  • Robustness to Real-World Variations: The method shows resilience to common issues like illumination changes, which often plague other VAD techniques.

Key Results

  • State-of-the-Art Performance: Ano-Graph achieves new state-of-the-art (SOTA) results on several challenging datasets, including ADOC and Street Scene, often by a significant margin.
  • Competitive Performance: It also remains competitive on other established datasets like Avenue, ShanghaiTech, and UCSD-Ped2.
  • Improved AUC: For instance, on ADOC, Ano-Graph surpasses the best-reported method in frame-level AUC by at least 16% on average. On Street Scene, it improves AUC, TBDR, and RBDR by 8.19%, 9.91%, and 16.7% respectively.
  • Data Efficiency: When training data is reduced by 50%, Ano-Graph's performance degrades much less severely than a competing method (MNAD), highlighting its data efficiency.
  • Real-time Potential: While object detection itself can be a bottleneck, the graph processing and anomaly detection parts are very fast, suggesting potential for real-time applications.

Limitations and Future Work

  • Object Detection Dependency: The performance of Ano-Graph is inherently tied to the quality of the object detector used. Errors or missed detections by the detector can impact the graph construction and subsequent anomaly detection. The authors note that using faster detectors like YOLOv4 could improve real-time performance.
  • Computational Cost of Object Detection: While the graph processing is fast, the initial object detection step can be computationally intensive, especially with complex scenes.
  • Early Anomaly Prediction: The authors suggest that Ano-Graph's ability to model interactions could be extended to "Early Anomaly Prediction," where potential future anomalies (like impending car crashes) could be predicted before they fully manifest. This opens up avenues for proactive safety systems.

Summary updated

Abstract

Video anomaly detection has proved to be a challenging task owing to its unsupervised training procedure and high spatio-temporal complexity existing in real-world scenarios. In the absence of anomalous training samples, state-of-the-art methods try to extract features that fully grasp normal behaviors in both space and time domains using different approaches such as autoencoders, or generative adversarial networks. However, these approaches completely ignore or, by using the ability of deep networks in the hierarchical modeling, poorly model the spatio-temporal interactions that exist between objects. To address this issue, we propose a novel yet efficient method named Ano-Graph for learning and modeling the interaction of normal objects. Towards this end, a Spatio-Temporal Graph (STG) is made by considering each node as an object's feature extracted from a real-time off-the-shelf object detector, and edges are made based on their interactions. After that, a self-supervised learning method is employed on the STG in such a way that encapsulates interactions in a semantic space. Our method is data-efficient, significantly more robust against common real-world variations such as illumination, and passes SOTA by a large margin on the challenging datasets ADOC and Street Scene while stays competitive on Avenue, ShanghaiTech, and UCSD.
BibTeX
@misc{pourreza2021ano,
  author = {Masoud Pourreza and Mohammadreza Salehi and Mohammad Sabokrou},
  title = {{Ano-Graph: Learning Normal Scene Contextual Graphs to Detect Video Anomalies}},
  year = {2021},
  eprint = {2103.10502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.10502},
}
Figure