Learning a scene contextual model for tracking and abnormality detection

Benjamin Z. Yao, Liang Wang, Song-Chun ZhuWorkshops @ CVPR 2008 · OpenReview

Abstract

This paper presents a novel framework for learning a contextual motion model involving multiple objects in far-field surveillance video. We represent the trajectories of multiple objects using a 3D graph $\mathcal{G}$ in $(x, y, t)$ space, augmented by spatio-temporal relations between moving and static objects. An inhomogeneous Markov model is defined over $\mathcal{G}$, with parameters estimated via Maximum Likelihood Estimation (MLE) and relations pursued through a minimax entropy principle. This approach allows for the synthesis of new video sequences that reproduce observed statistics. The learned model is applied to improve object tracking and detect sophisticated abnormal events, such as traffic rule violations, through a log-likelihood ratio test estimated by importance sampling.

Introduction

Automated surveillance systems often struggle to track multiple similar-looking objects and identify complex abnormal behaviors. Traditional methods frequently treat objects in isolation or rely on simplistic motion models like constant velocity. However, real-world motion is deeply contextual; a vehicle's path is constrained by road geometry (static context), and its dynamics are influenced by other agents (dynamic context). Without accounting for these interactions, trackers often fail during occlusions, and abnormality detection remains limited to simple outliers like speeding.

This framework addresses these challenges by learning a "Trajectory Graph" that captures the complex spatio-temporal relationships between multiple moving objects and their environment.

Modeling Contextual Motion

The core of the proposed system is the Trajectory Graph ($\mathcal{G}$). In this representation, nodes represent object states, and edges represent four distinct types of relations:

  1. Object-to-Scene: Connections to semantic regions such as sources (entries), sinks (exits), and paths.
  2. Single-Object Temporal: Smoothness, speed, and acceleration along a single track.
  3. Multi-Object Spatial: Relative distance and direction between different objects within the same frame.
  4. Multi-Object Interactive: Complex interactions over time, such as vehicles yielding to pedestrians at crosswalks.
Figure 1: Overview of the learning process and abnormality detection pipeline. The left side shows the iterative pursuit of relations from a library, while the right side illustrates the detection of abnormal events like collisions and off-road movement.

Learning through Relation Pursuit

The model is learned using a minimax entropy principle. Starting with a simple base model, the algorithm iteratively selects relations from a "Relation Library" that provide the highest information gain. This process minimizes the KL divergence between the model and the observed training data.

To validate the model, the authors employ an "analysis-by-synthesis" approach. A successful model should be capable of synthesizing entirely new, realistic traffic trajectories that match the statistical properties of the training data. As shown in Figure 1, as the number of relations ($n$) increases, the KL-divergence decreases, and the system's ability to detect sophisticated abnormal events improves.

Abnormality Detection Framework

Abnormality is defined using a log-likelihood ratio test. The system compares the probability of a specific trajectory subgraph under the learned contextual model versus a uniform "null" model. If a behavior is highly improbable given the learned "rules" of the scene, it is flagged.

Because detecting abnormalities in high-dimensional trajectory space is computationally intensive, the authors utilize importance sampling. As illustrated in Figure 6, the system samples "particles" from the learned model to estimate the expected behavior at any given time, allowing for efficient real-time abnormality measures and the generation of spatial-temporal heat maps.

Experimental Results

The model was evaluated using traffic surveillance sequences and a dataset of 5,000 trajectories containing synthetic abnormalities.

  • Improved Tracking: By using the scene model as a prior for a Particle Filter, the tracker leverages social and spatial constraints. This prevents the "mode collapse" common in standard filters during occlusions, where multiple white vehicles might otherwise be confused.
  • Sophisticated Detection: The system successfully identified complex events that simpler models miss, including "car moving off road," "collisions," and "traffic rule violations" (e.g., driving through a pedestrian-heavy crosswalk).
  • Quantitative Performance: Precision-Recall and ROC curves demonstrated significant area-under-the-curve (AUC) improvements as contextual relations were added to the model.

Conclusion

By integrating environmental and interactive context into a unified trajectory graph, this framework significantly enhances the robustness of surveillance systems. While current limitations include the need for manual semantic annotation and high computational demands for massive object counts, the "relation pursuit" methodology provides a scalable path toward automated, context-aware scene understanding.

Summary updated

Abstract

In this paper we present a novel framework for learning contextual motion model involving multiple objects in far-field surveillance video and apply the learned model to improving the performance of objects tracking and abnormal event detection. We represent trajectory of multiple objects by a 3D graph G in x,y,t, which is augmented by a number of spatio-temporal relations (links) between moving and static objects in the scene (e.g. relation between crosswalk, pedestrian and car). An inhomogeneous Markov model p is defined over G, whose parameters are estimated by MLE method and relations are pursued by a minimax entropy principle (as in texture modeling) [16] so that we can synthesize entirely new video sequences that reproduce the observed statistics from training video. With the learned model, we define the abnormality of a subgraph given its neighborhood by log-likelihood ratio test, which is estimated by importance sampling. The learned model is applied to tracking and abnormal event detection. Our experiments show that the learned model improve tracking performance and detect sophisticated abnormal events like traffic rule violation.
BibTeX
@inproceedings{yao2008learning,
  author = {Benjamin Z. Yao and Liang Wang and Song-Chun Zhu},
  title = {{Learning a scene contextual model for tracking and abnormality detection}},
  booktitle = {Workshops @ CVPR},
  year = {2008},
  url = {https://openreview.net/forum?id=HkZDmeG_br},
}
Figure