Video Activity Recognition in the Real World

Anthony Hoogs, A. G. Amitha PereraAAAI 2008 · OpenReview

Abstract: With recent advances in motion detection and tracking in video, more efforts are being directed at higher-level video analysis such as recognizing actions, events and activities. One of the more challenging problems is recognizing activities that involve multiple people and/or vehicles, whose relationships change over time, when motion detection and tracking are unreliable, as commonly occurs in busy scenes. We describe an approach to this problem based on Dynamic Bayesian Networks, and show how DBNs can be extended to compensate for track failures. We also show that defining DBNs with semantic concepts improves robustness vs. direct observable, and discuss implications and ideas for incorporating semantic, symbolic knowledge into the perceptual domain of activity recognition.

Introduction

In controlled laboratory settings, video activity recognition often focuses on isolated, short-duration actions like walking or waving. However, real-world environments—such as a busy airport tarmac—present a far greater challenge. Activities in these spaces involve multiple agents (vehicles and personnel) interacting over extended periods, often under poor visual conditions.

The primary hurdle in these environments is the inherent unreliability of automated tracking. Standard trackers frequently suffer from fragmentation (broken trajectories), occlusion (objects disappearing), and track switching (identity jumps between nearby objects). Most existing models assume "perfect" tracks and fail when data is compromised. This research addresses the "semantic gap" between raw pixel data and high-level activity understanding by moving toward semantic primitives.

Methodology: Semantics and Probabilistic Modeling

The proposed framework bridges the gap between raw tracking and symbolic reasoning through several key components:

  1. Dynamic Bayesian Networks (DBNs): The system uses DBNs to model the temporal progression of complex activities. For instance, an "aircraft refueling" event is decomposed into 17 discrete temporal states.
  2. Semantic Primitives (Cyc Ontology): Rather than feeding raw $(x, y)$ coordinates into the DBN, the system transforms data into semantic predicates such as "close to," "moving," or "contained in." These are derived from the Cyc ontology, a massive knowledge base of common-sense concepts.
  3. Viewpoint Invariance: By reasoning about qualitative relationships (e.g., "truck near plane") rather than absolute screen coordinates, models trained on one camera angle can successfully generalize to entirely different viewpoints.
  4. Handling Track Errors: The system evaluates the likelihood of an activity across various track fragments, effectively determining which combination of broken tracks best fits the known model of the activity.

Experimental Results and Robustness

The system was evaluated using video of aircraft ground operations. The models were tested against 303 observation sequences, including 300 random assignments to test false-positive resistance and 3 "confusable" sequences.

Separation of Likelihoods

A critical test of the model is its ability to distinguish true activity from random noise. As shown in the log-likelihood distribution, the "true" actor-role assignment is significantly separated from random permutations, demonstrating the model's discriminative power.

Figure 3: Histogram of log-likelihoods for 300 random actor-role assignments (blue) versus the correct assignment (red bar on the right).

Robustness to Fragmentation

The research specifically investigated how track fragmentation affects recognition. Surprisingly, the system maintained reasonable performance even when significant gaps were introduced into the tracking data. In the refueling scenario, the DBN could tolerate gaps of up to 400 frames (approximately 13 seconds at 30fps) before performance degraded sharply.

Figure 4: Event scores as a function of frame gap, showing the system's robustness to temporal fragmentation up to 400 frames.

Key Insights and Future Directions

The study yielded several counter-intuitive insights for real-world computer vision:

  • Fragmentation is manageable; Switching is fatal: While the system is robust to gaps, "track switching"—where a track identity swaps between different objects—is disastrous for high-level reasoning.
  • Short tracks can be superior: For high-level recognition, it is often better to have many short, reliable track fragments that the activity model can "stitch" together, rather than one long track prone to identity-switching errors.
  • Semantics as a Bridge: Qualitative descriptors provide massive robustness against changes in scene configuration and camera placement compared to quantitative coordinate-based learning.

Future work aims to integrate these probabilistic models more deeply into general-purpose ontologies and move toward "activity-aided tracking," where high-level models actively assist low-level trackers in linking fragments in real-time.

Summary updated

Abstract

With recent advances in motion detection and tracking in video, more efforts are being directed at higher-level video analysis such as recognizing actions, events and activities. One of the more challenging problems is recognizing activities that involve multiple people and/or vehicles, whose relationships change over time, when motion detection and tracking are unreliable, as commonly occurs in busy scenes. We describe an approach to this problem based on Dynamic Bayesian Networks, and show how DBNs can be extended to compensate for track failures. We also show that defining DBNs with semantic concepts improves robustness vs. direct observable, and discuss implications and ideas for incorporating semantic, symbolic knowledge into the perceptual domain of activity recognition.
BibTeX
@inproceedings{hoogs2008video,
  author = {Anthony Hoogs and A. G. Amitha Perera},
  title = {{Video Activity Recognition in the Real World}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2008},
  url = {https://openreview.net/forum?id=Hk4xGlZ_Zr},
}
Figure