Introduction
Most contemporary activity recognition systems rely heavily on supervised learning, necessitating vast amounts of manually annotated data. While these models excel at identifying simple "atomic" actions, they often struggle with complex, multi-threaded events where multiple objects interact simultaneously over time—such as the coordinated movements of loaders, trolleys, and aircraft during ground servicing. This paper presents a framework for the unsupervised discovery of these complex event classes by shifting from raw coordinate-based tracking to abstract qualitative relational representations.
Qualitative Event Representation
The core innovation of this approach is the use of Qualitative Spatio-Temporal Graphs (QSTGs). Rather than processing raw pixel values or metric coordinates, which vary significantly across different camera views and instances, the system abstracts object interactions into qualitative spatial relations. These include states such as "Partially Overlapping" (PO) or "Disconnected" (DR).
As shown in the representation above, tracklets are segmented into "episodes" based on stable spatial relations. These episodes are then linked via temporal relations (using Allen’s interval algebra, such as "meets," "starts," or "overlaps") to form a candidate event graph. This abstraction ensures that a specific activity, like a loader approaching a plane, maintains a consistent structural representation regardless of its precise location or speed.
Generative Model and Inference
The authors propose a hierarchical generative model that interprets video data through a three-step process:
- Event Class Sampling: Prototypical event graphs (templates) are sampled from a set of latent event classes.
- Activity Graph Construction: These individual event templates are combined into a global "activity graph" that defines how events share objects (e.g., multiple vehicles interacting with the same aircraft).
- Embedding: The abstract graph is embedded into real-world spatio-temporal coordinates to produce the observed object tracks.
To recover the most likely event structure from a video, the system employs a Markov Chain Monte Carlo (MCMC) search. This procedure iteratively proposes different ways to slice the video into events—splitting, merging, or adjusting boundaries—and evaluates these hypotheses against the generative model's probability distribution.
Unsupervised Discovery and Clustering
Within the MCMC optimization loop, the system utilizes spectral clustering to assign labels to discovered events. By comparing the structural similarity of the generated QSTGs, the model can group disparate video segments into cohesive classes without any human-provided labels.
This clustering mechanism allows the system to identify the underlying "building blocks" of a domain. For instance, in an airport environment, the model learns that various instances of a bridge attaching to a plane belong to the same functional category because they share the same qualitative relational signature.
Experimental Results
The framework was evaluated on a complex real-world dataset consisting of 12 hours of aircraft servicing video. The environment featured six object classes (Plane, Trolley, Carriage, Loader, Bridge, and Plane Puller) and significant overlapping activity.
- Event Discovery: The system successfully identified 14 distinct, semantically meaningful event classes without manual intervention.
- Accuracy: In quantitative tests against human ground truth for major categories like "Unloading" and "Bridge Attachment," the system demonstrated high precision. Notably, it correctly grouped 75% of all unloading events into two closely related clusters.
- Robustness: The qualitative abstraction allowed the model to remain effective despite noisy tracking data, such as unstable bounding boxes or temporary occlusions.
Conclusion and Future Work
By prioritizing relational abstraction over metric precision, this method enables the discovery of complex event structures in an entirely unsupervised manner. While the system currently depends on the quality of the underlying object trackers, it provides a robust foundation for understanding multi-threaded activities. Future research aims to extend this into a unified model that simultaneously learns both event classes and the functional categories of the objects involved, moving closer to a fully autonomous understanding of dynamic scenes.
Summary updated