Abstract
Events defined by the interaction of objects in a scene are often of critical importance; yet important events may have insufficient labeled examples to train a conventional deep model to generalize to future object appearance. Activity recognition models that represent object interactions explicitly have the potential to learn in a more efficient manner than those that represent scenes with global descriptors. We propose a novel inter-object graph representation for activity recognition based on a disentangled graph embedding with direct observation of edge appearance. We employ a novel factored embedding of the graph structure, disentangling a representation hierarchy formed over spatial dimensions from that found over temporal variation. We demonstrate the effectiveness of our model on the Charades activity recognition benchmark, as well as a new dataset of driving activities focusing on multi-object interactions with near-collision events. Our model offers significantly improved performance compared to baseline approaches without object-graph representations, or with previous graph-based models.
Introduction
Most deep learning models for activity recognition treat video frames as global scene descriptors. By applying convolutional filters to the entire image and pooling the results, these models often miss the fine-grained "who is doing what to whom" details. This limitation is particularly problematic for rare, high-stakes events like car accidents. If a model only observes specific object pairings (e.g., red trucks hitting blue sedans) during training, it may fail to recognize a collision between different objects because it hasn't learned the underlying relationship—the trajectory and proximity—independently of object appearance. The Spatio-Temporal Action Graph (STAG) network addresses this by decoupling object detection from interaction recognition, allowing for better generalization even when labeled data is scarce.
The STAG Architecture
The authors propose a multi-stage pipeline that explicitly models object interactions over space and time. The process begins with a Faster R-CNN backbone to identify objects within each frame.
Crucially, for every pair of detected objects, the model creates a "union box"—a bounding box encompassing both entities. Features extracted from these union boxes represent the "visual phrase" of the interaction, capturing relative distance and orientation directly. This approach is found to be significantly more effective than simply concatenating individual object features.
Disentangled Hierarchies
The core of the STAG module is its dual-hierarchy approach, which separates spatial reasoning from temporal evolution. This disentanglement prevents the model from falling into a "bag of words" trap where temporal order is lost, while ensuring spatial relations are fully processed before being compared across time.
- Spatial Context Hierarchy: Within each frame, a graph is constructed where nodes are objects and edges are their interactions. A Non-Local block (a self-attention mechanism) refines these edge features by considering all other relations in the frame. These are then pooled into a single per-frame descriptor.
- Temporal Context Hierarchy: The sequence of frame descriptors is passed through a second Non-Local block to aggregate context across the entire video. This allows the model to understand how the scene evolves over time.
Datasets and Evaluation
The paper evaluates STAG on two primary benchmarks:
- Charades: A standard, large-scale dataset of daily household activities.
- The Collision Dataset: A major contribution of this work, consisting of 803 real-world dashcam videos of actual collisions and near-collisions. This dataset captures the messy reality of driving, including a "few-shot" version to test learning from minimal examples of rare events.
Key Results
STAG demonstrates superior performance across both benchmarks:
- Collision Detection: On the new Collision dataset, STAG achieved 84.5% accuracy, outperforming standard video models like I3D (82.4%) and LSTM-based approaches (77.5%).
- Few-Shot Learning: In data-starved settings, STAG maintained a lead with 76.3% accuracy, while LSTM-based models dropped to near-random performance (67-69%).
- Charades Benchmark: STAG achieved 37.2 mAP, outperforming the previous state-of-the-art graph-based model (STRG) by 1.0%.
Conclusion
By explicitly modeling object interactions through a hierarchical graph structure and union-box features, STAG provides a more robust framework for activity recognition. While the model's performance is currently tied to the quality of the underlying object detector and the computational complexity of dense graphs, its ability to generalize to rare events like collisions marks a significant step forward in safety-critical computer vision.
Summary updated