Abstract
Human-Object Interaction (HOI) detection in video clips is a specialized form of Visual-Relationship Detection where the subject is a human. Traditionally, this is modeled as a space-time graph inference problem. This work explores alternative spatial approaches, proposing a hierarchical framework that decouples spatial and temporal modeling. We evaluate Graph Convolutional Networks (GCNs), Convolutional Networks, and Capsule Networks for spatial reasoning. Notably, our approach relies solely on RGB visual features, avoiding ground-truth depth or 3D pose data during inference. We achieve state-of-the-art results on the CAD-120 dataset (88.9% for sub-activity detection and 92.6% for affordance) and competitive performance on the V-COCO image dataset.
Introduction
Understanding Human-Object Interaction (HOI) is critical for applications ranging from assistive robotics to automated retail. While image-based HOI has advanced significantly, video-based detection remains challenging due to the complex temporal dynamics and the variety of spatial relationships.
Existing research often relies on "cheating" features—ground-truth metadata such as depth maps, 3D human poses, or object centroids. These features are rarely available in real-world RGB video streams, limiting the applicability of such models. Furthermore, there is no consensus on the optimal architecture for modeling the spatial relationships between humans and objects. This paper addresses these gaps by systematically exploring spatial modeling alternatives within a modular, hierarchical framework.
Methodology: A Hierarchical Framework
The proposed architecture decouples the problem into three distinct stages to handle spatial relationships, short-term motion, and long-term activity dependencies.
1. Spatial Subnet
This stage extracts relational features from individual frames. The authors compare three distinct modeling choices:
- Graph Convolutional Networks (GCNs): Represent humans and objects as nodes, with interactions modeled as edges.
- Capsule Networks: Utilized to learn inherent spatial hierarchies and "part-to-whole" relationships.
- Convolutional Networks (ConvNets): A baseline using standard Region of Interest (RoI) features.
A "Global Hull"—a bounding box encompassing all entities—is used to provide broader scene context to the spatial subnet.
2. Frame-level Temporal Subnet
A bi-directional Recurrent Neural Network (RNN) processes the spatial embeddings across frames within a video segment. This captures short-term motion cues necessary for identifying immediate actions.
3. Segment-level Temporal Subnet
Complex activities (e.g., "cleaning a microwave") are composed of multiple sub-activities (e.g., "reaching," "cleaning," "moving"). This layer employs an attention mechanism and a secondary RNN to model long-term dependencies across these segments.
Experimental Setup
The model was evaluated on two primary benchmarks:
- CAD-120: A video dataset of daily activities used to test sub-activity detection and object affordance (the potential use of an object).
- V-COCO: A subset of MS-COCO used to evaluate the model's performance on static images.
Results and Discussion
The experimental results demonstrate that sophisticated spatial-temporal modeling can outperform methods relying on specialized sensor data (like depth).
Quantitative Performance
On the CAD-120 dataset, the GCN-based variant achieved an F1 score of 88.9% for sub-activity detection and 92.6% for object affordance. The GCN proved more robust in complex scenes with multiple objects. Conversely, Capsule Networks performed better on the V-COCO dataset, which typically features simpler human-object pairs, achieving a competitive 47.26 mAP.
Qualitative Analysis
The model successfully identifies the progression of interactions and the changing affordances of objects throughout a video sequence.
Key Insights
- The Importance of Spatial Modeling: Ablation studies showed that removing the spatial modeling component caused sub-activity detection accuracy to drop significantly (to 61.5%), proving that object detection alone is insufficient for HOI.
- Human Context: Appending human node features to object nodes significantly boosted affordance detection, as the human's action provides a strong prior for the object's state.
Conclusion
This work establishes that decoupling spatial and temporal modeling is an effective strategy for video-based HOI. By achieving state-of-the-art results using only visual features, the proposed framework offers a more scalable and generalizable solution for real-world applications. Future work will focus on improving Capsule Network efficiency for multi-object scenes and refining the model's ability to distinguish between visually similar actions.
Summary updated