Exploration of Spatial and Temporal Modeling Alternatives for HOI

Rishabh Dabral, Srijon Sarkar, Sai Praneeth Reddy, Ganesh RamakrishnanWACV 2021 · OpenReview

Abstract

Human-Object Interaction (HOI) detection in video clips is a specialized form of Visual-Relationship Detection where the subject is a human. Traditionally, this is modeled as a space-time graph inference problem. This work explores alternative spatial approaches, proposing a hierarchical framework that decouples spatial and temporal modeling. We evaluate Graph Convolutional Networks (GCNs), Convolutional Networks, and Capsule Networks for spatial reasoning. Notably, our approach relies solely on RGB visual features, avoiding ground-truth depth or 3D pose data during inference. We achieve state-of-the-art results on the CAD-120 dataset (88.9% for sub-activity detection and 92.6% for affordance) and competitive performance on the V-COCO image dataset.

Introduction

Understanding Human-Object Interaction (HOI) is critical for applications ranging from assistive robotics to automated retail. While image-based HOI has advanced significantly, video-based detection remains challenging due to the complex temporal dynamics and the variety of spatial relationships.

Existing research often relies on "cheating" features—ground-truth metadata such as depth maps, 3D human poses, or object centroids. These features are rarely available in real-world RGB video streams, limiting the applicability of such models. Furthermore, there is no consensus on the optimal architecture for modeling the spatial relationships between humans and objects. This paper addresses these gaps by systematically exploring spatial modeling alternatives within a modular, hierarchical framework.

Methodology: A Hierarchical Framework

The proposed architecture decouples the problem into three distinct stages to handle spatial relationships, short-term motion, and long-term activity dependencies.

Overview of the hierarchical architecture for HOI detection, illustrating the spatial subnet, intra-segment RNN, and attention-based segment modeling.

1. Spatial Subnet

This stage extracts relational features from individual frames. The authors compare three distinct modeling choices:

  • Graph Convolutional Networks (GCNs): Represent humans and objects as nodes, with interactions modeled as edges.
  • Capsule Networks: Utilized to learn inherent spatial hierarchies and "part-to-whole" relationships.
  • Convolutional Networks (ConvNets): A baseline using standard Region of Interest (RoI) features.

A "Global Hull"—a bounding box encompassing all entities—is used to provide broader scene context to the spatial subnet.

2. Frame-level Temporal Subnet

A bi-directional Recurrent Neural Network (RNN) processes the spatial embeddings across frames within a video segment. This captures short-term motion cues necessary for identifying immediate actions.

3. Segment-level Temporal Subnet

Complex activities (e.g., "cleaning a microwave") are composed of multiple sub-activities (e.g., "reaching," "cleaning," "moving"). This layer employs an attention mechanism and a secondary RNN to model long-term dependencies across these segments.

Experimental Setup

The model was evaluated on two primary benchmarks:

  • CAD-120: A video dataset of daily activities used to test sub-activity detection and object affordance (the potential use of an object).
  • V-COCO: A subset of MS-COCO used to evaluate the model's performance on static images.

Results and Discussion

The experimental results demonstrate that sophisticated spatial-temporal modeling can outperform methods relying on specialized sensor data (like depth).

Quantitative Performance

On the CAD-120 dataset, the GCN-based variant achieved an F1 score of 88.9% for sub-activity detection and 92.6% for object affordance. The GCN proved more robust in complex scenes with multiple objects. Conversely, Capsule Networks performed better on the V-COCO dataset, which typically features simpler human-object pairs, achieving a competitive 47.26 mAP.

Qualitative Analysis

The model successfully identifies the progression of interactions and the changing affordances of objects throughout a video sequence.

Qualitative results on a video sequence showing the detection of sub-activities (reaching, moving, cleaning) and corresponding object affordances.

Key Insights

  • The Importance of Spatial Modeling: Ablation studies showed that removing the spatial modeling component caused sub-activity detection accuracy to drop significantly (to 61.5%), proving that object detection alone is insufficient for HOI.
  • Human Context: Appending human node features to object nodes significantly boosted affordance detection, as the human's action provides a strong prior for the object's state.

Conclusion

This work establishes that decoupling spatial and temporal modeling is an effective strategy for video-based HOI. By achieving state-of-the-art results using only visual features, the proposed framework offers a more scalable and generalizable solution for real-world applications. Future work will focus on improving Capsule Network efficiency for multi-object scenes and refining the model's ability to distinguish between visually similar actions.

Summary updated

Abstract

Human-Object Interaction detection from a video clip can be considered as a special case of video-based Visual-Relationship Detection wherein the subject must be a human. Specifically, it involves detecting the humans and objects in the clip as well as the interactions between them. Conventionally, the problem has been formulated as a space-time graph inference problem over the video clip features. In this work, we explore alternate spatial approaches for detecting Human-Object Interactions. We consider a hierarchical setup that decouples spatial and temporal aspects of the problem and analyse the impacts of a variety of design choices for the spatial networks. Particularly, to capture spatial relationships in the scene, we analyze the effectiveness of the traditionally used Graph Convolutional Networks against Convolutional Networks and Capsule Networks. Unlike current approaches, we avoid using ground truth data like depth maps or 3D human pose during inference, thus increasing generalization across non-RGBD datasets as well. We demonstrate a comprehensive analysis of the exploration, both quantitatively and qualitatively, while achieving state-of-the-art results in human-object interaction detection (88.9% and 92.6%) and anticipation tasks of CAD-120 and competitive results on image based HOI detection in V-COCO dataset, setting a new benchmark for visual features based approaches.
BibTeX
@inproceedings{dabral2021exploration,
  author = {Rishabh Dabral and Srijon Sarkar and Sai Praneeth Reddy and Ganesh Ramakrishnan},
  title = {{Exploration of Spatial and Temporal Modeling Alternatives for HOI}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=8l2c55QI9t},
}
Figure