LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos

Sai Praneeth Reddy Sunkesula, Rishabh Dabral, Ganesh Ramakrishnan

Abstract

Analyzing the interactions between humans and objects from a video includes identification of the relationships between humans and the objects present in the video. It can be thought of as a specialized version of Visual Relationship Detection, wherein one of the objects must be a human. While traditional methods formulate the problem as inference on a sequence of video segments, we present a hierarchical approach, LIGHTEN, to learn visual features to effectively capture spatio-temporal cues at multiple granularities in a video. Unlike current approaches, LIGHTEN avoids using ground truth data like depth maps or 3D human pose, thus increasing generalization across non-RGBD datasets as well. Furthermore, we achieve the same using only the visual features, instead of the commonly used hand-crafted spatial features. We achieve state-of-the-art results in human-object interaction detection (88.9% and 92.6%) and anticipation tasks of CAD-120 and competitive results on image based HOI detection in V-COCO dataset, setting a new benchmark for visual features based approaches.

Introduction

Understanding how humans interact with objects—such as "pouring water" or "opening a fridge"—is a cornerstone of computer vision. While image-based Human-Object Interaction (HOI) has seen significant progress, video-based HOI presents a much steeper challenge. In a video, the orientation of objects changes, and many actions (like "opening" vs. "closing" a jar) appear identical in a single frame; they can only be distinguished by analyzing the sequence of movements over time.

Examples of various human-object interactions including holding, hitting, and kicking.

Existing video HOI models often rely on hand-crafted features derived from ground-truth data, such as 3D human poses, object centroids, or depth maps. This reliance makes them brittle: in the absence of a depth sensor or a perfect pose estimator, these models fail. Furthermore, most models only look at temporal cues at a coarse "segment" level, missing the fine-grained, frame-by-frame nuances that define an interaction.

The LIGHTEN Architecture

The authors propose LIGHTEN (Learning Interactions with Graph and Hierarchical TEmporal Networks), an end-to-end pipeline that operates in three distinct stages to capture spatio-temporal cues at multiple granularities:

  1. Spatial Subnet: For every frame, the model treats the human and objects as nodes in a graph. It uses a Graph Convolutional Network (GCN) to learn the spatial relationships between them. Crucially, it uses an adaptive adjacency matrix that learns which interactions matter most for a specific input, rather than relying on a fixed structure.
  2. Frame-level Temporal Subnet: The spatial features from the GCN are fed into a bidirectional RNN. This stage captures the fine-grained motion within a single video segment (e.g., the specific arc of a hand during a "reach").
  3. Segment-level Temporal Subnet: Interactions are often sequential; "placing" usually follows "moving." This third stage uses another RNN to model the dependencies between consecutive segments. It employs an attention mechanism to identify which frames within a segment are most representative before passing that information up the hierarchy.

The entire system is trained on pure RGB visual features extracted from a standard ResNet-50 backbone, making it highly generalizable to "in-the-wild" videos.

Experimental Evaluation

The model was evaluated on two primary datasets:

  • CAD-120: A benchmark for video HOI containing 120 RGB-D videos of daily activities. LIGHTEN intentionally ignored the depth and 3D pose data, using only RGB frames and 2D bounding boxes.
  • V-COCO: A subset of MS-COCO for image-based HOI, used to demonstrate the model's ability to handle static images by treating them as single-frame sequences.
A CAD-120 video frame showing sub-activity (reaching) and object affordance (reachable) labels.

Key Results

LIGHTEN achieved state-of-the-art results on the CAD-120 dataset, even without the 3D/depth data used by competitors:

  • Sub-activity Detection: Achieved an 88.9% F1 score, matching the previous best (GPNN) while using only visual features.
  • Object Affordance Detection: Reached a 92.6% F1 score, a significant improvement over the previous state-of-the-art of 91.1%.
  • Anticipation: The model also excels at predicting future interactions, achieving 76.4% in sub-activity anticipation, significantly higher than the 75.6% of previous methods.

Conclusion

LIGHTEN demonstrates that learned visual features can outperform hand-crafted 3D features when processed through a sufficiently rich hierarchical structure. By modeling time at both frame and segment levels, the model effectively disambiguates similar actions. While its performance on static images (V-COCO) is competitive (38.28 mAP), future work could involve integrating more sophisticated spatial reasoning modules for single-image tasks or extending the model to handle multi-human interactions.

Summary updated

Abstract

Analyzing the interactions between humans and objects from a video includes identification of the relationships between humans and the objects present in the video. It can be thought of as a specialized version of Visual Relationship Detection, wherein one of the objects must be a human. While traditional methods formulate the problem as inference on a sequence of video segments, we present a hierarchical approach, LIGHTEN, to learn visual features to effectively capture spatio-temporal cues at multiple granularities in a video. Unlike current approaches, LIGHTEN avoids using ground truth data like depth maps or 3D human pose, thus increasing generalization across non-RGBD datasets as well. Furthermore, we achieve the same using only the visual features, instead of the commonly used hand-crafted spatial features. We achieve state-of-the-art results in human-object interaction detection (88.9% and 92.6%) and anticipation tasks of CAD-120 and competitive results on image based HOI detection in V-COCO dataset, setting a new benchmark for visual features based approaches. Code for LIGHTEN is available at https://github.com/praneeth11009/LIGHTEN-Learning-Interactions-with-Graphs-and-Hierarchical-TEmporal-Networks-for-HOI
BibTeX
@inproceedings{sunkesula2020lighten,
  author = {Sai Praneeth Reddy Sunkesula and Rishabh Dabral and Ganesh Ramakrishnan},
  title = {{LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=lkbyqViKL0},
}
Figure