Abstract
Analyzing the interactions between humans and objects from a video includes identification of the relationships between humans and the objects present in the video. It can be thought of as a specialized version of Visual Relationship Detection, wherein one of the objects must be a human. While traditional methods formulate the problem as inference on a sequence of video segments, we present a hierarchical approach, LIGHTEN, to learn visual features to effectively capture spatio-temporal cues at multiple granularities in a video. Unlike current approaches, LIGHTEN avoids using ground truth data like depth maps or 3D human pose, thus increasing generalization across non-RGBD datasets as well. Furthermore, we achieve the same using only the visual features, instead of the commonly used hand-crafted spatial features. We achieve state-of-the-art results in human-object interaction detection (88.9% and 92.6%) and anticipation tasks of CAD-120 and competitive results on image based HOI detection in V-COCO dataset, setting a new benchmark for visual features based approaches.
Introduction
Understanding how humans interact with objects—such as "pouring water" or "opening a fridge"—is a cornerstone of computer vision. While image-based Human-Object Interaction (HOI) has seen significant progress, video-based HOI presents a much steeper challenge. In a video, the orientation of objects changes, and many actions (like "opening" vs. "closing" a jar) appear identical in a single frame; they can only be distinguished by analyzing the sequence of movements over time.
Existing video HOI models often rely on hand-crafted features derived from ground-truth data, such as 3D human poses, object centroids, or depth maps. This reliance makes them brittle: in the absence of a depth sensor or a perfect pose estimator, these models fail. Furthermore, most models only look at temporal cues at a coarse "segment" level, missing the fine-grained, frame-by-frame nuances that define an interaction.
The LIGHTEN Architecture
The authors propose LIGHTEN (Learning Interactions with Graph and Hierarchical TEmporal Networks), an end-to-end pipeline that operates in three distinct stages to capture spatio-temporal cues at multiple granularities:
- Spatial Subnet: For every frame, the model treats the human and objects as nodes in a graph. It uses a Graph Convolutional Network (GCN) to learn the spatial relationships between them. Crucially, it uses an adaptive adjacency matrix that learns which interactions matter most for a specific input, rather than relying on a fixed structure.
- Frame-level Temporal Subnet: The spatial features from the GCN are fed into a bidirectional RNN. This stage captures the fine-grained motion within a single video segment (e.g., the specific arc of a hand during a "reach").
- Segment-level Temporal Subnet: Interactions are often sequential; "placing" usually follows "moving." This third stage uses another RNN to model the dependencies between consecutive segments. It employs an attention mechanism to identify which frames within a segment are most representative before passing that information up the hierarchy.
The entire system is trained on pure RGB visual features extracted from a standard ResNet-50 backbone, making it highly generalizable to "in-the-wild" videos.
Experimental Evaluation
The model was evaluated on two primary datasets:
- CAD-120: A benchmark for video HOI containing 120 RGB-D videos of daily activities. LIGHTEN intentionally ignored the depth and 3D pose data, using only RGB frames and 2D bounding boxes.
- V-COCO: A subset of MS-COCO for image-based HOI, used to demonstrate the model's ability to handle static images by treating them as single-frame sequences.
Key Results
LIGHTEN achieved state-of-the-art results on the CAD-120 dataset, even without the 3D/depth data used by competitors:
- Sub-activity Detection: Achieved an 88.9% F1 score, matching the previous best (GPNN) while using only visual features.
- Object Affordance Detection: Reached a 92.6% F1 score, a significant improvement over the previous state-of-the-art of 91.1%.
- Anticipation: The model also excels at predicting future interactions, achieving 76.4% in sub-activity anticipation, significantly higher than the 75.6% of previous methods.
Conclusion
LIGHTEN demonstrates that learned visual features can outperform hand-crafted 3D features when processed through a sufficiently rich hierarchical structure. By modeling time at both frame and segment levels, the model effectively disambiguates similar actions. While its performance on static images (V-COCO) is competitive (38.28 mAP), future work could involve integrating more sophisticated spatial reasoning modules for single-image tasks or extending the model to handle multi-human interactions.
Summary updated