A Graph-based Interactive Reasoning for Human-Object Interaction Detection

Dongming Yang, Yuexian Zou

TL;DR

This paper introduces "in-GraphNet," a novel graph-based framework for Human-Object Interaction (HOI) detection. It addresses the limitations of existing methods by explicitly modeling interactive semantics between humans and objects through a graph structure, leading to improved detection accuracy without requiring human pose annotations.

What Problem Does This Paper Address?

Human-Object Interaction (HOI) detection aims to identify not just what objects are present in an image, but also how humans are interacting with them (e.g., "person riding bicycle"). This is a challenging task because it requires understanding complex relationships and actions, going beyond simple object recognition. Existing HOI detection methods often rely on appearance features from convolutional neural networks (CNNs) and may neglect the crucial interactive semantics between humans and objects. Some advanced methods incorporate additional cues like human pose or body parts, but this significantly increases computational burden and requires costly annotations. The paper aims to bridge this gap by developing a method that effectively reasons about these interactions using a graph-based approach, while remaining computationally efficient and annotation-free regarding human pose.

Methods

The core of the proposed method is the in-Graph model, which reasons about interactive semantics between two visual targets. It comprises three key steps:

  1. Project Function: This function maps the feature tensors of two input targets (e.g., human and object) from the standard convolutional space into a unified "interactive space." This space is designed to facilitate the construction of a fully-connected graph where each target becomes a node. The process involves feature conversion, weights inference, and a linear combination to aggregate semantic information.
  2. Message Passing Process: Once in the interactive space, a graph-based message passing mechanism is employed. This allows semantic information to propagate and be aggregated among all nodes (targets) in the graph. The authors use a single-layer convolution for efficient communication, enabling the model to capture scene-wide and instance-wide interactions.
  3. Update Function: After reasoning in the interactive space, the update function provides a reverse projection. It transforms the reasoned semantic features back into the convolutional space, enhancing the representation for subsequent HOI detection tasks.

The in-GraphNet framework assembles these in-Graph models into a multi-stream architecture. It utilizes two levels of in-Graphs:

  • Scene-wide in-Graphs: These include a human-scene in-Graph and an object-scene in-Graph, capturing interactions related to the broader environment.
  • Instance-wide in-Graph: This is a human-object in-Graph, focusing on direct interactions between the human and object.

These different semantic representations are then combined within three branches (human-centric, object-centric, and spatial configuration) to ultimately predict HOI triplets.

Datasets

The paper evaluates its proposed method on two widely-used HOI detection benchmarks:

  • V-COCO: A subset of MS COCO, containing 10,346 images and 26 common HOI annotations.
  • HICO-DET: A larger dataset with approximately 48k images and 600 HOI categories, covering over 150K annotated (human, object) pairs.

Key Insights and Ideas

  • Explicit Interactive Reasoning: The paper argues that simply concatenating features or using attention mechanisms is insufficient. Explicitly modeling the interaction between visual targets via a graph structure is key to capturing richer semantic dependencies.
  • Two-Level Graph Reasoning: The hierarchical approach of using both scene-wide and instance-wide in-Graphs allows the model to capture different facets of interaction – environmental context and direct human-object relationships.
  • Pose-Free and Annotation-Free: A significant advantage is that the in-GraphNet achieves state-of-the-art performance without relying on human pose annotations, making it more practical and less computationally demanding.
  • Unified Interactive Space: The "project function" is crucial for creating a common ground where different targets' features can be meaningfully integrated and reasoned upon.

Key Results

The proposed in-GraphNet demonstrates strong performance on both V-COCO and HICO-DET datasets:

  • On V-COCO, it achieves 48.9 mAP, outperforming the baseline by 4.1 points (a relative improvement of 9.4%).
  • On HICO-DET (default full mode), it achieves 17.72 mAP, outperforming the baseline by 2.3 points (a relative improvement of 15%).
  • Notably, the in-GraphNet, as a pose-free method, outperforms many existing pose-based methods on these benchmarks.

Qualitative results show the model's ability to detect HOIs with higher scores and to handle complex scenes with multiple interacting people and objects.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the scene-wide in-Graphs are adept at modeling environment-related interactions, the instance-wide in-Graph is better for interactions related to human pose. This suggests a potential for further refinement in how these two levels are integrated or weighted. The authors also note that the choice of the number of nodes in the interactive space (N=512 yielding the best result) is an empirical finding that could be explored further.
  • Future Work: The paper's success in improving HOI detection without pose information opens avenues for further research into more efficient and robust graph-based reasoning for complex visual understanding tasks. The framework's modularity suggests it could be integrated with other advanced architectures.

Summary updated

Abstract

Human-Object Interaction (HOI) detection devotes to learn how humans interact with surrounding objects via inferring triplets of < human, verb, object >. However, recent HOI detection methods mostly rely on additional annotations (e.g., human pose) and neglect powerful interactive reasoning beyond convolutions. In this paper, we present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs, in which interactive semantics implied among visual targets are efficiently exploited. The proposed model consists of a project function that maps related targets from convolution space to a graph-based semantic space, a message passing process propagating semantics among all nodes and an update function transforming the reasoned nodes back to convolution space. Furthermore, we construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Beyond inferring HOIs using instance features respectively, the framework dynamically parses pairwise interactive semantics among visual targets by integrating two-level in-Graphs, i.e., scene-wide and instance-wide in-Graphs. Our framework is end-to-end trainable and free from costly annotations like human pose. Extensive experiments show that our proposed framework outperforms existing HOI detection methods on both V-COCO and HICO-DET benchmarks and improves the baseline about 9.4% and 15% relatively, validating its efficacy in detecting HOIs.
BibTeX
@inproceedings{yang2020graph,
  author = {Dongming Yang and Yuexian Zou},
  title = {{A Graph-based Interactive Reasoning for Human-Object Interaction Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/155},
}

Loading related content...

Figure