HEIR: Learning Human-Entity Interactions with Functional Roles

Di Wen, Wenhao Guo, Yuedong Tan, Yun Huang, Minheng Wu, Zhihang Chen, Haiwen Sun, Fei Teng, Zhiyuan Gao, Yufeng Zhang, Yuanhao Luo, Jingqi Zhang, Yufan Chen, Junwei Zheng, Ruiping Liu, Jiale Wei, Kailun Yang, Kunyu PengarXiv 2026 · arXiv

TL;DR

HEIR introduces a new benchmark and a novel model, CoRISP, that shift the focus of Human-Object Interaction (HOI) from detecting isolated triplets to recovering complete, grounded participant-role sets. By explicitly modeling functional roles (like "instrument" or "support") and shared entity identities, this work provides a more robust framework for embodied AI agents to understand and coordinate in complex, multi-actor environments.

What Problem Does This Paper Address?

Traditional HOI research typically focuses on detecting individual <person, action, object> relations. While useful, this "triplet-centric" view often misses the bigger picture. For example, if a parent is helping a child ride a bike, a standard model might detect "child riding bike" and "parent touching bike" as two separate, unrelated events.

The authors argue that true understanding requires recovering the complete event structure: identifying all participants, assigning their specific functional roles, and maintaining entity identity across concurrent actions. Existing benchmarks often undermeasure this composition, potentially rewarding models that find one correct link while missing other crucial participants in the same event.

Methods

The authors propose CoRISP (Compositional Role-aware Interaction Set Prediction), a model designed to predict normalized distributions over entire participant-role sets. Key technical innovations include:

  • Role-Preserving Context Aggregation: CoRISP shares information across three linked "neighborhoods": co-participants within a single event, different actions involving the same person-entity pair, and events across the image that share a common entity.
  • Structural Potentials: The model incorporates cardinality potentials (to reason about the number of participants) and role-multiplicity potentials (to handle cases where multiple entities fill the same role, like two people high-fiving a third).
  • Exact Set Normalization: Unlike models that treat each relation independently, CoRISP uses a dynamic programming approach to compute exact per-event normalization. This ensures that the predicted set of participants is globally consistent for that specific action.
  • Functional Role Ontology: Instead of simple object labels, the paper defines six functional roles: target, instrument, support, source, destination, and constraint.

Datasets

The paper introduces the HEIR (Human-Entity Interactions with Functional Roles) benchmark:

  • Scale: 18,730 images featuring 105 actions, 437 nouns, and 6 functional roles.
  • Complexity: Over 50% of images contain multiple actors, and 62% contain multiple actions. It specifically includes challenging cases like shared entities, role changes, and repeated fillers (multiple entities in one role).
  • Evaluation: It introduces Complete-set AP, a metric that requires a model to recover the entire set of participants for an event to get credit, alongside standard relation-level AP.

The authors also evaluate on the standard V-COCO dataset to demonstrate CoRISP's generalizability.

Figure 1: HEIR structure. (a) Shared participants across events. (b) Interaction structure (top): HHI denotes images with at least one human–human relation; other images are grouped by acting- person count versus distinct participant count. Bottom: actor counts. All split images are included. (c) Role shares for the 24 most frequent actions. (d) Training support of 2,972 observed classes, including 145 unseen classes with zero training relations.

Key Insights and Ideas

  • Relation AP $\neq$ Event AP: A surprising finding is that models which rank highly on individual relation detection often drop significantly when evaluated on complete event recovery. Improving local link accuracy doesn't automatically lead to better structural understanding.
  • Role Ambiguity is Context-Dependent: The function of an object (its role) is not fixed by its noun category. A "box" can be a target (being opened), a support (being sat on), or a destination (having items placed in it). HEIR forces models to resolve this ambiguity.
  • The Value of Shared Identity: By tracking the same entity across different concurrent actions, CoRISP can better constrain the possible roles and participants, leading to more coherent scene interpretations.

Key Results

  • HEIR Performance: CoRISP leads the field in complex scenarios, outperforming the strongest baselines by 2.87 points on "Repeat" role events and 3.82 points on "Shared" participant images in terms of Set mAP.
  • V-COCO State-of-the-Art: On the established V-COCO benchmark, CoRISP achieves a Role AP of 73.72/76.23 and a Complete-set AP of 61.06/68.59 for two-slot actions, setting a new high bar for joint slot recovery.
  • Qualitative Success: Qualitative analysis shows CoRISP successfully recovering all participants in multi-person events (like a group high-five) where baseline models often only pick up a single, most "obvious" partner.

Limitations and Future Work

  • Candidate Coverage: The model's performance is currently capped by the underlying object detector's ability to propose all relevant participants. If the detector misses a small or occluded object, the "complete set" can never be recovered.
  • Occlusion: While CoRISP handles shared entities well, heavily occluded participants (e.g., in a tight group hug) remain a significant challenge for both localization and role assignment.
  • Future Directions: The authors suggest that improving the "front end" (localization and noun recognition) while preserving CoRISP's strong ranked-set recovery is a promising path forward for more capable embodied AI.

Summary updated

Abstract

Understanding human-entity interactions requires recovering each person-action event's participants, roles, and shared identities. This structure can support embodied agents by clarifying who acts on which entities and how, informing anticipation and coordination in shared environments. Standard HOI metrics score individual links, leaving complete event composition undermeasured. We introduce HEIR (Human-Entity Interactions with Functional Roles), an image benchmark for complete grounded participant-role sets across object, interpersonal, and self-directed interactions. It contains 18,730 images, six roles, 105 actions, and 437 nouns, with shared entities, role changes, and repeated fillers; 51.6% of images contain multiple actors and 62.1% contain multiple actions. HEIR pairs relation AP with complete-set AP and structural evaluation. We also introduce CoRISP (Compositional Role-aware Interaction Set Prediction), which uses shared entity identities to combine role-conditioned evidence and predict normalized participant-role sets. Cardinality and role-multiplicity potentials couple assignments through event size and role composition, with exact per-event normalization. Across 16 baselines, relation and complete-event rankings diverge even after aligning action weights. CoRISP leads the evaluated systems on repeated-role events and shared-participant images in HEIR by 2.87 and 3.82 Set mAP points, respectively. On V-COCO, CoRISP achieves 73.72/76.23 role AP and 61.06/68.59 complete-set AP on two-slot actions under Scenarios 1/2. These results show the value of learning and evaluating event composition alongside individual relations. The code and dataset are publicly available at https://github.com/Kratos-Wen/HEIR.
BibTeX
@misc{wen2026heir,
  author = {Di Wen and Wenhao Guo and Yuedong Tan and Yun Huang and Minheng Wu and Zhihang Chen and Haiwen Sun and Fei Teng and Zhiyuan Gao and Yufeng Zhang and Yuanhao Luo and Jingqi Zhang and Yufan Chen and Junwei Zheng and Ruiping Liu and Jiale Wei and Kailun Yang and Kunyu Peng},
  title = {{HEIR: Learning Human-Entity Interactions with Functional Roles}},
  year = {2026},
  eprint = {2609.35955},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.35955},
}
Figure