TL;DR
HEIR introduces a new benchmark and a novel model, CoRISP, that shift the focus of Human-Object Interaction (HOI) from detecting isolated triplets to recovering complete, grounded participant-role sets. By explicitly modeling functional roles (like "instrument" or "support") and shared entity identities, this work provides a more robust framework for embodied AI agents to understand and coordinate in complex, multi-actor environments.
What Problem Does This Paper Address?
Traditional HOI research typically focuses on detecting individual <person, action, object> relations. While useful, this "triplet-centric" view often misses the bigger picture. For example, if a parent is helping a child ride a bike, a standard model might detect "child riding bike" and "parent touching bike" as two separate, unrelated events.
The authors argue that true understanding requires recovering the complete event structure: identifying all participants, assigning their specific functional roles, and maintaining entity identity across concurrent actions. Existing benchmarks often undermeasure this composition, potentially rewarding models that find one correct link while missing other crucial participants in the same event.
Methods
The authors propose CoRISP (Compositional Role-aware Interaction Set Prediction), a model designed to predict normalized distributions over entire participant-role sets. Key technical innovations include:
- Role-Preserving Context Aggregation: CoRISP shares information across three linked "neighborhoods": co-participants within a single event, different actions involving the same person-entity pair, and events across the image that share a common entity.
- Structural Potentials: The model incorporates cardinality potentials (to reason about the number of participants) and role-multiplicity potentials (to handle cases where multiple entities fill the same role, like two people high-fiving a third).
- Exact Set Normalization: Unlike models that treat each relation independently, CoRISP uses a dynamic programming approach to compute exact per-event normalization. This ensures that the predicted set of participants is globally consistent for that specific action.
- Functional Role Ontology: Instead of simple object labels, the paper defines six functional roles: target, instrument, support, source, destination, and constraint.
Datasets
The paper introduces the HEIR (Human-Entity Interactions with Functional Roles) benchmark:
- Scale: 18,730 images featuring 105 actions, 437 nouns, and 6 functional roles.
- Complexity: Over 50% of images contain multiple actors, and 62% contain multiple actions. It specifically includes challenging cases like shared entities, role changes, and repeated fillers (multiple entities in one role).
- Evaluation: It introduces Complete-set AP, a metric that requires a model to recover the entire set of participants for an event to get credit, alongside standard relation-level AP.
The authors also evaluate on the standard V-COCO dataset to demonstrate CoRISP's generalizability.
Key Insights and Ideas
- Relation AP $\neq$ Event AP: A surprising finding is that models which rank highly on individual relation detection often drop significantly when evaluated on complete event recovery. Improving local link accuracy doesn't automatically lead to better structural understanding.
- Role Ambiguity is Context-Dependent: The function of an object (its role) is not fixed by its noun category. A "box" can be a target (being opened), a support (being sat on), or a destination (having items placed in it). HEIR forces models to resolve this ambiguity.
- The Value of Shared Identity: By tracking the same entity across different concurrent actions, CoRISP can better constrain the possible roles and participants, leading to more coherent scene interpretations.
Key Results
- HEIR Performance: CoRISP leads the field in complex scenarios, outperforming the strongest baselines by 2.87 points on "Repeat" role events and 3.82 points on "Shared" participant images in terms of Set mAP.
- V-COCO State-of-the-Art: On the established V-COCO benchmark, CoRISP achieves a Role AP of 73.72/76.23 and a Complete-set AP of 61.06/68.59 for two-slot actions, setting a new high bar for joint slot recovery.
- Qualitative Success: Qualitative analysis shows CoRISP successfully recovering all participants in multi-person events (like a group high-five) where baseline models often only pick up a single, most "obvious" partner.
Limitations and Future Work
- Candidate Coverage: The model's performance is currently capped by the underlying object detector's ability to propose all relevant participants. If the detector misses a small or occluded object, the "complete set" can never be recovered.
- Occlusion: While CoRISP handles shared entities well, heavily occluded participants (e.g., in a tight group hug) remain a significant challenge for both localization and role assignment.
- Future Directions: The authors suggest that improving the "front end" (localization and noun recognition) while preserving CoRISP's strong ranked-set recovery is a promising path forward for more capable embodied AI.
Summary updated