Grounding human-object interaction to affordance behavior in multimodal datasets

Alexander Henlein, Anju Gopinath, Nikhil Krishnaswamy, Alexander Mehler, James PustejovskyFrontiers Artif. Intell 2023 · DOI

Abstract

Traditional Human-Object Interaction (HOI) detection often relies on simple "subject-verb-object" triplets that lack functional depth. This paper introduces a framework to bridge the gap between basic action recognition and true functional understanding by distinguishing between Gibsonian affordances (physical possibilities) and telic affordances (conventional or intended use). By augmenting the HICO-DET dataset and adapting a transformer-based model (AffordanceUPT), the researchers demonstrate that this higher-level abstraction allows AI to generalize more effectively to novel objects and actions.

The Intentionality Gap in Computer Vision

In current computer vision models, there is a fundamental lack of intentionality. For instance, a model might recognize a person holding a bottle, but it fails to distinguish between exploiting the bottle's physical structure (Gibsonian) and using it for its intended purpose, such as drinking (telic).

The authors argue that AI lacks an understanding of "habitats"—the specific preconditions, such as object orientation, that must be met for an object's intended function to be activated. For example, a cup only affords "drinking" if it is oriented upright. Standard datasets fail to capture these nuances, leading to models that can recognize familiar actions but struggle to understand why those actions are possible or how they apply to unseen objects.

Methodology: AffordanceUPT and Pose Estimation

The researchers developed a multi-pronged approach to address these limitations:

  1. Theoretical Framework: They utilized the distinction between Gibsonian (G) and Telic (T) affordances to categorize interactions.
  2. AffordanceUPT: An adaptation of the Unary-Pairwise Transformer (UPT). This model identifies entities (unary tokens) and then analyzes the relations between them (pairwise tokens). The architecture is modular, making affordance detection independent of the specific object detector.
  3. Pose Estimation and Habitats: They integrated PoseContrast, a viewpoint estimation system, to detect object orientations (habitats). This allowed them to test if specific orientations correlate with affordance types.
Figure 4: Visualization of 3D orientation and bounding boxes for a person riding a horse, illustrating the concept of 'habitat' in affordance detection.

Key Insights

  • Abstraction for Generalization: By grouping hundreds of specific verbs into just two categories (G or T), the model learns the underlying nature of interactions. This enables the model to identify a "telic" interaction with a car even if it was only trained on bicycles.
  • The Habitat Correlation: The study demonstrates that certain orientations are functional preconditions. "Drinking" (Telic) shows a strong correlation with a bottle being tilted toward the person, whereas "holding" (Gibsonian) occurs at various angles.
  • Visual Clustering of Intent: Using t-SNE visualizations, the authors found that training on the G/T distinction causes the model to naturally cluster objects by use. Bicycles and horses (both rideable) cluster together, as do bottles and cups (both containers).

Results and Evaluation

The AffordanceUPT model showed significant robustness in zero-shot scenarios. When tested on novel objects not seen during training, its performance (mAP) only dropped by 10–13%, whereas traditional HOI models often fail entirely.

Figure 6: AffordanceUPT evaluation across object types. The graph shows the number of Gibsonian (blue), telic (orange), and general (green) occurrences alongside mAP scores (line graph).

The model achieved a mean Average Precision (mAP) of 27.58 on the augmented HICO-DET split. However, the study also revealed a major gap in current pose estimation: while PoseContrast performed well on static objects (0.83 accuracy for apples), it struggled significantly with objects being actively manipulated (0.08 for knives), highlighting the need for better "in-the-wild" datasets.

Conclusion

This research highlights the importance of grounding HOI in functional affordances. While the current pose-annotated dataset is relatively small (1,200 images), it provides a foundation for future work. Next steps include scaling the dataset via crowdsourcing and integrating AffordanceUPT with large-scale grounded language models like CLIP to further enhance zero-shot reasoning and handle the complex orientations found in real-world human usage.

Summary updated

BibTeX
@article{henlein2023grounding,
  author = {Alexander Henlein and Anju Gopinath and Nikhil Krishnaswamy and Alexander Mehler and James Pustejovsky},
  title = {{Grounding human-object interaction to affordance behavior in multimodal datasets}},
  journal = {Frontiers Artif. Intell.},
  year = {2023},
  doi = {10.3389/frai.2023.1084740},
}
Figure