Abstract
Affordance learning—identifying "action possibilities" within an image—is a critical task for robotics and scene understanding. Traditional methods often struggle when visual appearances change or interactions become complex. This paper introduces Interactive Affinity, a novel framework that utilizes human pose as a structural guide to map specific body-part contacts to object regions. By transferring this "affinity" from human-demonstration images to static object images, the proposed method achieves state-of-the-art performance, particularly in generalizing to unseen objects and environments.
Introduction
In computer vision, "affordance" refers to the functional potential of an object, such as a chair being for sitting or a handle for grasping. Most existing models treat affordance as a simple image segmentation task, mapping visual textures directly to labels. However, this approach suffers from two major flaws: ambiguity (one object can have multiple functions) and poor generalization (models fail on visually novel but functionally similar objects).
The authors argue that the missing link is the human-object interaction. To truly understand affordance, a model must comprehend the "affinity" or the specific physical relationship between human body parts and object regions. By anchoring functional understanding in human pose, models can learn invariant representations of how objects are used.
Methods
The proposed pose-aided interactive affinity learning framework consists of two primary components designed to bridge the gap between interactive demonstrations and static objects:
- Interactive Feature Enhancement (IFE) Module: This module employs a cross-branch Transformer architecture to process two images simultaneously: an "interactive" image (showing a human using an object) and a "non-interactive" image (the object alone). The Transformer identifies common contextual cues, effectively teaching the static branch what to look for based on the demonstration branch.
- Keypoint Heuristic Perception (KHP) Scheme: This core innovation uses a Graph Convolutional Network (GCN) to process human pose keypoints. These keypoints act as a heuristic guide, indicating exactly which body parts (hands, hips, feet, etc.) are contacting specific object regions. This structural information is more robust than raw pixels, allowing the network to ignore background noise and focus on actual contact zones.
The Contact-driven Affordance Learning (CAL) Dataset
To support this research, the authors introduced the Contact-driven Affordance Learning (CAL) dataset. It comprises 5,258 images (2,689 interactive and 2,569 non-interactive) across 23 affordance categories and 47 object categories. Unlike standard datasets, CAL annotates contact points for six specific body parts: Hands, Feet, Mouth, Hips, Back, and Outside (representing environment-object contact, such as wheels on a road).
Key Results
The model was evaluated in both "Seen" (familiar objects) and "Unseen" (entirely new object categories) settings. In the Unseen setting—the primary test of generalization—the proposed model significantly outperformed previous state-of-the-art methods. For the KLD metric (where lower is better), the model achieved 2.823, compared to HSNet’s 3.016 and SegFormer’s 5.957.
Qualitative results demonstrate that the model accurately identifies narrow contact regions, such as the blade of a knife or the handle of a mug, where standard segmentation models often highlight the entire object or fail entirely. Ablation studies confirmed that both the IFE and KHP modules are essential for this performance gain.
Conclusion and Future Work
While the Interactive Affinity framework shows significant promise, challenges remain. Performance gaps exist in categories with sparse data, such as "Back" contacts, and heavy occlusion during interaction can hinder feature extraction. Future work will explore "co-relation" modeling—understanding how the movement of one body part implies the position of another—to better handle occlusions and improve performance in data-sparse scenarios.
Summary updated