Contrastively Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Grasping

Yan Zha, Siddhant Bhambri, Lin Guan

Abstract

Conventional works that learn grasping affordance from demonstrations need to explicitly predict grasping configurations, such as gripper approaching angles or grasping preshapes. Classic motion planners could then sample trajectories by using such predicted configurations. In this work, our goal is instead to fill the gap between affordance discovery and affordance-based policy learning by integrating the two objectives in an end-to-end imitation learning framework based on deep neural networks. From a psychological perspective, there is a close association between attention and affordance. Therefore, with an end-to-end neural network, we propose to learn affordance cues as visual attention that serves as a useful indicating signal of how a demonstrator accomplishes tasks, instead of explicitly modeling affordances. To achieve this, we propose a contrastive learning framework that consists of a Siamese encoder and a trajectory decoder. We further introduce a coupled triplet loss to encourage the discovered affordance cues to be more affordance-relevant. Our experimental results demonstrate that our model with the coupled triplet loss achieves the highest grasping success rate in a simulated robot environment.

Introduction

In robotics, "affordance" refers to the relationship between an object’s physical properties and the actions an agent can perform on it (e.g., a handle "affords" grabbing). Traditional approaches usually split this into two steps: first, a vision system predicts a grasp configuration (like an angle or hand shape), and second, a motion planner calculates the path.

This decoupling is often brittle and requires labor-intensive manual labeling of "correct" grasp points. Furthermore, humans don't explicitly calculate coordinates; we use visual attention to focus on relevant parts of an object based on the task. This research aims to bridge the gap between affordance discovery and policy execution by learning visual attention directly from human demonstrations in an end-to-end fashion.

Methods: Learning Attention as Affordance

The researchers propose a contrastive learning framework composed of two main parts:

  1. Siamese Encoder & Interaction Segments: The model extracts "interaction segments"—the specific part of a demonstration where the robot actually moves the object. Because these segments are low-dimensional (just state and action vectors), they are easier to cluster into affordance categories (e.g., "grasping the handle" vs. "grasping the body").
  2. Trajectory Encoder & Visual Attention: This module processes high-dimensional depth images to generate an attention map (the affordance cue) that highlights relevant regions of the object.
  3. Coupled Triplet Loss: This is the core innovation. Standard triplet losses compare similar and dissimilar examples. The coupled version forces the high-dimensional visual embeddings to align with the low-dimensional affordance embeddings from the interaction segments. This "guides" the visual encoder to pay attention to the features that actually matter for the physical interaction.
Figure 1: Visualization of the domain, depth observations, and the resulting affordance cues (attention maps) across three steps for two different mug geometries.

Experimental Setup

The study used the PyBullet simulator featuring a Franka Panda robotic arm. The dataset consisted of 24 different mugs with varying shapes, sizes, and handle configurations. 27 expert trajectories were collected via teleoperation and categorized into three distinct affordance types: body-grasp, handle-left-right-grasp, and handle-front-back-grasp.

Figure 6: The three primary affordance types used in the study: body-grasp, handle-left-right-grasp, and handle-front-back-grasp.

Key Results and Insights

The model was evaluated on its ability to pick up a mug and lift it 5cm. Testing was performed on 20 randomly sampled mugs over 20 grasp attempts.

  • Ours (Full Model): 65% success rate.
  • Ours (No Contrastive Learning): 45% success rate.
  • Ours (Normal Triplet Loss): 35% success rate.
  • Baseline (Standard Behavior Cloning): 25% success rate.

The researchers found that using a standard triplet loss on visual data alone actually hurt performance compared to no contrastive learning at all. However, coupling the visual data to the low-dimensional interaction data boosted success significantly. This suggests that visual data contains too much "noise" (like initial positions) for standard contrastive learning to find the true affordance signal without guidance. Qualitatively, the model successfully generated attention maps that shifted based on the mug's geometry—focusing on the handle for mugs where that was the demonstrated affordance, and the body for others.

Conclusion

By training the affordance discovery and the movement policy together, the robot learns features that are specifically useful for the task it is trying to execute, rather than general-purpose geometric features. Future work will focus on task-conditioned affordance cues—meaning the robot might look at the handle if asked to move the mug, but look at the rim if asked to clean it—and transitioning these cues to real-world visual data.

Summary updated

Abstract

Conventional works that learn grasping affordance from demonstrations need to explicitly predict grasping configurations, such as gripper approaching angles or grasping preshapes. Classic motion planners could then sample trajectories by using such predicted configurations. In this work, our goal is instead to fill the gap between affordance discovery and affordance-based policy learning by integrating the two objectives in an end-to-end imitation learning framework based on deep neural networks. From a psychological perspective, there is a close association between attention and affordance. Therefore, with an end-to-end neural network, we propose to learn affordance cues as visual attention that serves as a useful indicating signal of how a demonstrator accomplishes tasks, instead of explicitly modeling affordances. To achieve this, we propose a contrastive learning framework that consists of a Siamese encoder and a trajectory decoder. We further introduce a coupled triplet loss to encourage the discovered affordance cues to be more affordance-relevant. Our experimental results demonstrate that our model with the coupled triplet loss achieves the highest grasping success rate in a simulated robot environment. Our project website can be accessed at https://sites.google.com/asu.edu/affordance-aware-imitation/project.
BibTeX
@inproceedings{zha2021contrastively,
  author = {Yan Zha and Siddhant Bhambri and Lin Guan},
  title = {{Contrastively Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Grasping}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
}
Figure