Contrastively Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Grasping

Yantian Zha, Siddhant Bhambri, Lin Guan

TL;DR

This paper introduces an end-to-end imitation learning framework that simultaneously discovers grasping affordances and learns manipulation policies directly from human demonstrations. By framing affordances as latent visual attention maps and optimizing them via a novel coupled triplet loss, the robot learns where and how to grasp complex objects without requiring tedious, pixel-level ground-truth affordance labels.

What Problem Does This Paper Address?

When humans look at a mug, we don't just see a geometric shape; we implicitly see its affordances—the potential actions we can perform on it based on its features (e.g., grasping it by the handle to pour, or by the body to move it).

In robotics, learning these object-action-effect relationships from demonstrations typically happens in a decoupled, two-step process:

  1. An affordance predictor explicitly estimates grasping configurations (like gripper approach angles or preshapes).
  2. A separate, classic motion planner uses these configurations to execute a trajectory.

This decoupled approach is often brittle, engineered, and heavily reliant on labor-intensive ground-truth affordance labels. The authors of this paper tackle a core challenge: How can we bridge the gap between affordance discovery and policy learning within a single, end-to-end deep neural network?

Fig. 1. An example of two mugs: Humans should have different affordance- effect judgments on their bodies or handles. At each step, the robot takes one depth image (observation) and generates one attention map, a.k.a affordance cue, that further triggers an action. We also show how the robot eventually picks up a mug, either by holding the body or handle, by showing the final step observation.

Methods

The authors propose an end-to-end contrastive imitation learning framework consisting of two main components: a Siamese Encoder and a Trajectory Decoder.

[Demonstration Trajectory] 
       │
       ├──► Interaction Segment ──► LSTM Encoder ──────┐
       │                                               ▼ [Coupled Triplet Loss]
       └──► Step-by-Step Obs    ──► Trajectory Encoder ┴► [Affordance Cue / Attention]
                                                               │
                                                               ▼
                                                       Trajectory Decoder ──► Predicted Action

1. Siamese Encoder & Interaction Segments

To learn clean representations of affordances, the network extracts the interaction segment of a trajectory—the specific window of state-action pairs where the gripper actually manipulates the object. An LSTM encodes this low-dimensional interaction data into an affordance embedding ($Z^A$).

Concurrently, a two-branch trajectory encoder processes high-dimensional step-by-step observations (depth images, current state, and previous action) to generate an observation embedding ($z^{o,A}_t$). Crucially, this branch outputs a latent, channel-wise attention map—the affordance cue.

2. Coupled Triplet Loss

Standard contrastive learning frameworks often fail here because the network might exploit affordance-irrelevant shortcuts (like background context or initial positions) to distinguish trajectories. To prevent this, the authors introduce a coupled triplet loss. This loss forces the high-dimensional observation embeddings to be guided by the spatial relationships of the cleaner, low-dimensional affordance embeddings.

3. Trajectory Decoder

The trajectory decoder acts as the policy network. It multiplies the discovered affordance cue (attention map) with the visual feature maps, concatenates the robot's current state and contrastive embedding, and passes them through convolutional and fully connected layers to output the next action. The entire architecture is trained jointly using a combination of the coupled triplet loss and a standard behavior cloning loss.

Datasets

The framework was evaluated in a simulated environment using the PyBullet simulator and a Franka Panda robot arm.

  • Objects: 24 distinct mug models featuring diverse geometric structures, sizes, and handle shapes.
  • Demonstrations: 27 trajectories (each 8 steps long) collected via human teleoperation. Trajectories were categorized into three affordable grasp types: body-grasp, handle-left-right-grasp, and handle-front-back-grasp.

Key Insights and Ideas

  • Attention as an Implicit Affordance Proxy: Instead of forcing a network to explicitly output bounding boxes or gripper angles, visual attention maps can naturally serve as a reliable latent feature for policy execution.
  • The Danger of Naive Contrastive Learning: One of the most surprising findings was that applying a standard triplet loss directly to raw visual observations actually degraded performance compared to not using contrastive learning at all. Without guidance, the network latched onto irrelevant environmental features, misguiding the policy.
  • Low-Dim Guiding High-Dim: By coupling the visual encoder's loss to the low-dimensional interaction segment encoder, the framework successfully anchors visual attention to the physical reality of the grasp.

Key Results

The models were evaluated on their ability to successfully grasp and lift randomly sampled mugs by 5 centimeters.

Model Grasping Success Rate
Ours (Full Model): Siamese + Coupled Triplet Loss 65%
Ours (Ablation): Siamese + Normal Triplet Loss 35%
Ours (Ablation): Without Contrastive Learning 45%
Baseline (Standard Behavior Cloning) 25%

The full model significantly outperformed the baseline and ablation variants. Qualitatively, the generated attention maps clearly highlighted the handles for handle-graspable mugs and the bodies for mugs that required a body grasp, validating that the network truly discovered meaningful affordance cues.

Limitations and Future Work

While promising, the authors note a few limitations that pave the way for future research:

  1. Single-Level Interactions: The current setup only handles direct interactions between a gripper and a single object. Complex tasks involving tool use (e.g., using a hook to pull a mug closer) would require multi-level interaction modeling.
  2. Fixed High-Level Tasks: The affordance cues discovered were tied to a single task ("picking-to-pour-water"). If the high-level goal changes to something like "pick-and-relocate", the ideal affordance cue changes. Future iterations could integrate this method into a hierarchical imitation learning framework to condition attention on varying high-level goals.

Summary updated

Abstract

Conventional works that learn grasping affordance from demonstrations need to explicitly predict grasping configurations, such as gripper approaching angles or grasping preshapes. Classic motion planners could then sample trajectories by using such predicted configurations. In this work, our goal is instead to fill the gap between affordance discovery and affordance-based policy learning by integrating the two objectives in an end-to-end imitation learning framework based on deep neural networks. From a psychological perspective, there is a close association between attention and affordance. Therefore, with an end-to-end neural network, we propose to learn affordance cues as visual attention that serves as a useful indicating signal of how a demonstrator accomplishes tasks, instead of explicitly modeling affordances. To achieve this, we propose a contrastive learning framework that consists of a Siamese encoder and a trajectory decoder. We further introduce a coupled triplet loss to encourage the discovered affordance cues to be more affordance-relevant. Our experimental results demonstrate that our model with the coupled triplet loss achieves the highest grasping success rate in a simulated robot environment. Our project website can be accessed at https://sites.google.com/asu.edu/affordance-aware-imitation/project.
BibTeX
@inproceedings{zha2021contrastively,
  author = {Yantian Zha and Siddhant Bhambri and Lin Guan},
  title = {{Contrastively Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Grasping}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
}
Figure