Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control

Chen Wang, Rui Wang, Danfei Xu, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese

Introduction

Imitation learning (IL) has emerged as a powerful paradigm for teaching robots complex skills by mimicking expert demonstrations. However, standard IL models often suffer from extreme brittleness, failing to generalize when objects are moved to unseen locations. This failure typically stems from the model overfitting to absolute spatial coordinates in the training data rather than learning the underlying task logic.

Humans overcome this limitation through hand-eye coordination. When performing a task, such as reaching for a teapot, we fixate our gaze on a specific part of the object and guide our hand relative to that focal point. This mechanism provides a natural spatial invariance. This paper introduces Hand-eye Action Networks (HAN), a novel architecture designed to learn this coordination directly from standard robot demonstrations without requiring explicit gaze or bounding-box supervision.

Conceptual comparison between human visual attention and the proposed robot predicted visual attention mechanism.

The HAN Architecture

The HAN framework replaces the traditional direct mapping from images to joint torques with a three-stage differentiable pipeline that enforces a spatially invariant action space.

The three-stage architecture of the Hand-eye Action Network (HAN), comprising visual attention, attention switching, and action target networks.

1. 3D Visual Attention Network

Instead of processing the entire visual field as a monolithic input, this module processes random 2D image crops. It utilizes a spatial softmax layer to extract 2D keypoints, which are then projected into 3D space using depth information. This process generates a pool of candidate 3D keypoints representing potential points of interest in the scene.

2. Attention Switching Network

For multi-stage tasks, the robot must shift its focus between different objects (e.g., from a tool to a target). This network assigns a confidence score to each candidate keypoint and computes a weighted average, resulting in a single active 3D target location for the current timestep.

3. Action Target Network

To prevent the model from ignoring the attention mechanism, the final action output is constrained. The network predicts a local 3D offset and a scalar gain relative to the selected 3D keypoint and the current end-effector position. This architectural bottleneck forces the policy to rely on the attention mechanism to reach the target area.

Experimental Evaluation

The authors evaluated HAN in MuJoCo/robosuite environments across three tasks: Lifting, Stacking, and a long-horizon Tool-using task. Models were trained on demonstrations within a specific interpolation region and tested on their ability to generalize to an unseen extrapolation region.

Key Results

HAN demonstrated superior zero-shot generalization compared to standard behavioral cloning (BC) baselines:

  • Lifting: HAN achieved a 0.73 success rate in extrapolation, significantly higher than the baseline's 0.13.
  • Stacking: HAN reached an 0.87 success rate, while the baseline remained at 0.13.
  • Tool-using: In this complex task, HAN achieved a 0.60 success rate, whereas baselines (including those with ground-truth state access) largely failed (0.00 - 0.07).
  • Robustness: When trained on noisy human teleoperation data, HAN maintained a 0.57 success rate in stacking, while baselines collapsed to 0.1.

Discussion and Conclusion

The success of HAN highlights the power of spatial invariance as a structural prior. By forcing the policy to act relative to dynamically attended visual keypoints, the model naturally ignores absolute coordinates. Furthermore, interpretable attention maps emerge purely from action trajectories without explicit supervision. While currently limited to 3D translation, extending HAN to 6-DoF control and integrating it into Reinforcement Learning frameworks represent promising avenues for future research.

Summary updated

Abstract

Imitation Learning (IL) is an effective framework to learn visuomotor skills from offline demonstration data. However, IL methods often fail to generalize to new scene configurations not covered by training data. On the other hand, humans can manipulate objects in varying conditions. Key to such capability is hand-eye coordination, a cognitive ability that enables humans to adaptively direct their movements at task-relevant objects and be invariant to the objects' absolute spatial location. In this work, we present a learnable action space, Hand-eye Action Networks (HAN), that can approximate human's hand-eye coordination behaviors by learning from human teleoperated demonstrations. Through a set of challenging multi-stage manipulation tasks, we show that a visuomotor policy equipped with HAN is able to inherit the key spatial invariance property of hand-eye coordination and achieve zero-shot generalization to new scene configurations. Additional materials available at https://sites.google.com/stanford.edu/han
BibTeX
@inproceedings{wang2021generalization,
  author = {Chen Wang and Rui Wang and Danfei Xu and Ajay Mandlekar and Li Fei-Fei and Silvio Savarese},
  title = {{Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636023},
}
Figure