Introduction
Imitation learning (IL) has emerged as a powerful paradigm for teaching robots complex skills by mimicking expert demonstrations. However, standard IL models often suffer from extreme brittleness, failing to generalize when objects are moved to unseen locations. This failure typically stems from the model overfitting to absolute spatial coordinates in the training data rather than learning the underlying task logic.
Humans overcome this limitation through hand-eye coordination. When performing a task, such as reaching for a teapot, we fixate our gaze on a specific part of the object and guide our hand relative to that focal point. This mechanism provides a natural spatial invariance. This paper introduces Hand-eye Action Networks (HAN), a novel architecture designed to learn this coordination directly from standard robot demonstrations without requiring explicit gaze or bounding-box supervision.
The HAN Architecture
The HAN framework replaces the traditional direct mapping from images to joint torques with a three-stage differentiable pipeline that enforces a spatially invariant action space.
1. 3D Visual Attention Network
Instead of processing the entire visual field as a monolithic input, this module processes random 2D image crops. It utilizes a spatial softmax layer to extract 2D keypoints, which are then projected into 3D space using depth information. This process generates a pool of candidate 3D keypoints representing potential points of interest in the scene.
2. Attention Switching Network
For multi-stage tasks, the robot must shift its focus between different objects (e.g., from a tool to a target). This network assigns a confidence score to each candidate keypoint and computes a weighted average, resulting in a single active 3D target location for the current timestep.
3. Action Target Network
To prevent the model from ignoring the attention mechanism, the final action output is constrained. The network predicts a local 3D offset and a scalar gain relative to the selected 3D keypoint and the current end-effector position. This architectural bottleneck forces the policy to rely on the attention mechanism to reach the target area.
Experimental Evaluation
The authors evaluated HAN in MuJoCo/robosuite environments across three tasks: Lifting, Stacking, and a long-horizon Tool-using task. Models were trained on demonstrations within a specific interpolation region and tested on their ability to generalize to an unseen extrapolation region.
Key Results
HAN demonstrated superior zero-shot generalization compared to standard behavioral cloning (BC) baselines:
- Lifting: HAN achieved a 0.73 success rate in extrapolation, significantly higher than the baseline's 0.13.
- Stacking: HAN reached an 0.87 success rate, while the baseline remained at 0.13.
- Tool-using: In this complex task, HAN achieved a 0.60 success rate, whereas baselines (including those with ground-truth state access) largely failed (0.00 - 0.07).
- Robustness: When trained on noisy human teleoperation data, HAN maintained a 0.57 success rate in stacking, while baselines collapsed to 0.1.
Discussion and Conclusion
The success of HAN highlights the power of spatial invariance as a structural prior. By forcing the policy to act relative to dynamically attended visual keypoints, the model naturally ignores absolute coordinates. Furthermore, interpretable attention maps emerge purely from action trajectories without explicit supervision. While currently limited to 3D translation, extending HAN to 6-DoF control and integrating it into Reinforcement Learning frameworks represent promising avenues for future research.
Summary updated