Hand Centric Vision for Robotic Manipulation

47 concepts31 papers1 proposal

When we watch a human reach for a cup, their eyes track both the cup and their hand from a distance. But as their fingers close around the handle, their brain relies heavily on a localized, egocentric frame of reference. In robotics, traditional manipulation systems have long favored a "God's eye view"—static, third-person cameras mounted on the ceiling or a nearby tripod. While this setup provides an excellent global overview of the workspace, it introduces severe geometric complexities, calibration overhead, and a critical vulnerability: the moment the robot arm moves to grasp an object, it blocks its own line of sight.

To bypass these limitations, a growing body of research advocates for hand-centric vision (often called eye-in-hand perception). By mounting cameras, depth sensors, or even tactile arrays directly onto the robot's wrist, palm, or fingertips, researchers fundamentally alter the geometry of the manipulation problem. Instead of tracking an object relative to the world and then translating that into the robot's base coordinates, hand-centric vision transforms the task into a local alignment problem.

The Geometry of Invariance

Why does moving the camera from a fixed tripod to a moving wrist make learning so much easier? The answer lies in spatial invariance.

In a third-person setup, if you change the height of the table or shift the background clutter, the visual scene changes entirely. The neural network must learn to generalize across all possible global positions. As Kyle Hsu et al. demonstrate in Vision-Based Manipulators Need to Also See from Their Hands, a hand-centric perspective provides a local viewpoint that is naturally invariant to these global environmental shifts. Because the camera moves with the gripper, the spatial relationship between the tool and the target remains constant in the image frame.

This geometric simplification acts as a practical shortcut to achieving SE(3) invariance. Dian Wang et al. show in A Practical Guide for Incorporating Symmetry in Diffusion Policy that combining a gripper-mounted camera with relative action outputs allows policies to inherently respect spatial symmetries without requiring complex, custom equivariant neural network architectures. Similarly, Albert Wilcox et al. leverage this in Learning to Localize, Grasp, and Hand Over Unmodified Surgical Needles by using zoomed-in, ego-centric image crops centered on the end-effector. This representational strategy strips away background noise and prevents the model from overfitting to global scene features, enabling high-precision tracking of reflective surgical tools.

This invariance is so powerful that it even allows a single policy to bridge the gap between entirely different robot designs. In Polybot: Training One Policy Across Robots While Embracing Variability, Jonathan Heewon Yang demonstrates that utilizing wrist-mounted cameras reduces cross-robot visual variance to just a few dimensions of height and angle. By focusing strictly on the local interaction between the end-effector and the object, the policy can transfer manipulation skills across completely different hardware platforms with minimal demonstrations.

Banishing the Blind Spot

The most acute failure mode of third-person vision occurs during the final centimeters of an approach—a phenomenon often called the "blind drive" problem. As a robotic arm closes in on a target, the bulk of the arm or the gripper itself inevitably occludes the camera's view of the object.

Third-Person Vision:
[Camera] ------> [Robot Arm] xxxxx> [Target Object] (Occluded!)

Hand-Centric Vision:
[Robot Arm] ---> [Camera/Gripper] ----> [Target Object] (Clear Sight!)

To maintain a clear line of sight during delicate contact tasks, Nicholas Ohi et al. integrate an endoscope camera directly into the end-effector in Design of an Autonomous Precision Pollination Robot, ensuring continuous visual servoing up to sub-centimeter contact. When the task demands complex, multi-fingered manipulation, this occlusion problem worsens because the fingers themselves block the view. To solve this, Felix Hundhausen et al. embed miniature cameras directly into the fingertips in A Soft Humanoid Hand with In-Finger Visual Perception, providing uninterrupted segmentation data right up to the moment of physical contact.

For dexterous in-hand manipulation, placing the sensor in the palm offers a remarkably stable perspective. As Zilin Si et al. show in Tilde: Teleoperation for Dexterous In-Hand Manipulation Learning with a DeltaHand, a palm-mounted camera provides a first-person view of finger-object interactions that resists external occlusions. This continuous visibility is critical for vision-conditioned control policies like diffusion models. In HannesImitation: Grasping with the Hannes Prosthetic Hand via Imitation Learning, Carlo Alessi et al. utilize a palm-embedded camera to provide high-resolution feedback during the final stages of an approach, allowing a prosthetic hand to autonomously manage complex wrist movements and finger closures without placing an excessive cognitive burden on the human user.

Closed-Loop Precision and Active Sensing

Beyond solving occlusion and invariance, hand-centric perception fundamentally alters the control loop by turning the robot's motion into an advantage for its sensors. When a camera is fixed to the wrist, every movement of the arm acts as an active adjustment of the sensor's viewpoint.

In Learning a visuomotor controller for real world robotic grasping using simulated depth images, Ulrich Viereck et al. demonstrate that a wrist-mounted depth sensor ensures the target object remains within the field of view throughout the entire approach trajectory. This allows a closed-loop visuomotor policy to dynamically correct for unexpected object displacements in real-time. Similarly, Kim Wheeler et al. show in Hekateros: A Desktop 5 Degree-of-Freedom Robot Arm for the Small-Scale Manipulation Robot Chess Challenge that an eye-in-hand camera can perform both global environment surveying and local alignment, providing the real-time feedback loop necessary to compensate for mechanical drift during the final approach.

This dynamic vantage point is also highly effective for minimizing parallax errors. As Mireia Ruiz Maymo et al. explain in Fast Orient: Lightweight Computer Vision for Wrist Control in Assistive Robotic Grasping, mounting the vision sensor directly on the end-effector ensures the target object stays centered in the coordinate frame, simplifying the computer vision algorithms required to calculate orientation and align the gripper.

Furthermore, the ability to actively reposition the sensor allows the robot to optimize its data gathering on the fly. Davide De Tommaso et al. exploit this in A Robotized Projective Interface for Human-Robot Learning Scenarios by mounting a projector-sensor pair on the end-effector, enabling the robot to actively move closer to a surface to maximize projection resolution and avoid shadows. In shared autonomy settings, Atharv Belsare et al. demonstrate in Toward Zero-Shot User Intent Recognition in Shared Autonomy that a wrist camera provides a continuously updating environment model that refines the robot's understanding of user intent as it draws nearer to potential targets.

Expanding the Egocentric Workspace

The benefits of hand-centric sensing extend naturally to mobile and multi-arm systems. In teleoperation, fixed cameras restrict the operator to a predefined workspace. Shuang Li breaks through this constraint in A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU by mounting depth cameras directly onto the human operator's forearm. This maintains a consistent, mobile field of view that prevents loss of control when the operator moves across large environments.

For high-precision bimanual manipulation, where two arms must coordinate in close proximity, static overhead views often fail to capture fine-grained spatial relationships. As Toru Lin et al. show in Learning Visuotactile Skills with Two Multifingered Hands, wrist-mounted cameras provide dynamic, close-up spatial cues that minimize inter-arm occlusions, allowing policies to maintain a clear line of sight during complex tasks like pouring or stacking.

Ultimately, hand-centric vision shifts the paradigm of robotic perception: instead of forcing a neural network to understand the entire world from a distance, it brings the sensor to the action, transforming complex global scenes into manageable, local geometric relationships.

Key Foundational Work

Hekateros: A Desktop 5 Degree-of-Freedom Robot Arm for the Small-Scale Manipulation Robot Chess Challenge (2011) — contributed 2 concept elements to this direction.
A Robotized Projective Interface for Human-Robot Learning Scenarios (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_eye_in_hand_perception_890ad887,
  title = {eye-in-hand perception},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/eye-in-hand-perception/rdr_890ad887fc7d4ae186128cb264a0c5cd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_eye_in_hand_perception_890ad887_v1,
  title = {eye-in-hand perception},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/eye-in-hand-perception/rdr_890ad887fc7d4ae186128cb264a0c5cd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake