TL;DR
The EyeSight Hand is a low-cost ($<2500), 7-DoF humanoid hand that combines robust, compliant quasi-direct drive actuation with dense, whole-hand tactile sensing. By integrating a new, simulation-friendly vision-based tactile sensor called GelSim(ple) and employing a "vision dropout" training strategy, the researchers achieved significant performance boosts in complex, contact-rich tasks like bottle opening and tool manipulation.
What Problem Does This Paper Address?
Replicating the human hand's dexterity remains a "holy grail" in robotics. Most existing designs force a compromise:
- Stiff Hands: High gear ratios make them precise but fragile and non-compliant, leading to breakage during contact-heavy learning.
- Soft Hands: Inherently compliant but difficult to control precisely and hard to equip with dense sensors.
- Sensing Gaps: Most tactile-enabled hands only have sensors on the fingertips, missing the "whole-hand" feedback (palm and finger segments) crucial for complex manipulation.
The EyeSight Hand aims to bridge these gaps by providing a platform that is robust enough for large-scale data collection, compliant enough to handle collisions, and sensitive enough to "feel" its environment across its entire surface.
Methods
The authors take a co-design approach, integrating novel actuation, sensing, and learning strategies:
- Hybrid Actuation: The hand uses a mix of actuators to balance strength and compliance. The main flexor joints (MCP) use Quasi-Direct Drive (QDD)—brushless DC motors with low-reduction planetary gears—allowing the hand to be backdriven by external forces. The distal joints (PIP/IP) use micro DC motors with worm gears for self-locking strength.
- GelSim(ple) Sensors: Traditional GelSight sensors use three-color directional lighting for 3D reconstruction, which is computationally expensive to simulate. The authors introduced GelSim(ple), which uses white perimeter lighting and a single-channel image. This design is easier to adapt to different finger shapes and significantly simpler to model for sim-to-real applications.
- Imitation Learning with Vision Dropout: The team used Action Chunking with Transformers (ACT) to train the hand via teleoperation. A key innovation is vision dropout: during training, the global and wrist camera inputs are randomly zeroed out (30% chance). This forces the neural network to rely on the tactile stream rather than using vision as a "crutch," which is vital when the object is occluded by the hand itself.
Datasets
The researchers collected a custom dataset for three challenging tasks:
- Bottle Opening: Requires applying downward force while precisely flicking a lid.
- Plate Pick and Place: Requires sliding a plate to a graspable position without overshooting.
- Plasticine Cutting: Requires forceful tool use and sensing when the cut is complete.
For each task, they collected 100 teleoperated demonstrations consisting of proprioception, global/wrist video, and 8 synchronized tactile image streams.
Key Insights and Ideas
- Compliance is Robustness: By using QDD actuators, the hand can survive the "bumps and grinds" of hundreds of training trials that would strip the gears of a traditional servo-driven hand.
- The "Vision Crutch" Problem: In multi-modal learning, models often ignore tactile data because vision provides a strong global signal. However, vision fails during the most critical moments of contact (occlusion). Vision dropout is a simple but effective fix to ensure the policy actually "listens" to the tactile sensors.
- Whole-Hand Sensing Matters: By placing sensors on the palm and every finger segment, the hand can perform non-prehensile tasks (like sliding a plate with the side of a finger) that fingertip-only hands cannot.
Key Results
The integration of tactile sensing, especially when combined with vision dropout (VTA), showed dramatic improvements over vision-only (VO) policies:
- Plate Pick and Place: Success jumped from 50% (VO) to 100% (VTA). Tactile feedback allowed the robot to sense exactly when it made contact to slide the plate.
- Plasticine Cutting: Success increased from 50% (VO) to 90% (VTA). The tactile sensors helped the robot avoid applying excessive force that would cause the cutting board to slide.
- Bottle Opening: This was the hardest task. Vision-only and standard vision-tactile policies failed completely (0% success). With vision dropout, the success rate rose to 30%, proving that tactile cues are the only way to navigate the high occlusion of this task.
Limitations and Future Work
While impressive, the EyeSight Hand makes some kinematic compromises. It lacks the distal interphalangeal (DIP) joints and some lateral degrees of freedom found in human hands. Furthermore, while 30% success on bottle opening is a breakthrough, it remains low for reliable deployment.
The authors plan to extend the design with more fingers and explore more sophisticated algorithms to better fuse the high-dimensional tactile data with visual and proprioceptive inputs.
Summary updated