EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation

Pulkit Agrawal, Branden Romero, Haoshu Fang, Pulkit Agrawal 0001, Edward H. Adelson

TL;DR

This paper introduces the EyeSight Hand, a novel 7-DoF robotic hand designed for dexterous manipulation, featuring integrated vision-based tactile sensors and compliant actuation. The hand's design and tactile sensing capabilities significantly improve imitation learning performance on challenging tasks, demonstrating the critical role of rich tactile feedback for robust robotic manipulation.

What Problem Does This Paper Address?

Replicating the dexterity of the human hand in robots is a long-standing challenge. Human hands possess a complex kinematic structure, compliant actuation, and a dense network of tactile sensors that provide rich feedback on object properties and contact forces. Existing robotic hands often compromise on one or more of these aspects. Fully actuated hands can be rigid and prone to damage, while compliant soft hands may lack full actuation or the ability to integrate dense tactile sensing. Furthermore, hands with dense tactile sensors often have limited degrees of freedom or don't closely mimic human morphology, hindering intuitive teleoperation and data collection for machine learning. This paper aims to bridge this gap by presenting a robotic hand that balances these crucial features for enhanced manipulation capabilities.

Methods

The EyeSight Hand is designed with three core principles:

  • Fully-Actuated Dexterous Kinematics: The hand features 7 degrees of freedom, closely mimicking human hand morphology to facilitate intuitive teleoperation and leverage human demonstration data. While some compromises are made (e.g., missing DIP joints on some fingers), the overall structure is designed for dexterity.
  • Compliant Actuation Scheme: The hand utilizes a "quasi-direct drive" actuation scheme. This involves a combination of micro DC motors with worm gear reduction for some joints and BLDC motors with planetary gearboxes for others. This design aims to provide human-level strength and speed while maintaining compliance and robustness against external forces, crucial for large-scale data collection.
  • Integrated Vision-Based Tactile Sensing: A novel vision-based tactile sensor, dubbed "GelSim(ple)," is developed and integrated into the hand. This sensor uses a unique illumination strategy that provides non-directional overhead lighting to a semi-specular sensing surface, enabling geometric information capture without casted shadows. This simplifies simulation and allows for single-channel image transmission, increasing data throughput. Eight such sensors are embedded across the hand, including fingertips and the palm.

For learning manipulation skills, the paper employs an Action Chunking with Transformers (ACT) model. This generative model uses a conditional variational autoencoder (CVAE) to predict future action trajectories. The input to the policy includes visual data (from global and wrist cameras), tactile data (processed from the GelSim(ple) sensors), and robot joint positions. A key innovation is the vision dropout strategy introduced during training, which randomly masks visual inputs to encourage the network to rely more heavily on tactile feedback.

Datasets

The paper does not introduce new datasets but rather focuses on collecting data for three specific tasks using the EyeSight Hand:

  • Bottle Opening: Ketchup bottles are randomly placed with slight orientation perturbations.
  • Plate Pick and Place: Stacked plates are placed on a slightly perturbed foam block.
  • Plasticine Cutting: A knife is retrieved from a box, and a plasticine roll is cut.

Data for these tasks is collected via teleoperation, with 100 demonstrations recorded for each task.

Key Insights and Ideas

  • Tactile Sensing is Crucial for Dexterity: The results strongly suggest that rich tactile feedback is not just beneficial but critical for achieving high success rates in complex manipulation tasks, especially those involving subtle contact forces or visual ambiguity.
  • Vision Dropout Enhances Tactile Reliance: The vision dropout strategy proved effective in forcing the learning model to leverage tactile information, leading to significant performance improvements, particularly in tasks where visual cues are limited or ambiguous (like bottle opening).
  • GelSim(ple) Sensor Design is Effective: The novel GelSim(ple) sensor design, with its simplified illumination and simulation capabilities, successfully integrates dense tactile sensing into a dexterous robotic hand, demonstrating its practical utility.
  • Compliant Actuation Enables Robust Data Collection: The quasi-direct drive actuation scheme allows for robust teleoperation and data collection without damaging the hardware, which is a significant advantage for training machine learning models.

Key Results

The paper demonstrates significant improvements in task success rates by incorporating tactile sensing and the vision dropout strategy:

  • Plate Pick and Place: The vision-only policy achieved a 50% success rate, while the vision-tactile policy reached 100%.
  • Plasticine Cutting: The vision-only policy achieved 50% success, vision-tactile improved it to 70%, and the vision dropout strategy boosted it to 90%.
  • Bottle Opening: Both vision-only and vanilla vision-tactile policies failed all trials. The vision dropout strategy achieved a 30% success rate, highlighting the task's reliance on tactile cues.

Limitations and Future Work

  • Limited Degrees of Freedom: While the hand is 7-DoF, it still has some kinematic differences from a human hand, which might limit certain highly nuanced manipulations.
  • Task Complexity: The current evaluation is limited to three specific tasks. Future work could explore a wider range of manipulation challenges.
  • Algorithm Exploration: The paper focuses on imitation learning with ACT. Future work could explore other learning paradigms, such as reinforcement learning, to further leverage the hand's capabilities.
  • Extending to More Fingers: The authors express interest in extending the hand to include additional fingers and exploring algorithms that can better utilize the rich tactile sensing capabilities.

Summary updated

Abstract

In this work, we introduce the EyeSight Hand, a novel 7 degrees of freedom (DoF) humanoid hand featuring integrated vision-based tactile sensors tailored for enhanced whole-hand manipulation. Additionally, we introduce an actuation scheme centered around quasi-direct drive actuation to achieve human-like strength and speed while ensuring robustness for large-scale data collection. We evaluate the EyeSight Hand on three challenging tasks: bottle opening, plasticine cutting, and plate pick and place, which require a blend of complex manipulation, tool use, and precise force application. Imitation learning models trained on these tasks, with a novel vision dropout strategy, showcase the benefits of tactile feedback in enhancing task success rates. Our results reveal that the integration of tactile sensing dramatically improves task performance, underscoring the critical role of tactile information in dexterous manipulation.
BibTeX
@inproceedings{agrawal2024eyesight,
  author = {Pulkit Agrawal and Branden Romero and Haoshu Fang and Pulkit Agrawal and Edward H. Adelson},
  title = {{EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802778},
}
Figure