Abstract
Dexterous in-hand manipulation remains a significant challenge in robotics due to the mechanical complexity of high-degree-of-freedom (DoF) hands and the difficulty of collecting high-quality demonstration data. This paper presents Tilde, an end-to-end imitation learning system that utilizes a customized "DeltaHand" and a passive mechanical twin, the "TeleHand," for intuitive teleoperation. By leveraging a Diffusion Policy trained on 1:1 kinematic demonstrations, Tilde achieves a 90% success rate across diverse tasks, including rotational skills and high-force applications like syringe pushing.
1. Introduction
The ability to reposition or rotate objects within a robotic grip—dexterous manipulation—is often considered a "holy grail" in the field of robotics. While anthropomorphic hands offer high versatility, they are frequently expensive, complex to control, and difficult to train. A primary bottleneck is data collection: imitation learning requires high-fidelity human demonstrations, but teleoperating high-DoF hands via visual tracking (e.g., Leap Motion) often introduces noise and "correspondence issues" where the human hand's movements do not map perfectly to the robot's geometry, leading to suboptimal learning outcomes.
2. The Tilde Framework
Tilde addresses these challenges through a three-pronged approach integrating hardware innovation, a specialized teleoperation interface, and robust policy learning.
2.1 DeltaHand Hardware
The DeltaHand is a 12-DoF robotic hand composed of four 3-DoF Delta modules. To balance strength and safety, the researchers developed "hybrid" fingers. These fingers feature rigid PLA plastic cores embedded within soft TPU shells. This design allows the hand to exert significant force (up to 4.0N)—crucial for tasks like pushing a syringe—while maintaining the compliance necessary to adapt to object geometries passively.
2.2 TeleHand Interface
To solve the correspondence problem, Tilde utilizes the TeleHand, a passive, non-motorized "kinematic twin" of the DeltaHand. Because the TeleHand shares identical geometry with the robot, a human operator can physically manipulate the twin to provide demonstrations with a perfect 1:1 joint state mapping. This hardware-to-hardware approach reduces data collection time by 3x to 5x compared to traditional visual tracking methods and eliminates mapping errors and latency.
2.3 Learning with Diffusion Policies
The system employs a CNN-based Diffusion Policy to process demonstration data. Unlike standard behavior cloning, which can struggle with the multi-modal nature of manipulation (where multiple valid paths exist to reach a goal), diffusion models excel at capturing complex, non-linear action sequences. The framework also incorporates DAgger (Dataset Aggregation), allowing human operators to provide corrective demonstrations when the autonomous policy deviates from the desired path.
3. Experimental Evaluation
The system was evaluated on seven real-world tasks categorized by complexity:
- Basic Skills: Grasping, Block Sliding (XY translation), and Block Lifting (Z translation).
- Rotational Skills: Ball Rolling and Cap Twisting.
- Long-Horizon Tasks: Syringe Pushing and Shape Insertion (aligning a flower-shaped block into a template).
A critical component of the setup is an integrated Arducam located in the palm. This in-hand vision provides an unobstructed view of the object, which is essential for precision tasks like shape insertion where external cameras are often blocked by the robot's fingers.
4. Results and Discussion
Tilde demonstrated high performance and robustness across all test scenarios:
- Success Rate: The system achieved an average success rate of 90% across all seven tasks.
- Generalization: The "Block Slide" policy, though trained only on rectangular blocks, successfully generalized to irregular objects such as a screwdriver, a knife, and a glue stick.
- Force and Precision: The hybrid finger design enabled the robot to push a syringe with a rubber seal—a task requiring substantial force that pure soft hands cannot handle. In the complex "Shape Insert" task, the robot reached a 74% success rate with 100 demonstrations.
- Compliance: The soft TPU shells allowed for a 100% success rate in grasping tasks, as the fingers could conform to various shapes without requiring complex visual feedback.
5. Conclusion and Future Work
Tilde demonstrates that combining intuitive hardware-twin teleoperation with diffusion-based learning can effectively bridge the gap in dexterous manipulation. While successful, the system currently lacks tactile sensing, which would improve the handling of fragile objects. Future research will focus on integrating fingertip pressure sensors, mounting the hand on a mobile arm for extrinsic manipulation, and improving vision robustness against environmental lighting changes.
Summary updated