Abstract
Recent advancements in robotic manipulation have highlighted the importance of tactile sensing for tasks involving occlusions or fine-grained contact. This paper introduces a reinforcement learning (RL) framework that enables robots to manipulate previously unseen objects using high-resolution tactile sensors. By training in a physics-accurate simulation with diverse object geometries and utilizing a robust "binary" tactile representation, the proposed method achieves successful zero-shot Sim2Real transfer. Experimental results demonstrate an 80% success rate on real-world pivoting tasks with unknown objects, significantly outperforming vision-based and proprioception-only baselines.
Introduction
While computer vision has significantly improved robot perception, "blind" manipulation remains a formidable challenge. Tasks such as re-orienting an object held in a gripper—where the object is often occluded by the robot's own hand—require precise contact information that vision alone cannot provide. Tactile sensors, such as the DIGIT sensor, offer high-resolution contact data, yet they introduce two primary hurdles: generalization across infinite object shapes and the "Sim2Real gap," where simulated tactile images differ substantially from real-world data due to lighting and sensor noise.
This research addresses these challenges by focusing on the pivoting task: re-orienting an object to a specific target angle using only tactile feedback and internal joint sensors.
Methodology
The authors propose an end-to-end RL pipeline based on Proximal Policy Optimization (PPO). To facilitate efficient training, they developed a fast tactile simulation environment by extending the SAPIEN simulator and TACTO engine with PyTorch3D-based GPU tensor rendering.
Tactile Representations
A core contribution of this work is the investigation of tactile data representations to bridge the Sim2Real gap. Three formats were evaluated:
- RGB: Raw color images from the sensor.
- Diff: The difference between current and reference "no-contact" images.
- Binary: A thresholded version of the "Diff" image, indicating only the contact area.
Surprisingly, the Binary representation proved most effective for transfer. By stripping away domain-specific noise like lighting and texture, it allows the policy to focus on the essential geometric features of the contact.
Experimental Setup and Datasets
To ensure the policy generalizes to novel shapes, the model was trained on a massive variety of 3D objects from the PartNet and Breaking Bad datasets. The training set included 22 simulated objects, while evaluation was conducted on 16 entirely unseen real-world objects, including screwdrivers, pens, and kitchen utensils. Domain randomization was applied to environment factors such as surface height, object length, and initial poses to further enhance robustness.
Results and Discussion
The proposed tactile-binary policy achieved an 80% success rate in real-world pivoting tasks. Key findings include:
- Tactile vs. Proprioception: Without tactile feedback, the success rate dropped to 33%, underscoring the necessity of touch for precise manipulation.
- Tactile vs. Vision: The tactile-based policy outperformed point-cloud-based vision policies, which often struggle with fine-grained contact physics and occlusions during in-hand tasks.
- Generalization: The policy successfully transferred across different table surfaces (hard vs. soft) and handled a wide variety of object geometries without retraining.
Conclusion and Future Work
This work demonstrates that simplified tactile representations and diverse simulation training are key to robust Sim2Real transfer for unknown object manipulation. Future research will aim to expand the action space from 2D planes to full 6-DOF manipulation and address more complex dynamic tasks such as in-hand shuffling or re-grasping.
Summary updated