Abstract
Tactile information is a critical tool for dexterous manipulation. As humans, we rely heavily on tactile information to understand objects in our environments and how to interact with them. We use touch not only to perform manipulation tasks but also to learn how to perform these tasks. Therefore, to create robotic agents that can learn to complete manipulation tasks at a human or super-human level of performance, we need to properly incorporate tactile information into both skill execution and skill learning. In this paper, we investigate how we can incorporate tactile information into imitation learning platforms to improve performance on manipulation tasks. We show that incorporating visuo-tactile pretraining improves imitation learning performance, not only for tactile agents (policies that use tactile information at inference), but also for non-tactile agents (policies that do not use tactile information at inference). For these non-tactile agents, pretraining with tactile information significantly improved performance (for example, improving the accuracy on USB plugging from 20% to 85%), reaching a level on par with visuo-tactile agents, and even surpassing them in some cases.
Introduction
Dexterous manipulation tasks, such as threading a needle or plugging in a USB cable, represent a significant challenge for modern robotics. While humans instinctively use tactile feedback to adjust grip and detect contact, most robotic systems rely primarily on vision. Integrating tactile sensors like the GelSight into control policies is difficult because these sensors are fragile and produce high-dimensional data that is hard to map directly to actions. Furthermore, the cost and maintenance of tactile sensors make it impractical to require them on every deployed robot. This research introduces VITaL (Visuo-Tactile Pretraining), asking whether tactile data can be used during the learning phase to help a robot "understand" contact, even if it only uses cameras during deployment.
Methods
The VITaL framework utilizes a two-stage approach to bridge the gap between vision and touch:
- Visuo-Tactile Contrastive Pretraining: Using a CLIP-inspired architecture, a vision encoder and a tactile encoder are trained to project data into a shared latent space. A contrastive loss forces the model to learn that specific visual scenes (e.g., a gripper making contact) should have similar mathematical representations to corresponding tactile strain maps. This process primes the vision encoder to recognize features related to contact and force.
- Imitation Learning (IL) Integration: These pretrained encoders are integrated into state-of-the-art IL frameworks, specifically Action Chunking Transformers (ACT) and Diffusion Policy. For "vision-only" agents, the tactile encoder is discarded after pretraining, leaving a vision encoder that has implicitly learned to "see" tactile interactions.
Experimental Setup and Datasets
The system was evaluated using a Franka Emika Panda robot equipped with a GelSight Mini sensor. To ensure sensor longevity, a custom 1mm silicon rubber protective topper was developed, extending the sensor's life significantly. Data was collected via VR teleoperation across three distinct tasks:
- USB Plugging: 100 demonstrations of retrieving a USB cable and plugging it into a hub with 2.5mm noise injection to increase difficulty.
- Rectangle Stacking: 100 demonstrations requiring precise force control to stack thin blocks without toppling them.
- Cube Stacking: 100 demonstrations of a simpler stacking task to test performance on less tactile-intensive movements.
Key Results
The results demonstrate that tactile data serves as a powerful supervisor for vision. The most striking improvement was seen in the high-precision USB plugging task:
- Vision-Only Success: Using the ACT framework, the success rate for a vision-only agent rose from 20% without pretraining to 85% with VITaL.
- Diffusion Policy: Success rates improved from 45% to 75%.
- Force Reduction: Pretraining led to more "gentle" manipulation. Agents trained with VITaL exerted up to 20% less median strain on objects, suggesting a more contact-aware control strategy that mimics human-like delicacy.
Conclusion
VITaL proves that robots can benefit from the "sense of touch" during training even if they are restricted to vision during deployment. By aligning visual and tactile information into a shared latent space, the vision encoder learns to perceive the physics of interaction. Future work will focus on large-scale visuo-tactile pretraining to develop "Foundation Models for Touch" that can generalize across diverse objects and tasks without task-specific training.
Summary updated