Abstract
Tactile information is a critical tool for dexterous manipulation. As humans, we rely heavily on tactile information to understand objects in our environments and how to interact with them. We use touch not only to perform manipulation tasks but also to learn how to perform these tasks. Therefore, to create robotic agents that can learn to complete manipulation tasks at a human or super-human level of performance, we need to properly incorporate tactile information into both skill execution and skill learning. In this paper, we investigate how we can incorporate tactile information into imitation learning platforms to improve performance on manipulation tasks. We show that incorporating visuo-tactile pretraining improves imitation learning performance, not only for tactile agents (policies that use tactile information at inference), but also for non-tactile agents (policies that do not use tactile information at inference). For these non-tactile agents, pretraining with tactile information significantly improved performance (for example, improving the accuracy on USB plugging from 20% to 85%), reaching a level on par with visuo-tactile agents, and even surpassing them in some cases. For demonstration videos and access to our codebase, see the project website: https://sites.google.com/andrew.cmu.edu/visuo-tactile-pretraining
Introduction
Dexterous manipulation remains a "grand challenge" in robotics. While humans rely instinctively on touch to modulate grip and detect subtle object properties, integrating tactile sensors into robotic pipelines is difficult. Tactile sensors like GelSight are high-dimensional, prone to wear and tear, and often difficult to calibrate across different units. The authors address a critical question: How can we leverage the rich information provided by tactile sensors during training without being tethered to their hardware limitations during deployment? They aim to improve imitation learning (IL) frameworks—specifically Action Chunking Transformers (ACT) and Diffusion Policy—by imbuing them with a "tactile-aware" visual understanding.
Methodology
The VITaL (Vision-only Imitation using Tactile Latent) approach utilizes a two-stage training pipeline: visuo-tactile contrastive pretraining followed by imitation learning.
Visuo-Tactile Contrastive Pretraining
In the first stage, a multimodal dataset comprising vision, tactile (GelSight strain maps), and robot position data is used to train vision and tactile encoders. These encoders project observations into a shared latent space. Using a CLIP-inspired contrastive loss, the model is trained to maximize the similarity between embeddings from the same timestep while minimizing similarity between different timesteps. This process forces the vision encoder to "attend" to visual features that correlate with tactile events, such as contact or stable grasping.
Imitation Learning (IL)
The pretrained encoders are integrated into ACT or Diffusion Policy backbones. The study compares two policy types:
- Visuo-Tactile Policies: Utilize both vision and tactile encoders during inference.
- VITaL (Vision-only): Discards the tactile encoder, using only the "tactile-aware" vision encoder. This allows the robot to benefit from implicit tactile knowledge while operating solely with camera inputs.
Experimental Setup and Results
The method was evaluated on three real-world tasks using a Franka Emika Panda robot: USB Plugging, Rectangle Stacking, and Cube Stacking. Data was collected via VR teleoperation, with approximately 100 demonstrations per task.
Key Findings
- Performance Boost: In the high-precision USB Plugging task, a standard vision-only ACT policy achieved only 20% success. With VITaL pretraining, this jumped to 85%. Similarly, Diffusion Policy performance rose from 45% to 75%.
- Robustness and Generalization: Pretraining mitigated overfitting in the Cube Stacking task, improving the visuo-tactile agent's success rate from 30% to 95%.
- Contact-Aware Control: VITaL-trained agents exhibited a more "gentle" strategy, showing an 8-20% reduction in tangential strain compared to non-pretrained baselines.
Conclusion
VITaL pretraining demonstrates that tactile data can significantly enhance vision-only manipulation policies by aligning visual and tactile latent spaces. While currently task-specific, future work aims to scale this to generalizable foundation-style visuo-tactile models. This approach effectively bridges the gap between the richness of tactile sensing and the practical advantages of vision-only deployment.
Summary updated