VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policies

Abraham George, Selam Gano, Pranav Katragadda, Amir Barati Farimani

Abstract

Tactile information is a critical tool for dexterous manipulation. As humans, we rely heavily on tactile information to understand objects in our environments and how to interact with them. We use touch not only to perform manipulation tasks but also to learn how to perform these tasks. Therefore, to create robotic agents that can learn to complete manipulation tasks at a human or super-human level of performance, we need to properly incorporate tactile information into both skill execution and skill learning. In this paper, we investigate how we can incorporate tactile information into imitation learning platforms to improve performance on manipulation tasks. We show that incorporating visuo-tactile pretraining improves imitation learning performance, not only for tactile agents (policies that use tactile information at inference), but also for non-tactile agents (policies that do not use tactile information at inference). For these non-tactile agents, pretraining with tactile information significantly improved performance (for example, improving the accuracy on USB plugging from 20% to 85%), reaching a level on par with visuo-tactile agents, and even surpassing them in some cases. For demonstration videos and access to our codebase, see the project website: https://sites.google.com/andrew.cmu.edu/visuo-tactile-pretraining

Introduction

Dexterous manipulation remains a "grand challenge" in robotics. While humans rely instinctively on touch to modulate grip and detect subtle object properties, integrating tactile sensors into robotic pipelines is difficult. Tactile sensors like GelSight are high-dimensional, prone to wear and tear, and often difficult to calibrate across different units. The authors address a critical question: How can we leverage the rich information provided by tactile sensors during training without being tethered to their hardware limitations during deployment? They aim to improve imitation learning (IL) frameworks—specifically Action Chunking Transformers (ACT) and Diffusion Policy—by imbuing them with a "tactile-aware" visual understanding.

Methodology

The VITaL (Vision-only Imitation using Tactile Latent) approach utilizes a two-stage training pipeline: visuo-tactile contrastive pretraining followed by imitation learning.

Overview of the VITaL two-stage training pipeline, illustrating the transition from multimodal pretraining to vision-only or visuo-tactile imitation learning.

Visuo-Tactile Contrastive Pretraining

In the first stage, a multimodal dataset comprising vision, tactile (GelSight strain maps), and robot position data is used to train vision and tactile encoders. These encoders project observations into a shared latent space. Using a CLIP-inspired contrastive loss, the model is trained to maximize the similarity between embeddings from the same timestep while minimizing similarity between different timesteps. This process forces the vision encoder to "attend" to visual features that correlate with tactile events, such as contact or stable grasping.

Detailed mechanism of the contrastive pretraining, showing the alignment of vision (V) and tactile (T) embeddings in a similarity matrix.

Imitation Learning (IL)

The pretrained encoders are integrated into ACT or Diffusion Policy backbones. The study compares two policy types:

  • Visuo-Tactile Policies: Utilize both vision and tactile encoders during inference.
  • VITaL (Vision-only): Discards the tactile encoder, using only the "tactile-aware" vision encoder. This allows the robot to benefit from implicit tactile knowledge while operating solely with camera inputs.

Experimental Setup and Results

The method was evaluated on three real-world tasks using a Franka Emika Panda robot: USB Plugging, Rectangle Stacking, and Cube Stacking. Data was collected via VR teleoperation, with approximately 100 demonstrations per task.

Key Findings

  • Performance Boost: In the high-precision USB Plugging task, a standard vision-only ACT policy achieved only 20% success. With VITaL pretraining, this jumped to 85%. Similarly, Diffusion Policy performance rose from 45% to 75%.
  • Robustness and Generalization: Pretraining mitigated overfitting in the Cube Stacking task, improving the visuo-tactile agent's success rate from 30% to 95%.
  • Contact-Aware Control: VITaL-trained agents exhibited a more "gentle" strategy, showing an 8-20% reduction in tangential strain compared to non-pretrained baselines.

Conclusion

VITaL pretraining demonstrates that tactile data can significantly enhance vision-only manipulation policies by aligning visual and tactile latent spaces. While currently task-specific, future work aims to scale this to generalizable foundation-style visuo-tactile models. This approach effectively bridges the gap between the richness of tactile sensing and the practical advantages of vision-only deployment.

Summary updated

Abstract

Tactile information is a critical tool for dexterous manipulation. As humans, we rely heavily on tactile information to understand objects in our environments and how to interact with them. We use touch not only to perform manipulation tasks but also to learn how to perform these tasks. Therefore, to create robotic agents that can learn to complete manipulation tasks at a human or super-human level of performance, we need to properly incorporate tactile information into both skill execution and skill learning. In this paper, we investigate how we can incorporate tactile information into imitation learning platforms to improve performance on manipulation tasks. We show that incorporating visuo-tactile pretraining improves imitation learning performance, not only for tactile agents (policies that use tactile information at inference), but also for non-tactile agents (policies that do not use tactile information at inference). For these non-tactile agents, pretraining with tactile information significantly improved performance (for example, improving the accuracy on USB plugging from 20% to 85%), reaching a level on par with visuo-tactile agents, and even surpassing them in some cases. For demonstration videos and access to our codebase, see the project website: https://sites.google.com/andrew.cmu.edu/visuo-tactile-pretraining
BibTeX
@inproceedings{george2025vital,
  author = {Abraham George and Selam Gano and Pranav Katragadda and Amir Barati Farimani},
  title = {{VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policies}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2025},
  doi = {10.1109/icra55743.2025.11128336},
}
Figure