Connecting Touch and Vision via Cross-Modal Prediction

Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, Antonio Torralba

Introduction

Humans possess an intuitive link between sight and touch; we can look at a piece of silk and "feel" its smoothness, or reach into a dark bag and identify a key by touch alone. For robots, replicating this cross-modal connection is a significant challenge due to the massive scale discrepancy between wide-angle vision and localized tactile sensing. While a standard camera perceives an entire scene, a high-resolution tactile sensor, such as the GelSight sensor, only captures a tiny 1.5cm x 1.5cm patch.

Overview of the cross-modal prediction tasks, including setup and sample results for vision-to-touch and touch-to-vision synthesis.

Researchers at MIT have developed a system that allows robots to bridge this gap, synthesizing tactile signals from visual input and "imagining" visual interactions from tactile data. This capability requires the model to understand not just geometry, but also material properties like texture, roughness, and hardness.

Methodology

The problem is framed as an image-to-image translation task using Conditional Generative Adversarial Networks (cGANs), specifically building on the pix2pix framework. To handle the scale and modality differences, the authors introduced three key innovations:

  1. Reference Images: Instead of generating scenes from scratch, the model is provided with "reference" images (a visual image of the scene before contact and a tactile image of the sensor touching nothing). This allows the network to focus on the changes caused by the interaction, such as membrane deformation.
  2. Temporal Cues: Touch is inherently dynamic. The model processes a sequence of five consecutive frames rather than a single snapshot, helping the system maintain temporal coherence and accurately predict the "moment of contact."
  3. Data Rebalancing: To prevent "mode collapse"—where the GAN only produces flat, featureless outputs—the authors used a rarity score to ensure the model trained on a diverse set of interactions (e.g., edges and textures) rather than just flat surfaces.
The specialized conditional GAN architecture featuring dual ResNet encoders for reference images and vision sequences, feeding into a common decoder.

The VisGel Dataset

To train and evaluate this system, the authors introduced the VisGel Dataset, the largest synchronized vision-touch dataset to date. It includes:

  • Scale: 12,000 unique touch actions resulting in 3 million paired vision and tactile frames.
  • Diversity: 195 everyday objects, including the standard YCB robotic set and various consumer goods like fabrics and tools.
  • Hardware: Data was collected using a KUKA LBR iiwa robot arm equipped with a GelSight sensor and an external webcam.

Key Results and Insights

The model demonstrated a remarkable ability to learn underlying physical representations. Internal units of the network were found to specialize in specific features, such as "sharp edges" or "flat surfaces," relevant to both modalities.

Evaluation via human perceptual studies (Amazon Mechanical Turk) showed:

  • Vision → Touch: Human evaluators identified synthesized tactile signals as "real" 46.6% of the time, nearly doubling the performance of the standard pix2pix baseline (28.1%).
  • Touch → Vision: The model's predicted touch locations shared the same "feel" as the ground truth 89.2% of the time for seen objects and 83.4% for unseen objects. Notably, the model learned to accurately localize touch without explicit coordinate supervision.
  • Temporal Accuracy: The inclusion of temporal cues significantly reduced errors in predicting the exact moment the sensor contacted an object.

Conclusion and Future Work

While the system successfully bridges vision and touch, limitations remain. The model occasionally struggles with highly diverse textures due to data distribution biases, and the current experiments are limited to static environments. Future research may explore more complex, dynamic scenes with moving cameras or multi-fingered robotic hands. Ultimately, these cross-modal representations could enable higher-level tasks, such as robotic grasping in low-light conditions or identifying objects purely through tactile probing.

Summary updated

BibTeX
@inproceedings{li2019connecting,
  author = {Yunzhu Li and Jun-Yan Zhu and Russ Tedrake and Antonio Torralba},
  title = {{Connecting Touch and Vision via Cross-Modal Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.01086},
}
Figure