Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images (Extended Abstract)

Kuniyuki Takahashi, Jethro TanIJCAI 2020 · DOI

TL;DR

This paper introduces a novel deep learning approach to estimate tactile properties like softness, roughness, and friction directly from RGB images. By training an encoder-decoder network on visual and tactile sensor data, the model learns a continuous latent space that represents these properties, enabling generalization to unseen materials without explicit manual labeling.

What Problem Does This Paper Address?

Effectively interacting with the physical world, whether for humans or robots, relies heavily on understanding the tactile properties of objects – their softness, roughness, slipperiness, etc. While humans can infer these properties from visual cues, robots often struggle. Traditional approaches for robots typically involve discrete classification of materials or require extensive, manually labeled datasets. This paper addresses the challenge of enabling robots to infer a continuous range of tactile properties from visual information alone, bridging the gap between visual perception and tactile understanding in a more scalable and generalizable way.

Methods

The core of the proposed method is a "deep visuo-tactile learning" network. This architecture is based on an encoder-decoder model. The encoder takes an image of a material as input and processes it through convolutional layers. The decoder then reconstructs tactile sensor data (forces in x, y, and z axes) from a latent representation. Crucially, the latent space is designed to embed both visual and tactile features. The network is trained in an unsupervised manner, learning to correlate visual input with corresponding tactile sensor readings. After training, the network can infer tactile properties by extracting the latent variables from an image, effectively learning a continuous representation of tactile properties without relying on discrete, pre-defined classes.

Datasets

The authors collected their own dataset for training and evaluation. This dataset consists of RGB images and corresponding tactile sensor data from 25 different materials. 15 of these materials were used for training, and the remaining 10 were held out as "unknown materials" to test the model's generalization capabilities. The tactile data was collected using a uSkin tactile sensor mounted on a robotic arm that stroked the material surfaces.

Key Insights and Ideas

  • Continuous Representation is Key: Moving beyond discrete material classes allows for a more nuanced understanding of tactile properties, enabling finer-grained distinctions.
  • Unsupervised Learning for Scalability: The unsupervised approach significantly reduces the burden of manual labeling, making it easier to scale to a wider variety of materials.
  • Generalization to Unknowns: The model demonstrates the ability to infer tactile properties for materials it has not seen during training, a critical capability for real-world applications.
  • Latent Space as a Tactile Map: The learned latent space effectively acts as a continuous map where different tactile properties are organized, allowing for visual interpretation of these properties.

Key Results

The paper presents qualitative results visualizing the learned latent space. Figure 5 shows that latent variables can indeed correlate with tactile properties:

  • Latent Variable z1: Appears to capture softness and roughness. Materials perceived as hard and rough were mapped to lower values of z1, while soft and smooth materials mapped to higher values.
  • Latent Variable z2: Seems to represent friction. Materials with lower friction (like glossy plastics) mapped to lower z2 values, while fabric-like materials with higher friction mapped to higher values. The model also showed promising generalization, with an unknown black carpet clustering near its known brown carpet counterpart in the latent space.

Limitations and Future Work

The authors acknowledge that their current model has limitations in accurately estimating tactile properties from two-dimensional images when the tactile sensor's perception is insufficient (e.g., for a Japanese straw mat where tactile values did not match the perceived roughness). They also note that the latent variables are sensitive to contact differences (posture, speed, wear), which is why they focus on extracting latent variables rather than directly inferring time-series tactile data. Future work could explore more complex network architectures (like VAEs or RNNs) to potentially improve accuracy and the representation of tactile properties.

Summary updated

BibTeX
@inproceedings{takahashi2020deep,
  author = {Kuniyuki Takahashi and Jethro Tan},
  title = {{Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/665},
}
Figure