UniT: Data Efficient Tactile Representation With Generalization to Unseen Objects

Zhengtong Xu, Raghava Uppuluri, Xinwei Zhang, Cael Fitch, Philip G. Crandall, Wan Shou, Dongyi Wang, Yu She

TL;DR

This paper introduces UniT, a novel method for learning tactile representations from GelSight sensors using a VQGAN autoencoder. UniT demonstrates remarkable data efficiency, learning generalizable representations from a single, simple object that can be zero-shot transferred to various downstream perception and manipulation tasks, outperforming existing methods.

What Problem Does This Paper Address?

Robotic manipulation often relies heavily on visual information, but this can overlook crucial details about object interactions and dynamics that tactile sensing can provide. While tactile sensing offers rich information, learning effective tactile representations that generalize across different objects and tasks has been a significant challenge. Existing methods often require extensive data or are task-specific, limiting their practical application. This paper aims to bridge this gap by developing a data-efficient tactile representation learning method that can generalize to unseen objects and tasks.

Methods

The core of UniT is a VQGAN autoencoder, adapted for tactile image representation learning. The VQGAN is trained in a self-supervised manner to reconstruct tactile images. The encoder maps tactile images into a compact latent space, which then serves as the tactile representation. This latent space is designed to be structured and low-variance, facilitating generalization. The authors highlight that UniT can be trained on data from a single, simple object (like an Allen key or a small ball) and still achieve good performance on unseen objects. For downstream tasks, the UniT encoder is frozen, and a task-specific decoder is trained.

Datasets

The paper uses the GelSight sensor for data collection. For training the UniT autoencoder, they collected datasets of:

  • An Allen key (10,854 images)
  • A small ball (4,831 images)

These datasets were collected at 10 Hz, requiring less than 20 minutes of data acquisition per object.

For evaluation, they used:

  • Perception Tasks:
    • USB plug 3D and 6D pose estimation (using a dataset collected with OptiTrack motion capture).
  • YCB-Sight dataset for tactile classification (real-environment data).
  • Manipulation Tasks:
    • Three real-world tasks using the Aloha hardware platform: Chicken Legs Hanging, Chips Grasping, and Allen Key Insertion.
  • A simulated peg insertion task using TacSL.

Key Insights and Ideas

  • Data Efficiency is Key: The most striking insight is that a highly generalizable tactile representation can be learned from a single, simple object. This drastically reduces the data collection burden for robotic manipulation.
  • VQGAN's Suitability for Tactile Data: The VQGAN architecture, with its ability to learn structured, low-variance latent spaces, is particularly well-suited for the compact color distribution of GelSight tactile images.
  • Zero-Shot Transfer is Powerful: UniT's ability to be zero-shot transferred to downstream tasks without fine-tuning the encoder is a significant advantage, making it a plug-and-play solution.
  • Tactile Modality is Crucial for Complex Interactions: For tasks involving intricate force interactions and precise alignments (like hanging chicken legs or inserting an Allen key), tactile feedback is shown to be essential, significantly outperforming vision-only policies.

Key Results

  • Pose Estimation: UniT outperformed existing methods like BYOL, MAE, and T3 on both 3D and 6D pose estimation tasks for USB plugs, achieving lower mean absolute errors. For example, in 6D pose estimation, UniT achieved a mean absolute error of 0.156 mm (Allen key dataset) compared to T3's 0.166 mm.
  • Classification: UniT achieved the best classification accuracy on the YCB-Sight dataset, particularly in the zero-shot setup, outperforming MAE and T3.
  • Manipulation Tasks: In real-world manipulation tasks (Chicken Legs Hanging, Chips Grasping, Allen Key Insertion), visual-tactile policies using UniT achieved significantly higher success rates (e.g., 23/30 for UniT vs. 17/30 for visual-tactile from scratch in Table V) compared to vision-only or visual-tactile policies trained from scratch.
  • Reconstruction: UniT reconstructions preserved critical information like contact geometry and configuration, even for unseen objects.

Limitations and Future Work

  • Sensor Specificity: UniT's current framework does not support transfer across drastically different sensor types, as it relies on minimal data from a single sensor type.
  • Future Work: The authors suggest improving data efficiency by incorporating a more diverse set of objects with finer textures to further enhance generalization. Investigating how training with a broader range of physical objects, including those with intricate patterns, can enrich the representation's ability to capture subtle tactile details is also proposed.

Summary updated

Abstract

UniT is an approach to tactile representation learning, using VQGAN to learn a compact latent space and serve as the tactile representation. It uses tactile images obtained from a single simple object to train the representation with generalizability. This tactile representation can be zero-shot transferred to various downstream tasks, including perception tasks and manipulation policy learning. Our benchmarkings on in-hand 3D pose and 6D pose estimation tasks and a tactile classification task show that UniT outperforms existing visual and tactile representation learning methods. Additionally, UniT's effectiveness in policy learning is demonstrated across three real-world tasks involving diverse manipulated objects and complex robot-object-environment interactions. Through extensive experimentation, UniT is shown to be a simple-to-train, plug-and-play, yet widely effective method for tactile representation learning. For more details, please refer to our open-source repository https://github.com/ZhengtongXu/UniT and the project website https://zhengtongxu.github.io/unit-website/.
BibTeX
@article{xu2025unit,
  author = {Zhengtong Xu and Raghava Uppuluri and Xinwei Zhang and Cael Fitch and Philip G. Crandall and Wan Shou and Dongyi Wang and Yu She},
  title = {{UniT: Data Efficient Tactile Representation With Generalization to Unseen Objects}},
  journal = {IEEE RA-L},
  year = {2025},
  doi = {10.1109/lra.2025.3559835},
}
Figure