ManiSkill-ViTac 2025: Challenge on Manipulation Skill Learning With Vision and Tactile Sensing

Chuanyu Li, Renjun Dang, Xiang Li, Zhiyuan Wu, Jing Xu, Hamidreza Kasaei, Roberto Calandra, Nathan F. Lepora, Shan Luo, Hao Su, Rui ChenarXiv 2024 · DOI · arXiv

Abstract

The ManiSkill-ViTac 2025 Challenge is a benchmark and competition designed to advance robotic manipulation by integrating high-fidelity tactile sensing with computer vision. By providing a standardized simulation-to-real pipeline, the challenge tasks researchers with mastering "contact-rich" scenarios—such as inserting keys into locks—where vision alone often fails. This paper introduces the framework, simulation tools, and competition tracks designed to bridge the "sim-to-real gap" in dexterous manipulation.

Introduction: The Challenge of the "Last Centimeter"

While AI has made massive strides in computer vision and high-level reasoning, robots still struggle with the "last centimeter" of manipulation. Most current systems rely almost exclusively on visual data. However, in the real world, fingers often occlude the objects they are touching, and lighting can be unpredictable. To achieve human-like dexterity, robots need a sense of touch (tactile sensing).

The primary hurdles in this field are the lack of standardized benchmarks and the "sim-to-real gap"—the difficulty of training a robot in a fast simulation and having that knowledge translate successfully to a physical machine. The ManiSkill-ViTac 2025 Challenge addresses these problems by providing the community with the tools, simulation environments, and hardware standards needed to solve contact-rich tasks.

A High-Fidelity Simulation-to-Real Pipeline

The authors present a comprehensive framework for training and evaluating manipulation policies, bridging the gap between virtual training and physical execution. To ensure a level playing field, the organizers provide a standardized hardware platform (3-axis stages, GelSight sensors, and RealSense cameras) for real-world evaluation of participants' code.

Comparison between the real-world experimental setup and the high-fidelity simulation environment used in the challenge.

The platform utilizes the SAPIEN robotics simulator integrated with Incremental Potential Contact (IPC). Unlike standard rigid-body simulators, IPC allows for the accurate simulation of soft, deformable materials, such as the silicone "skin" of a tactile sensor, using Finite Element Method (FEM) discretization. This ensures that the simulation accurately models how a rubber sensor tip deforms against a metal object, which is critical for policy transfer.

Modeling Contact and Vision

Tactile sensing is modeled after the GelSight Mini sensor. The simulation tracks the displacement of "markers" on a virtual elastomer, which are then transformed into signals mimicking real-world tactile data.

Modeling the GelSight Mini sensor: a tetrahedral mesh represents the elastomer, and marker displacements are projected onto a virtual camera's image plane.

To simulate depth sensing, the framework employs an active stereo sensor simulation that accounts for infrared (IR) patterns, providing a realistic representation of cameras like the Intel RealSense D415. For processing this data, the paper suggests using Twin Delayed Deep Deterministic Policy Gradient (TD3) for reinforcement learning, combined with Shared Weight Encoder Networks and the Segment Anything Model (SAM) for visual object segmentation.

Competition Tracks and Objectives

The challenge is divided into three distinct tracks:

  1. Track 1: Tactile Manipulation: Focuses on "blind" manipulation (e.g., Peg Insertion and Lock Opening) using only touch data.
  2. Track 2: Tactile-Vision Fusion: Requires the robot to use vision to locate targets and tactile sensing to perform the final, precise insertion.
  3. Track 3: Sensor Structure Design: A novel track where participants optimize the physical shape and structure of the tactile sensor's silicone component to improve information gain.

Key Insights and Future Directions

A central theme of the challenge is the synergy between "distal" and "proximal" cues. Vision provides distal cues for locating objects from a distance, while tactile sensing provides proximal cues for detecting slippage or misalignment upon contact. Successful manipulation requires a tight fusion of both.

As a technical specification for the 2025 challenge, the paper establishes a clear evaluation metric based on "Surface Point Consistency" and "Excessive Error Checks" to ensure precision and stability. While current tasks like peg-in-hole are relatively static, future iterations aim to include multi-stage assembly, more complex dexterous manipulation, and interactions with deformable materials like cloth or food. The 2025 challenge features a total prize pool of $14,500 to incentivize research in this critical area of robotics.

Summary updated

Abstract

This article introduces the ManiSkill-ViTac Challenge 2025, which focuses on learning contact-rich manipulation skills using both tactile and visual sensing. Expanding upon the 2024 challenge, ManiSkill-ViTac 2025 includes 3 independent tracks: tactile manipulation, tactile-vision fusion manipulation, and tactile sensor structure design. The challenge aims to push the boundaries of robotic manipulation skills, emphasizing the integration of tactile and visual data to enhance performance in complex, real-world tasks. Participants will be evaluated using standardized metrics across both simulated and real-world environments, spurring innovations in sensor design and significantly advancing the field of vision-tactile fusion in robotics.
BibTeX
@misc{li2024maniskill,
  author = {Chuanyu Li and Renjun Dang and Xiang Li and Zhiyuan Wu and Jing Xu and Hamidreza Kasaei and Roberto Calandra and Nathan F. Lepora and Shan Luo and Hao Su and Rui Chen},
  title = {{ManiSkill-ViTac 2025: Challenge on Manipulation Skill Learning With Vision and Tactile Sensing}},
  year = {2024},
  eprint = {2411.12503},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.12503},
}
Figure