ViTacTip: Design and Verification of a Novel Biomimetic Physical Vision-Tactile Fusion Sensor

Wen Fan, Haoran Li, Weiyong Si, Shan Luo, Nathan F. Lepora, Dandan Zhang

TL;DR

Researchers have developed ViTacTip, a novel biomimetic sensor that combines high-resolution tactile "feeling" with clear visual "seeing" in a single unit. By using a transparent skin embedded with internal pins and a GAN-based software layer, the sensor outperforms traditional single-modality sensors in tasks like identifying textures, locating contact points, and estimating force.

What Problem Does This Paper Address?

In robotics, "seeing" (vision) and "feeling" (tactile sensing) are often treated as separate systems. Traditional vision-based tactile sensors (VBTSs) like the popular TacTip or GelSight usually have opaque or reflective skins. While great at sensing deformation, these skins act like a blindfold, preventing the robot from seeing the object's color or geometry before and during contact.

Conversely, "eye-in-hand" systems that use standard cameras are prone to occlusion—the moment the robot touches an object, its own "finger" blocks the view. The challenge is creating a sensor that can seamlessly integrate both modalities without one interfering with the other.

Methods

The authors introduce a hardware-software co-design to solve the fusion problem:

  1. Biomimetic Hardware: Inspired by the layered structure of human skin, ViTacTip features a 3D-printed, transparent, flexible skin (made of Agilus30 Clear). Inside this skin are "pins" or markers that mimic the structure of the human epidermis.
  2. The "See-Through-Skin" Mechanism: An internal wide-angle camera looks through the transparent skin. When the sensor is 5–20mm away from an object, it captures visual data (color, shape). Upon contact, the internal pins deform, providing tactile data.
  3. GAN-Based Modality Switching: To handle the "noise" of having both pins and visual data in one image, the team used Pix2Pix Generative Adversarial Networks (GANs).
  4. One GAN (ViTacTip-to-ViTac) "removes" the markers to provide a clean visual feed.
  5. The other (ViTacTip-to-TacTip) "removes" the background visual noise to focus purely on the tactile markers for precise force and pose estimation.

Datasets

The researchers collected three primary datasets using a desktop robot arm (Dobot MG400):

  • Grating Identification: 3,500 data points across seven different line-spacing densities (from 0mm to 2mm).
  • Edge Pose Regression: 3,000 images capturing various horizontal distances, press depths, and rotation angles against a square stimulus.
  • Force & Localization: A dataset mapping internal camera images to ground-truth force data provided by an NBIT force sensor, including variations in external lighting to test robustness.

Key Insights and Ideas

  • The Best of Both Worlds: By making the skin transparent and adding pins, the sensor gains the high-sensitivity force detection of a TacTip while maintaining the contextual awareness of a standard camera.
  • Software as a Filter: The use of GANs is a clever way to solve a physical design trade-off. Instead of trying to make markers invisible, they let the markers be visible and use AI to "edit" them out when the robot needs to focus on visual features.
  • Visual Cues for Orientation: The study found that while pins are essential for sensing depth (Z-axis) and horizontal distance (X-axis), visual features are actually superior for determining the orientation (angle) of an edge, as the camera can "see" the line of the edge more clearly than the pins can represent it.

Key Results

ViTacTip was compared against two baselines: TacTip (tactile only, opaque) and ViTac (vision only, no pins).

  • Grating Identification: ViTacTip achieved 99.72% accuracy, significantly higher than TacTip’s 94.60%.
  • Error Reduction: Compared to the standard TacTip, ViTacTip reduced:
  • Pose regression error by 46%.
  • Contact localization error by 60%.
  • Force estimation error by 52%.
  • Precision: It achieved a minimum pose error of 0.08 mm and a force estimation error of just 0.03N, outperforming the vision-only ViTac (0.12 mm and 0.15N).

Limitations and Future Work

While the GANs help mitigate ambient light interference, the sensor's performance is still somewhat dependent on external lighting conditions due to its transparent nature.

The authors note that future work will involve:

  • Testing the sensor under more extreme and varied environmental conditions.
  • Integrating ViTacTip into multi-fingered robotic hands to perform complex, dexterous manipulation tasks (like picking up and rotating small, fragile objects) where both sight and touch are critical.

Summary updated

Abstract

Tactile sensing is significant for robotics since it can obtain physical contact information during manipulation. To capture multimodal contact information within a compact framework, we designed a novel sensor called ViTacTip, which seamlessly integrates both tactile and visual perception capabilities into a single, integrated sensor unit. ViTacTip features a transparent skin to capture fine features of objects during contact, which can be known as the see-through-skin mechanism. In the meantime, the biomimetic tips embedded in ViTacTip can amplify touch motions during tactile perception. For comparative analysis, we also fabricated a ViTac sensor devoid of biomimetic tips, as well as a TacTip sensor with opaque skin. Furthermore, we develop a Generative Adversarial Network (GAN)-based approach for modality switching between different perception modes, effectively alternating the emphasis between vision and tactile perception modes. We conducted a performance evaluation of the proposed sensor across three distinct tasks: i) grating identification, ii) pose regression, iii) contact localization and force estimation. In the grating identification task, ViTacTip demonstrated an accuracy of 99.72%, surpassing TacTip, which achieved 94.60%. It also exhibited superior performance in both pose and force estimation tasks with the minimum error of 0.08 mm and 0.03N, respectively, in contrast to ViTac’s 0.12 mm and 0.15N. Results indicate that ViTacTip outperforms single-modality sensors.
BibTeX
@inproceedings{fan2024vitactip,
  author = {Wen Fan and Haoran Li and Weiyong Si and Shan Luo and Nathan F. Lepora and Dandan Zhang},
  title = {{ViTacTip: Design and Verification of a Novel Biomimetic Physical Vision-Tactile Fusion Sensor}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=HIUknOfFFY},
}

Loading related content...

Figure