Abstract
Tactile sensing is significant for robotics since it can obtain physical contact information during manipulation. To capture multimodal contact information within a compact framework, we designed a novel sensor called ViTacTip, which seamlessly integrates both tactile and visual perception capabilities into a single, integrated sensor unit. ViTacTip features a transparent skin to capture fine features of objects during contact, which can be known as the see-through-skin mechanism. In the meantime, the biomimetic tips embedded in ViTacTip can amplify touch motions during tactile perception. For comparative analysis, we also fabricated a ViTac sensor devoid of biomimetic tips, as well as a TacTip sensor with opaque skin. Furthermore, we develop a Generative Adversarial Network (GAN)-based approach for modality switching between different perception modes, effectively alternating the emphasis between vision and tactile perception modes. We conducted a performance evaluation of the proposed sensor across three distinct tasks: i) grating identification, ii) pose regression, and iii) contact localization and force estimation. In the grating identification task, ViTacTip demonstrated an accuracy of 99.72%, surpassing TacTip, which achieved 94.60%. It also exhibited superior performance in both pose and force estimation tasks with the minimum error of 0.08mm and 0.03N, respectively, in contrast to ViTac's 0.12mm and 0.15N. Results indicate that ViTacTip outperforms single-modality sensors.
Introduction
In robotics, "seeing" (vision) and "feeling" (tactile sensing) are often treated as separate systems. Traditional vision-based tactile sensors (VBTSs), such as the popular TacTip or GelSight, typically utilize opaque or reflective skins. While highly effective at sensing deformation, these skins act as a blindfold, preventing the robot from perceiving an object's color or geometry before and during contact. Conversely, "eye-in-hand" systems using standard cameras are prone to occlusion; the moment the robot touches an object, its own manipulator blocks the view. The challenge lies in creating a sensor that integrates both modalities without mutual interference.
System Design and Hardware
The ViTacTip sensor employs a hardware-software co-design to solve the fusion problem. Inspired by the layered structure of human skin, ViTacTip features a 3D-printed, transparent, flexible skin made of Agilus30 Clear. Embedded within this skin are biomimetic "pins" or markers that mimic the structure of the human epidermis.
An internal wide-angle camera facilitates the "see-through-skin" mechanism. When the sensor is positioned 5–20mm from an object, it captures visual data such as color and shape. Upon contact, the internal pins deform, providing high-resolution tactile data. This dual-purpose design allows the sensor to maintain the contextual awareness of a standard camera while gaining the high-sensitivity force detection of a tactile sensor.
Modality Switching via GANs
To manage the simultaneous presence of tactile markers and visual background data in a single image, the researchers implemented a software layer based on Pix2Pix Generative Adversarial Networks (GANs). This approach allows for effective modality switching:
- ViTacTip-to-ViTac GAN: This network "removes" the tactile markers to provide a clean visual feed for identifying object features.
- ViTacTip-to-TacTip GAN: This network "removes" the background visual noise to focus purely on the tactile markers, enabling precise force and pose estimation.
Experimental Evaluation and Results
The performance of ViTacTip was evaluated against two baselines: TacTip (tactile-only, opaque skin) and ViTac (vision-only, no pins). Data was collected using a Dobot MG400 robot arm across three primary tasks:
- Grating Identification: ViTacTip achieved a remarkable 99.72% accuracy in identifying line-spacing densities, significantly outperforming TacTip’s 94.60%.
- Pose and Force Estimation: ViTacTip demonstrated superior precision, achieving a minimum pose error of 0.08 mm and a force estimation error of 0.03N. In comparison, the vision-only ViTac achieved errors of 0.12 mm and 0.15N, respectively.
- Error Reduction: When compared to the standard TacTip, the fusion sensor reduced pose regression error by 46%, contact localization error by 60%, and force estimation error by 52%.
Conclusion
ViTacTip successfully demonstrates that integrating visual and tactile modalities into a single biomimetic unit provides superior performance over single-modality sensors. By utilizing a transparent skin and GAN-based filtering, the sensor overcomes traditional issues of occlusion and limited contextual awareness. Future work will focus on testing the sensor under more varied environmental lighting conditions and integrating it into multi-fingered robotic hands for complex, dexterous manipulation tasks.
Summary updated