Abstract
Robotic grasping remains a fundamental challenge in manipulation, often hindered by the limitations of vision-only systems in perceiving contact dynamics. This paper demonstrates that high-resolution tactile sensing, when fused with vision in an end-to-end deep learning framework, significantly improves a robot's ability to predict grasp success. By utilizing GelSight sensors to "feel" contact before lifting, the researchers achieved a 94% success rate on previously unseen objects, far outperforming traditional vision-only methods.
Introduction
While computer vision has enabled robots to locate objects with high precision, vision alone is often insufficient for reliable grasping. Once a gripper makes contact, the object may be occluded, or the robot may fail to perceive subtle cues like surface compliance, local geometry, or incipient slip. Traditional analytic methods attempt to model these interactions through complex physics equations, which frequently fail in real-world scenarios due to calibration errors. This research proposes a multi-modal deep learning approach to let a robot learn the "feeling of success" by combining visual context with tactile confirmation.
Methodology
Hardware and Experimental Setup
The researchers equipped a 7-DoF Sawyer robot arm with a parallel jaw gripper. Crucially, each finger was outfitted with a GelSight sensor—a high-resolution tactile sensor that uses an internal camera to image the deformation of a soft rubber skin. This allows the robot to capture fine-grained details of the contact surface.
The Multi-Modal Model
The core of the approach is a convolutional neural network (CNN) based on the ResNet-50 architecture, employing a "late fusion" strategy:
- Visual Branch: Processes RGB images taken both before the grasp (unoccluded) and during the grasp.
- Tactile Branch: Processes the "temporal difference" between GelSight images (before vs. during contact) to capture how the sensor skin deformed.
Features from both branches are concatenated and fed into fully connected layers to output a single probability of grasp success. To execute a grasp, the robot samples potential configurations, "pre-closes" the gripper, and only proceeds to lift if the predicted success probability exceeds 90%.
Datasets
The model was trained on a massive, automated dataset of over 9,269 grasping trials performed on 106 unique objects. Performance was evaluated on a held-out test set and a generalization set of 12 completely new objects never seen during training, such as a computer mouse and a stapler.
Key Results and Insights
- Tactile Superiority: In isolation, the tactile-only model (75.6% accuracy) outperformed the vision-only model (68.8%), suggesting that physical contact state is a more reliable indicator of success than external appearance.
- Multi-modal Synergy: The combined visuo-tactile model achieved 77.8% prediction accuracy on held-out data. In real-world trials on novel objects, the multi-modal approach reached a 94% success rate, compared to 80% for vision alone.
- Error Reduction: The visuo-tactile model virtually eliminated "empty grasps"—a common failure where vision-only systems attempt to lift without making solid contact.
- Complex Objects: For challenging items like a computer mouse, the success rate jumped from 10% (vision-only) to 60% (visuo-tactile).
Discussion and Future Work
The research proves that end-to-end deep learning can effectively extract nuanced information from high-resolution tactile images, outperforming hand-engineered features. However, the current system acts primarily as a rejection filter, which can be slow. Future work aims to use these models to actively guide the gripper toward optimal grasps. Additionally, addressing sensor durability and post-lift dynamics (such as slip during transport) remains a priority for long-term robotic autonomy.
Summary updated