The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?

Roberto Calandra, Andrew Owens, Manu Upadhyaya, Wenzhen Yuan, Justin Lin, Edward H. Adelson, Sergey Levine

Abstract

Robotic grasping remains a fundamental challenge in manipulation, often hindered by the limitations of vision-only systems in perceiving contact dynamics. This paper demonstrates that high-resolution tactile sensing, when fused with vision in an end-to-end deep learning framework, significantly improves a robot's ability to predict grasp success. By utilizing GelSight sensors to "feel" contact before lifting, the researchers achieved a 94% success rate on previously unseen objects, far outperforming traditional vision-only methods.

Introduction

While computer vision has enabled robots to locate objects with high precision, vision alone is often insufficient for reliable grasping. Once a gripper makes contact, the object may be occluded, or the robot may fail to perceive subtle cues like surface compliance, local geometry, or incipient slip. Traditional analytic methods attempt to model these interactions through complex physics equations, which frequently fail in real-world scenarios due to calibration errors. This research proposes a multi-modal deep learning approach to let a robot learn the "feeling of success" by combining visual context with tactile confirmation.

Methodology

Hardware and Experimental Setup

The researchers equipped a 7-DoF Sawyer robot arm with a parallel jaw gripper. Crucially, each finger was outfitted with a GelSight sensor—a high-resolution tactile sensor that uses an internal camera to image the deformation of a soft rubber skin. This allows the robot to capture fine-grained details of the contact surface.

The Sawyer robot arm equipped with GelSight tactile sensors performing a grasp on a toy robot.

The Multi-Modal Model

The core of the approach is a convolutional neural network (CNN) based on the ResNet-50 architecture, employing a "late fusion" strategy:

  • Visual Branch: Processes RGB images taken both before the grasp (unoccluded) and during the grasp.
  • Tactile Branch: Processes the "temporal difference" between GelSight images (before vs. during contact) to capture how the sensor skin deformed.
A high-resolution tactile image from the GelSight sensor showing the deformation pattern during contact.

Features from both branches are concatenated and fed into fully connected layers to output a single probability of grasp success. To execute a grasp, the robot samples potential configurations, "pre-closes" the gripper, and only proceeds to lift if the predicted success probability exceeds 90%.

Datasets

The model was trained on a massive, automated dataset of over 9,269 grasping trials performed on 106 unique objects. Performance was evaluated on a held-out test set and a generalization set of 12 completely new objects never seen during training, such as a computer mouse and a stapler.

Key Results and Insights

  • Tactile Superiority: In isolation, the tactile-only model (75.6% accuracy) outperformed the vision-only model (68.8%), suggesting that physical contact state is a more reliable indicator of success than external appearance.
  • Multi-modal Synergy: The combined visuo-tactile model achieved 77.8% prediction accuracy on held-out data. In real-world trials on novel objects, the multi-modal approach reached a 94% success rate, compared to 80% for vision alone.
  • Error Reduction: The visuo-tactile model virtually eliminated "empty grasps"—a common failure where vision-only systems attempt to lift without making solid contact.
  • Complex Objects: For challenging items like a computer mouse, the success rate jumped from 10% (vision-only) to 60% (visuo-tactile).

Discussion and Future Work

The research proves that end-to-end deep learning can effectively extract nuanced information from high-resolution tactile images, outperforming hand-engineered features. However, the current system acts primarily as a rejection filter, which can be slow. Future work aims to use these models to actively guide the gripper toward optimal grasps. Additionally, addressing sensor durability and post-lift dynamics (such as slip during transport) remains a priority for long-term robotic autonomy.

Summary updated

BibTeX
@inproceedings{calandra2017feeling,
  author = {Roberto Calandra and Andrew Owens and Manu Upadhyaya and Wenzhen Yuan and Justin Lin and Edward H. Adelson and Sergey Levine},
  title = {{The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?}},
  booktitle = {Conference on Robot Learning},
  year = {2017},
}
Figure