GelSight Svelte: A Human Finger-shaped Single-camera Tactile Robot Finger with Large Sensing Coverage and Proprioceptive Sensing

Jialiang Zhao, Edward H. Adelson

Abstract

Camera-based tactile sensing is a low-cost, popular approach to obtain highly detailed contact geometry information. However, most existing camera-based tactile sensors are fingertip sensors, and longer fingers often require extraneous elements to obtain an extended sensing area similar to the full length of a human finger. Moreover, existing methods to estimate proprioceptive information such as total forces and torques applied on the finger from camera-based tactile sensors are not effective when the contact geometry is complex. We introduce GelSight Svelte, a curved, human finger-sized, single-camera tactile sensor that is capable of both tactile and proprioceptive sensing over a large area. GelSight Svelte uses curved mirrors to achieve the desired shape and sensing coverage. Proprioceptive information, such as the total bending and twisting torques applied on the finger, is reflected as deformations on the flexible backbone of GelSight Svelte, which are also captured by the camera. We train a convolutional neural network to estimate the bending and twisting torques from the captured images. We conduct gel deformation experiments at various locations of the finger to evaluate the tactile sensing capability and proprioceptive sensing accuracy. To demonstrate the capability and potential uses of GelSight Svelte, we conduct an object holding task with three different grasping modes that utilize different areas of the finger.

Introduction

Most camera-based tactile sensors, such as the original GelSight, are limited to "fingertip" applications. They are typically flat, bulky, and only provide high-resolution data at the very end of a gripper. However, human dexterity relies on the ability to sense touch along the entire length of the finger.

Previous attempts to create elongated tactile sensors often required multiple cameras or external "exoskeletons," which increase the size and complexity of the robot hand. Furthermore, while these sensors are excellent at capturing local surface geometry (tactile sensing), they struggle to estimate global forces like bending or twisting (proprioception) without additional hardware. GelSight Svelte aims to integrate both high-resolution tactile sensing and proprioception into a compact, human-sized form factor.

Sensor Design and Methodology

The researchers achieved this through a combination of novel optical and mechanical design, packing complex sensing capabilities into a slim profile.

The GelSight Svelte sensor: physical prototype, CAD model, and exploded view of internal components.
  • Folded Optical Path: To fit a camera inside a human-sized finger while covering a large area, the team used a single 120° wide-angle Raspberry Pi camera and two mirrors. One of these mirrors is curved, which redistributes the light and allows the camera to "see" the entire length of the curved sensing skin.
  • The "Svelte" Backbone: The finger’s core is a flexible backbone 3D-printed from Markforged Onyx (a carbon-fiber-filled nylon). This material is strong yet elastic, serving as the structural foundation for the sensor.
  • Dual-Purpose Imaging:
    • Tactile Sensing: Like other GelSight sensors, it uses red and green LEDs to illuminate a reflective coating on a silicone gel. The camera captures how the gel deforms against an object to reconstruct 3D surface geometry.
  • Proprioceptive Sensing: The LED strips are glued directly to the flexible backbone. When the finger bends or twists under load, the backbone deforms, and the LEDs move within the camera's field of view.
Cross-sectional views (A-A and B-B) illustrating the internal layout of the mirrors, LEDs, and flexible backbone.

Data Collection and Training

The authors generated a custom dataset to train a Convolutional Neural Network (CNN) to estimate bending and twisting torques ($T_{bend}$ and $T_{twist}$) from captured images.

  • Proprioceptive Training Data: The finger was mounted and probed with a load cell to measure applied forces. Simultaneously, an OptiTrack motion capture system tracked the relative positions of the finger and the probe. This allowed the team to calculate ground-truth torques to pair with the captured camera images.
  • Evaluation: The model was evaluated on a separate test set collected from a different physical finger to ensure the system generalizes across hardware variations.

Technical Innovations

  • Curved Mirrors for FOV: The use of a curved mirror solves the geometric constraint of trying to look at a long, curved surface from a very close distance without adding more cameras.
  • Backbone as a Strain Gauge: Instead of calculating global force from local skin deformation—which is computationally expensive and often inaccurate due to the viscoelasticity of silicone—they move the sensing of global force to the structural backbone. Tracking the backbone's movement via the LEDs provides a much cleaner signal for proprioception.
  • The Air-Gel Interface: A flat interface between the camera (in air) and the silicone gel would severely limit the field of view due to refraction. The researchers solved this by gluing a 7mm glass ball into the mold to create a hemispherical "dome" interface, significantly widening the camera's effective view.

Experimental Results

  • Human-Like Dimensions: The sensor is 83.5mm long and roughly 20mm wide, making it comparable to a human finger.
  • High Resolution: The spatial resolution ranges from 30 px/mm at the base to 19 px/mm at the tip. In tests, the sensor could clearly resolve the fine threads of an M8 screw at multiple points along the finger.
  • Accurate Torque Sensing: The CNN estimated bending torques with an RMSE of 9.4 Nmm and twisting torques with an RMSE of 7.6 Nmm.
  • Versatile Grasping: The team demonstrated the finger's utility in three distinct modes: a two-finger pinch grasp (LEGO brick), a two-finger lateral pinch (screwdriver), and a three-finger power grasp (flashlight).

Conclusion and Future Work

While GelSight Svelte represents a significant step forward, several areas for future development remain. Currently, the demonstrations are open-loop; the next step is to use real-time torque and tactile feedback for closed-loop manipulation. Additionally, because of the camera's perspective, the resolution at the fingertip is lower than at the base, which may be addressed by further optimizing mirror curvature. Finally, long-term durability of the reflective paint layer under heavy use remains a standard challenge for GelSight-style sensors.

Summary updated

Abstract

Camera-based tactile sensing is a low-cost, popular approach to obtain highly detailed contact geometry information. However, most existing camera-based tactile sensors are fingertip sensors, and longer fingers often require extraneous elements to obtain an extended sensing area similar to the full length of a human finger. Moreover, existing methods to estimate proprioceptive information such as total forces and torques applied on the finger from camera-based tactile sensors are not effective when the contact geometry is complex. We introduce GelSight Svelte, a curved, human finger-sized, single-camera tactile sensor that is capable of both tactile and proprioceptive sensing over a large area. GelSight Svelte uses curved mirrors to achieve the desired shape and sensing coverage. Proprioceptive information, such as the total bending and twisting torques applied on the finger, is reflected as deformations on the flexible backbone of GelSight Svelte, which are also captured by the camera. We train a convolutional neural network to estimate the bending and twisting torques from the captured images. We conduct gel deformation experiments at various locations of the finger to evaluate the tactile sensing capability and proprioceptive sensing accuracy. To demonstrate the capability and potential uses of GelSight Svelte, we conduct an object holding task with three different grasping modes that utilize different areas of the finger. More information is available on our website: https://gelsight-svelte.alanz.info
BibTeX
@inproceedings{zhao2023gelsight,
  author = {Jialiang Zhao and Edward H. Adelson},
  title = {{GelSight Svelte: A Human Finger-shaped Single-camera Tactile Robot Finger with Large Sensing Coverage and Proprioceptive Sensing}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=MPP2P2JX1C},
}
Figure