For a robot to manipulate the world with human-like dexterity, it must do more than just see an object from a distance; it must "feel" the interaction at the point of contact. Traditional tactile sensors—often based on resistive or capacitive arrays—frequently struggle with low resolution, complex wiring, and fragility.
A transformative direction in robotics bypasses these mechanical constraints by treating touch as a computer vision problem. By embedding a high-resolution camera inside a robotic finger to observe the deformation of a soft, flexible skin, researchers have unlocked a way to capture incredibly detailed surface geometry, force distributions, and even the robot’s own internal state.
The Core Mechanism: Touch as Vision
The fundamental intuition behind vision-based tactile sensing, exemplified by the GelSight sensor developed by Rui Li and colleagues, is the conversion of physical contact into a controlled optical signal.
The sensor consists of a clear elastomer block coated with a reflective, opaque membrane. When an object presses against this membrane, it deforms to match the object's micro-geometry. An internal camera, illuminated by colored LEDs from different angles, captures these deformations as an image. Because the lighting environment is entirely self-contained, the system eliminates external confounds like shadows or ambient light. This allows standard convolutional neural networks (CNNs) to process tactile "images" to estimate object orientation, as demonstrated by Luca Lach and Jan Peters in their work on precise object placing.
Fusing Vision and Touch into Neural Fields
While a single touch provides a high-resolution "snapshot" of a surface, real-world manipulation requires a global understanding of the object. Sudharshan Suresh, Jitendra Malik, and their team proposed NeuralFeels, a framework that treats tactile sensors as short-range, high-resolution cameras.
By fusing these local tactile "views" with global visual data from external cameras, they build a unified 3D representation called a neural field. This allows a robot to track and reconstruct an object in real-time, even when the object is almost entirely occluded by the robot’s own hand. The tactile data acts as a "geometric ground truth," refining the global model where visual cameras are blind.
Proprioception Through Internal Observation
The utility of internal cameras extends beyond external contact. In soft robotics, knowing the configuration of one's own body (proprioception) is notoriously difficult because the structures are compliant and non-linear.
Researchers like Peiyu Yu and Edward H. Adelson developed GelFlex, a soft finger that uses an internal camera to track high-contrast markers painted on its interior walls. As the finger bends or twists, the camera observes the displacement of these markers. This transforms the complex problem of physical strain into a high-dimensional visual pattern, allowing the robot to "see" its own pose without the need for fragile, embedded electronic strain gauges. Similarly, Peter Werner demonstrated that visual proprioception can outperform traditional sensors in tracking the complex, non-linear movements of inflatable actuators.
Bridging the Sim-to-Real Gap
A major hurdle for tactile sensing is the difficulty of collecting large-scale real-world data. To solve this, researchers are developing high-fidelity simulators that model the physics of elastomer deformation.
Zixi Chen and Shan Luo introduced Tacchi, a simulator that uses particle-based physics to generate realistic tactile images. By making these simulators differentiable, as seen in the DIFFTACTILE framework by Zilin Si and colleagues, robots can use gradient-based optimization to learn complex manipulation skills in simulation and transfer them directly to the physical world.
Beyond Geometry: Hardness and Micro-texture
Visuotactile sensing isn't limited to shape. Because the camera captures the dynamics of deformation, it can infer material properties. Wenzhen Yuan and Andrew Owens demonstrated that a GelSight sensor could estimate the hardness of objects by observing how the contact patch expands over time.
More recently, Kangle Deng and Jun-Yan Zhu showed in Tactile DreamFusion that tactile sensors can capture micro-textures—details too fine for standard optical cameras—to generate physically accurate 3D assets. By integrating these sensors into multimodal fingertips like Digit 360, robots are beginning to perceive the world with a richness that rivals, and in some cases exceeds, human touch.