Computer vision is remarkably efficient at capturing global scene structure instantaneously, but it suffers from two fundamental limitations: line-of-sight occlusions and optical ambiguities. When a robotic hand grasps an object, it inherently blocks the camera's view of the contact area. Furthermore, transparent, highly reflective, or specular surfaces violate the multi-view consistency assumptions that standard optical reconstruction algorithms rely on.
To overcome these challenges, a powerful paradigm has emerged: fusing global, coarse visual data with local, precise tactile feedback. Touch provides an orthogonal sensing modality that is entirely immune to lighting conditions, transparency, and visual obstructions. By treating physical contact as a localized geometric truth, robots can anchor, correct, and complete visual models of the physical world.
The Global-Local Complementarity
The core intuition behind visuo-tactile fusion is a division of labor between "sight" and "touch." Vision establishes a rapid, global prior of an object's shape, while tactile sensors provide high-fidelity local surface details at the exact points of interaction.
This synergy dates back to foundational robotics research. For instance, Aida et al. demonstrated that sparse physical contact data could be converted into tangential plane equations to mathematically correct and fill in obscure or low-quality visual imagery. Modern frameworks scale this intuition using dense tactile sensors and deep learning. In ShapeMap 3-D, Suresh et al. integrated local dense touch from GelSight sensors into a global probabilistic Gaussian Process spatial graph, allowing the robot to map objects even under poor lighting conditions.
[Global Vision System] ───> Coarse Geometric Prior ──┐
│ ──> [Multimodal Fusion] ──> Accurate 3D Model
[Local Tactile Sensor] ───> High-Fidelity Contact ──┘
This global-local partitioning is equally effective in specialized engineering tasks. Wang et al. combined tactile pneumatic probes with a multi-camera vision system to measure engine crankshafts; the tactile probes established a stable physical reference axis on rough surfaces, while the vision system provided rapid localization of complex features. To make these multimodal networks robust, Wang et al. introduced a particle-based elastic interaction simulation that allows networks to learn "global-to-local" fusion in simulation before transferring to real-world sensors, significantly reducing geometric error by resolving distant visual ambiguities.
Overcoming Manipulation Occlusions and Deformable Dynamics
When a robot actively manipulates an object, its fingers inevitably create severe visual occlusions. Visuo-tactile fusion solves this by using touch as a physical proxy for the hidden state of the object.
In the context of rigid object manipulation, Xu et al. developed the VTacO framework, which fuses global visual context with local data from high-resolution DIGIT tactile sensors. This allows the system to "see" through the manipulating hand and accurately reconstruct the fine-grained surface details of the object. For downstream tasks like tracking, Wan introduced the VinT-6D dataset and a neural architecture that integrates RGB-D data with whole-hand tactile point clouds, ensuring precise 6D pose estimation even when the object is almost entirely obscured by a dexterous hand.
When objects are deformable, the geometry changes dynamically under pressure, making vision-only tracking impossible once contact occurs. To capture these complex physics, Jiang et al. combined high-density tactile force maps from a stretchable glove with depth camera point clouds, successfully reconstructing the state of highly compliant objects during intricate tasks like kneading dough.
Similarly, the VIRDO++ framework developed by Wi et al. pairs partial visual data with wrist-mounted force-torque sensors. By learning a latent representation of object dynamics, the system treats force feedback as a proxy for deformation, allowing it to infer the shape of occluded, flexible components (like a bending spatula) that are completely hidden from the camera.
Visual Data (Occluded/Incomplete) ──┐
├──> [Dynamics & Geometry Fusion] ──> Deformable State Tracking
Tactile Force/Torque Feedback ──┘
Even when completing static point clouds, touch prevents the geometric errors common to vision-only models. As Shi et al. demonstrated, standard visual point cloud completion networks tend to smooth over fine structures; integrating local tactile geometry forces the network to preserve sharp edges and intricate local features that are typically lost in occluded regions.
Correcting Vision-Breaking Surfaces
Some of the most significant breakthroughs in this field address "vision-breaking" surfaces—such as mirrors, glass, and highly reflective metals—where standard optical reconstruction pipelines fail entirely. Algorithms like 3D Gaussian Splatting (3DGS) rely on multi-view consistency, assuming that a physical point looks the same from different angles. Reflections and refractions break this assumption.
Touch provides a direct solution because physical contact does not change based on surface reflectance. Swann et al. introduced Touch-GS, a framework that supervises 3DGS using Gaussian Process-based touch surfaces. By anchoring the radiance fields with physical depth measurements, the system successfully reconstructs mirrors and transparent prisms from only a few viewpoints.
Reflective/Transparent Surface ──> Breaks Multi-View Consistency ──> Visual Artifacts
│
Physical Contact (Tactile) ──> Provides Absolute Local Depth ───> [Corrected 3DGS]
Crucially, touch does not need to be dense to fix these visual artifacts. Tonioni et al. showed that integrating a minimal amount of sparse tactile information (as few as five contact points) into a 3DGS pipeline drastically reduces geometric reconstruction errors on non-Lambertian surfaces. This demonstrates that sparse physical touch can effectively compensate for limited or corrupted camera perspectives.
This physical grounding also enhances generative models. Deng et al. developed Tactile DreamFusion, which incorporates high-resolution tactile priors into a diffusion-based 3D generation pipeline. By forcing the generative model to adhere to real-world tactile textures, the system produces 3D assets with intricate, physically accurate surface details that far surpass the geometric realism of vision-only models.
Latent Optimization and Active Exploration
Fusing these modalities effectively requires algorithms that know how to resolve conflicts between sight and touch, and how to guide the robot to gather the most informative data.
One approach is to frame fusion as a constrained optimization problem in a shared latent space. Saund et al. developed CLASP (Constrained Latent Shape Projection), which takes an initial visual shape prediction and refines it by performing gradient descent directly within the latent space of a shape-completion network. The optimization is constrained not just by where the robot touched the object (collision points), but also by the free-space volumes the robot's arm traversed without collision. This dual constraint continuously shrinks geometric uncertainty in occluded regions until the model matches physical reality.
Initial Visual Prediction ──> [Latent Space Optimization] <── Constraints: Collision Points & Free Space
│
└──> Physically Consistent 3D Shape
Instead of touching blindly, robots can also use visual uncertainty to decide where to feel next. Kyrki proposed Act-VH, an active shape completion method that calculates reconstruction uncertainty from visual data and directs haptic exploration specifically toward those high-uncertainty zones, leading to significantly higher success rates in robotic grasping. For continuous surface mapping, Chen et al. utilized a sliding touch strategy combined with Bayesian Optimization, allowing a multi-fingered hand to efficiently map unknown shapes with minimal movements.
This active interplay is also vital for high-precision inspection. Jiang et al. developed a vision-guided active tactile system for crack detection. Computer vision first scans the environment to identify candidate defect areas, and a high-resolution tactile sensor is then deployed to actively verify and reconstruct the micro-geometry. This prevents the system from being fooled by visual illusions like paint marks, achieving sub-millimeter precision at a fraction of the speed of a blind tactile scan.
Finally, this active loop has been integrated into radiance field frameworks. Strong et al. extended the FisherRF uncertainty metric to account for depth, creating an autonomous "Next Best Sense" pipeline. The system evaluates the current 3DGS model and mathematically determines whether the robot should take another picture or physically touch a surface, directing physical contact precisely toward regions of high surface curvature or visual ambiguity to fill in geometric holes.
Key Foundational Work
Visual-Tactile Symbiotic System for Stereometric Pattern Recognition (1971) — contributed 1 concept elements to this direction.
The Fast Measurement for the Relative Position of Flange-End Holes (2019) — contributed 1 concept elements to this direction.