Imagine trying to plug a cable into the back of a desktop computer entirely by sight. If your hand blocks your view of the port, or if the lighting is poor, you quickly abandon a purely visual strategy. Instead, you guide your hand to the general area using your eyes, and then let your fingertips "feel" for the edges of the slot until the plug clicks into place.
For decades, robotic manipulation has relied overwhelmingly on computer vision. While vision is excellent for mapping a room or locating an object from a distance, it fails during the most critical phase of physical interaction: the moment of contact. To bridge this gap, a major research direction focuses on visuotactile fusion—the deep integration of global visual context with local tactile feedback to enable robots to perform precise, contact-rich tasks.
The Sensory Divide: Why Vision Alone Fails
The fundamental challenge in robotic manipulation stems from an asymmetry between what a robot sees and what it feels. As Yuan notes, vision provides a broad environmental overview but is frequently obstructed by the robot's own structure, whereas tactile data offers direct contact information but lacks broader spatial context.
When a robot relies exclusively on raw visual feedback or point clouds, its performance degrades in cluttered, occluded, or highly dynamic environments (Yuan et al.; Cheng et al.). Vision can tell a robot where an object is, but it cannot specify the exact micro-alignments, friction forces, or stability of a grasp once the robot's hand closes around that object.
The Macro-to-Micro Pipeline: See-Then-Feel
To resolve this sensory divide, researchers have converged on a hierarchical "division of labor" between sight and touch.
[ Visual Perception ] ---> Global Context & Initial Approach (Macro)
|
v
[ Tactile Feedback ] ---> Local Alignment & Force Regulation (Micro)
Kim describes this as a sequential "see-then-feel" strategy. High-level visual data is leveraged for initial object localization and coarse path planning. Once the robot makes contact, fine-grained tactile scanning takes over to map irregular surfaces or adjust the grasp.
This synergy is highly effective for complex assembly tasks. Shi et al. demonstrate that using vision-based guidance for large-scale movements combined with force-feedback reinforcement learning for local contact tasks significantly improves success rates in uncertain environments. The visual cues navigate the arm to the general workspace, and tactile sensing resolves the fine-grained positioning errors that vision alone cannot detect.
Bridging the Occlusion Gap
The moment a robotic hand grasps an object, it creates a visual blind spot by blocking its own cameras. Visuotactile fusion treats touch as a proxy for vision during these periods of occlusion.
In precision-heavy tasks like plug insertion, tactile feedback provides the necessary alignment information to overcome visual obstructions (Sharma et al.). Without touch, a robot is forced into inefficient "grasp-drop-regrasp" cycles. Integrating tactile sensing significantly reduces these failures by providing direct, real-time confirmation of contact when the hand blocks the camera's view, as shown by the authors of ADEPT. This real-time feedback loop allows the robot to maintain stable control and track the 6D pose of an in-hand object even when it is completely hidden from the camera (Liu et al.).
Physical Grounding and Material Properties
Vision is inherently limited because it cannot directly perceive physical forces, compliance, or weight. Visuotactile fusion provides the "physical grounding" that purely visual features lack (Yang et al.).
Consider three distinct manipulation challenges where this grounding is essential:
- Fragile Objects: In automated chemistry labs, vision-only systems frequently drop or break slippery glassware. Integrating haptic feedback allows the system to gauge the minimum force required to maintain a secure hold without crushing the equipment (Darvish et al.).
- Compliant Tools: When a robot manipulates a flexible tool, such as a spatula, the tool bends under pressure. By combining visual point clouds (which identify the spatial location of contact) with tactile force data, the robot can predict how the tool will deform, preventing damage while ensuring consistent surface interaction (Van der Merwe et al.).
- Deformable Materials: Manipulating cloth or garments is notoriously difficult due to constant self-occlusion and multi-layer ambiguities. Fusing tactile and force-torque sensors allows a manipulation policy to monitor material tension and resistance, enabling the robot to detect grasp quality and manage layers effectively (Longhini et al.; Gu et al.).
Learning Multimodal Representations
How do modern robotic policies actually digest and combine these two disparate streams of data?
Early systems often treated vision and touch as separate stages, but recent architectures fuse them into shared latent spaces. Si et al. point out that high-resolution contact data serves as a critical fine-tuning signal for delicate interactions that are underspecified by vision or object pose alone. When this dense tactile signal is integrated with global visual data, reinforcement learning policies exhibit massive performance gains—improving success rates by up to 40% over baseline behavioral cloning in high-precision bimanual assembly (Huang et al.).
Furthermore, training models on aligned visuo-tactile datasets allows the system to build internal "world models" that capture physical dynamics. By embedding tactile feedback directly into a world model's latent representation, robots can anticipate slips, collisions, and material resistance before they cause a catastrophic task failure (Lai et al.). Ultimately, direct tactile sensor input provides unique physical information that cannot be fully inferred from visual context alone (Fu et al.), making visuotactile fusion a cornerstone for truly adaptive, human-like robotic dexterity.