Adaptive Visuo-Tactile Fusion with Predictive Force Attention for Dexterous Manipulation

Jinzhou Li, Tianhao Wu, Jiyao Zhang, Zeyuan Chen, Haotian Jin, Mingdong Wu, Yujun Shen, Yaodong Yang, Hao Dong

Abstract

This paper introduces AdapTac, a novel imitation learning framework for robotic dexterous manipulation that dynamically fuses visual and tactile sensory data. By using a self-supervised future force prediction task to guide a cross-attention mechanism, the system learns exactly when to rely on vision versus touch without requiring manual labeling. AdapTac achieves a 93% success rate on complex, contact-rich tasks and demonstrates robust zero-shot generalization to unseen objects.

Introduction

For robots to perform dexterous manipulation, such as flipping a sponge or opening a box, they must integrate vision for global context and touch for precise local contact. However, these sensory modalities are fundamentally heterogeneous, possessing different resolutions, sampling rates, and noise profiles. Existing fusion methods typically rely on simple concatenation, which often leads to "vision dominance," or rigid manual thresholds that fail under the fluctuating forces of real-world tasks. AdapTac addresses these challenges by enabling a policy to dynamically shift its attention between vision and touch based on physical interaction states.

Methodology

AdapTac builds upon a 3D diffusion policy architecture (RISE) and introduces two primary innovations for multimodal fusion:

  1. Force-Guided Attention Fusion: Instead of simple concatenation, a cross-attention module is employed. The physical force signal serves as the query, while visual (3D point cloud) and tactile features act as keys and values. This allows the network to weight the importance of each modality based on the robot's physical sensations.
  2. Future Force Prediction and Guidance: To prevent the model from ignoring tactile data, the authors introduce a self-supervised auxiliary task. A transformer-based diffusion head predicts the future net force based on current inputs. This predicted force is combined with the observed force to query the attention module, allowing the robot to anticipate contact and smooth the transition from reaching to manipulation.
The detailed architecture of AdapTac, illustrating the extraction of tactile and geometric features and their fusion via force-guided attention and future force prediction.

Experimental Setup

The framework was evaluated using a 7-DoF Flexiv Rizon 4 arm equipped with a 16-DoF Leap Hand and magnetic tactile sensors. The authors collected approximately 30 expert demonstrations per task via human teleoperation. Evaluation focused on three contact-rich tasks: Open Box, Reorientation (rotating a cup), and Flipping (flipping a dish sponge under heavy visual occlusion). Generalization was tested using five unseen objects with varying geometries and colors.

Results and Discussion

AdapTac achieved an average success rate of 93% across the three primary tasks, significantly outperforming vision-only (73%) and simple visuo-tactile concatenation (40%) baselines. A critical finding was the impact of the future force prediction task; removing it caused the success rate to drop from 93% to 40%, highlighting its role in preventing modality collapse.

The AdapTac framework in action, showing how the policy adaptively weights visual and tactile features throughout the stages of a manipulation task.

Visualizations of the attention weights confirm that the model prioritizes vision during the reaching phase and shifts weight to tactile features the moment contact begins. Furthermore, the policy maintained a 75% success rate on unseen objects, proving that it learns generalizable manipulation skills rather than object-specific memorization.

Conclusion

AdapTac demonstrates that dynamic, force-guided attention is a robust and scalable alternative to manual thresholding for multimodal robotic manipulation. By explicitly reinforcing the value of tactile data through future force prediction, the system achieves high performance in contact-rich environments. Future work will explore combining this imitation learning approach with reinforcement learning to further enhance robustness and error recovery.

Summary updated

Abstract

Effectively utilizing multi-sensory data is important for robots to generalize across diverse tasks. However, the heterogeneous nature of these modalities makes fusion challenging. Existing methods propose strategies to obtain comprehensively fused features but often ignore the fact that each modality requires different levels of attention at different manipulation stages. To address this, we propose a force-guided attention fusion module that adaptively adjusts the weights of visual and tactile features without human labeling. We also introduce a self-supervised future force prediction auxiliary task to reinforce the tactile modality, improve data imbalance, and encourage proper adjustment. Our method achieves an average success rate of 93% across three fine-grained, contactrich tasks in real-world experiments. Further analysis shows that our policy appropriately adjusts attention to each modality at different manipulation stages. The videos can be viewed at https://adaptac-dex.github.io/.
BibTeX
@inproceedings{li2025adaptive,
  author = {Jinzhou Li and Tianhao Wu and Jiyao Zhang and Zeyuan Chen and Haotian Jin and Mingdong Wu and Yujun Shen and Yaodong Yang and Hao Dong},
  title = {{Adaptive Visuo-Tactile Fusion with Predictive Force Attention for Dexterous Manipulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246476},
}
Figure