VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation

Zhengnan Sun, Zhaotai Shi, Jiayin Chen, Qingtao Liu, Yu Cui, Jiming Chen, Qi Ye

Introduction

Bimanual dexterous manipulation—the ability to use two multi-fingered hands to manipulate objects—represents a significant milestone in robotics. Unlike single-hand tasks, bimanual coordination requires managing high-dimensional joint spaces while maintaining object stability. Researchers have introduced VTAO-BiManip, a framework that leverages masked multimodal pre-training across Vision, Tactile, Action, and Object state (VTAO) to master complex tasks like unscrewing a bottle cap.

Figure 1: The bimanual bottle-cap turning task across human demonstrations, simulation, and real-world robot deployment.

The Challenges of Bimanual Coordination

The paper identifies three primary hurdles in bimanual dexterous manipulation:

  1. Kinematic Coordination: Synchronizing over 30 joints across two hands to maintain stability.
  2. Object-State Decoupling: The "passive" hand must dynamically adjust to the object's pose changes caused by the "active" hand's manipulation.
  3. Sub-skill Composability: Sequencing complex actions like grasping, holding, and rotating between two hands in a perfectly timed flow.

The VTAO-BiManip Framework

The core of the system is a Masked Autoencoder (MAE) architecture pre-trained on human demonstration data. The model processes four distinct data streams (VTAO):

  • Vision (V): RGB visual input.
  • Tactile (T): Binarized signals from tactile gloves.
  • Action (A): Hand joint angles, treated as an input modality to learn inter-hand synergies.
  • Object (O): Explicit 6D pose and 3D geometry of the object.
Figure 2: The hardware setup and the four modalities (Vision, Tactile, Action, Object) used for multimodal pre-training.

By forcing the model to reconstruct masked portions of these inputs and predict future action sequences, VTAO-BiManip learns a latent representation that captures the underlying physics and coordination required for the task. Treating action as an input modality allows the robot to anticipate movements between the active and passive hands.

Curriculum Reinforcement Learning

To bridge the gap between pre-training and execution, the authors employ a two-stage curriculum reinforcement learning (RL) approach:

  • Stage 1 (Sub-skill Learning): Focuses on basic grasping and twisting with a fixed object.
  • Stage 2 (Collaborative Learning): Introduces full collaborative learning where the object is free-moving, requiring active stabilization from both hands to keep the bottle upright while unscrewing the cap.

Key Results and Impact

VTAO-BiManip achieves a 63% success rate on seen bottles and 51% on unseen bottles, a 20% improvement over previous state-of-the-art methods. Ablation studies confirm that including "Object" reconstruction and "Action" as an input modality are critical for success. The model's effectiveness was further validated through successful sim-to-real transfer on a physical platform using dual Leap Hands and a Unitree Z1 arm.

Conclusion

VTAO-BiManip demonstrates that multimodal pre-training, particularly when incorporating object geometry and action sequences, provides a robust foundation for complex bimanual tasks. While challenges remain in scaling datasets and handling extreme object variations, this work paves the way for more capable and versatile robotic assistants in unconstrained real-world scenarios.

Summary updated

Abstract

Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination. Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills. In this paper, we introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation. We improve prior learning by incorporating hand motion data, providing more effective guidance for dual-hand coordination than binary tactile feedback. Our pretraining model predicts future actions as well as object pose and size using masked multimodal inputs, facilitating cross-modal regularization. To address the multi-skill learning challenge, we introduce a two-stage curriculum RL approach to stabilize training. We evaluate our method on a bottle-cap unscrewing task, demonstrating its effectiveness in both simulated and real-world environments. Our approach achieves a success rate that surpasses existing visual-tactile pretraining methods by over 20%.
BibTeX
@inproceedings{sun2025vtao,
  author = {Zhengnan Sun and Zhaotai Shi and Jiayin Chen and Qingtao Liu and Yu Cui and Jiming Chen and Qi Ye},
  title = {{VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
}
Figure