Interface-enhanced Musculoskeletal Hand Modeling for Contact-aware Dexterous Manipulation

Haixin Gong, Chen Zhang, Yanan SuiContact Learning @ IROS 2026 · OpenReview

TL;DR

This paper introduces an interface-enhanced musculoskeletal hand model that integrates a deformable skin layer with over 700 tactile sensing elements. By combining compliant contact mechanics with a temporal-attention encoder for tactile feedback, the framework significantly improves reinforcement learning performance on complex dexterous manipulation tasks, outperforming traditional rigid-hand models.

What Problem Does This Paper Address?

In the realm of dexterous robotic manipulation, learning robust interaction strategies remains a major hurdle. Most current reinforcement learning (RL) frameworks treat physical contact simplistically—as sparse, rigid collision events.

However, human hands rely on soft, deformable skin. This biological interface naturally distributes impact forces and provides a continuous, rich stream of spatiotemporal tactile feedback. Rigid-body simulations throw away this valuable information. To bridge this gap, the authors address a core question: Can we leverage deformable contact mechanics and dense tactile sensing to build better learning-based controllers?

Methods

The authors developed a unified framework that combines realistic musculoskeletal dynamics with a compliant, sensor-rich skin interface.

Fig. 1: Overview of the proposed framework. (a) A skeleton-adapted deformable interface is coupled to a muscle-driven hand. (b) Proprioception, retargeted motion, and temporally encoded tactile feedback support closed-loop manipulation through value-guided flow control. Contact and force maps visualize the distributed interactions.

The approach consists of two primary components:

  1. Interface-Enhanced Hand Modeling: Built upon the existing MS-Human-700 musculoskeletal model, the authors remeshed a parametric MANO hand model to ensure anatomical consistency. They designed a physics-driven coupling mechanism between the deformable skin and the underlying skeleton, resulting in over 2,000 passive degrees of freedom (DoFs). A physics-aware motion retargeting pipeline uses inverse kinematics and dynamic relaxation to ensure the skin deforms naturally during collisions rather than relying on direct actuation.
  2. Contact-Aware Manipulation Learning: The hand features more than 700 tactile sensing elements. Each element generates a "tactile token" containing its ID, surface UV coordinates, contact status, and both normal and tangential forces. A temporal-attention encoder aggregates these tokens over four consecutive frames to capture how the interaction evolves over time.

The policy is trained using reinforcement learning with value-guided flow exploration. To guide the policy smoothly, the authors introduced a contact-aware reward schedule: a sigmoid gate monitors contact activity, prioritizing reaching and imitation goals initially, and gradually ramping up object-tracking objectives once stable contact is established.

Datasets

The framework was evaluated using a simulated, partially observable Phone Use task. In this environment, the object's pose and the hand's passive surface states are intentionally withheld from the policy, forcing the controller to rely entirely on internal proprioception and tactile feedback.

Key Insights and Ideas

  • Compliance is a Natural Stabilizer: Deformable interfaces don't just provide better data; they fundamentally alter the physics of interaction. The soft skin dampens impacts and distributes forces, making the environment inherently more forgiving for an RL policy to navigate.
  • Tactile Feedback is Non-Negotiable for Complex Tasks: When tactile observations were zeroed out (simulating a hand with soft mechanics but no sensors), the policies failed to learn effectively, remaining near their initial baseline returns.
  • Hardware-Software Synergy: The combination of flexible mechanics and tactile sensing yielded a massive synergistic effect. The flexible hand significantly outperformed the rigid hand even when both were given identical sensing capabilities, proving that compliance and perception must be designed jointly.

Key Results

The authors compared four experimental conditions (flexible vs. rigid embodiments, each with or without tactile feedback) at 3 million environment steps:

  • Performance Boost: The tactile-enabled flexible hand achieved a 40.0% higher return per step than its rigid counterpart and successfully completed the full task horizon across all evaluation episodes.
  • Error Reduction: The flexible hand reduced object-position error by 60.3%, object-orientation error by 33.5%, and joint-angle root-mean-square error (RMSE) by 18.3% compared to the rigid baseline.
  • Mechanical Efficiency: The flexible interface drastically reduced physical stress. The mean total normal force was just 16.2 N for the flexible hand compared to a harsh 84.8 N for the rigid hand. Additionally, the aggregate wrist and finger muscle force was 9.5% lower in the flexible model.

Limitations and Future Work

While the results are promising, the authors note a major bottleneck: the high computational cost associated with simulating deformable physics and dense tactile arrays in real-time.

Consequently, future work will focus on optimizing simulation speeds, exploring even richer tactile representations, and tackling the "sim-to-real" challenge to validate these insights on physical robotic hardware.

Summary updated

Abstract

Dexterous manipulation requires learning robust interaction strategies under complex contact dynamics, but contact structure is rarely exploited beyond collision detection. We present an interface-enhanced musculoskeletal hand that combines deformable contact mechanics with rich tactile feedback. A skeleton-adapted surface and physics-aware motion retargeting support muscle-driven control, while temporal tactile encoding closes the sensorimotor loop. Experiments on a partially observable phone-use task show that tactile observation improves both rigid and flexible hands, with the flexible hand outperforming the rigid counterpart under matched sensing conditions. These results suggest that both deformable interfaces and tactile-rich feedback are important for contact-aware manipulation learning.
BibTeX
@inproceedings{gong2026interface,
  author = {Haixin Gong and Chen Zhang and Yanan Sui},
  title = {{Interface-enhanced Musculoskeletal Hand Modeling for Contact-aware Dexterous Manipulation}},
  booktitle = {Contact Learning @ IROS},
  year = {2026},
  url = {https://openreview.net/forum?id=PcmOsyxjKL},
}
Figure