MoDex: Planning High-Dimensional Dexterous Control via Learning Neural Hand Models

Wenbo Ding (丁文伯), Tong Wu, Shoujie Li, Chuqiao Lyu, Kit Wa Sou, Wang Sing Chan, Wenbo DingarXiv 2024 · DOI · arXiv

Abstract

Controlling high-degree-of-freedom (DoF) robotic hands remains a significant challenge due to the "curse of dimensionality." Traditional reinforcement learning methods often struggle with data efficiency and generalization because they couple the robot's dynamics with the environment. This paper introduces MoDex, a framework that mimics the biological principle of internal models by learning decoupled hand dynamics. By training a pair of neural networks—forward and inverse models—through random exploration, MoDex enables efficient bidirectional planning. When integrated with Large Language Models (LLMs) for heuristic generation, MoDex achieves state-of-the-art performance in dexterous tasks, demonstrating up to 100x better data efficiency than traditional model-free baselines.

Introduction

Robotic hands with high DoF, such as the 24-DoF Shadowhand or the 39-DoF Myohand, offer human-like dexterity but are notoriously difficult to control. Standard Deep Reinforcement Learning (DRL) approaches treat the robot and its environment as a single complex system, requiring an astronomical number of trials to learn even basic movements.

The MoDex framework overview, showing the transition from exploration to diverse dexterous tasks including reaching, gesture generation, and in-hand manipulation.

In contrast, neuroscience suggests that humans utilize an internal model—a mental representation of the body's dynamics—to predict limb movement and determine necessary motor commands. MoDex adopts this biological advantage, learning a task-agnostic "hand model" that can be "plugged in" to various external environments and tasks without retraining from scratch.

Methodology

The MoDex architecture is built upon three core components:

1. Neural Internal Models

MoDex begins by learning the hand's intrinsic physics through task-free random exploration. This results in two complementary models:

  • Forward Model ($f_\theta$): Predicts future fingertip states based on the current state and action.
  • Inverse Model ($g_\phi$): Maps a desired target state to a distribution of possible actions. Modeling this as a Gaussian distribution accounts for the redundancy in high-DoF systems where multiple joint configurations can achieve the same fingertip position.

2. Bidirectional Planning

Rather than relying on random shooting for trajectory optimization, MoDex employs a bidirectional strategy. The Inverse Model provides a high-quality initial "warm start" for action sequences, which is then iteratively refined by a Cross-Entropy Method (CEM) planner using the Forward Model. This approach significantly accelerates convergence compared to standard model-based planning.

The MoDex pipeline: from exploration to the construction of Neural Internal Models (Forward and Inverse), and their application in in-hand manipulation and LLM-driven gesture generation.

3. Factorized Dynamics and LLM Integration

MoDex factorizes system dynamics into a fixed hand model and a task-specific external model (e.g., object dynamics). For high-level task specification, MoDex leverages Large Language Models (LLMs). By prompting an LLM (like GPT-4) to generate Python-based cost functions for specific gestures (e.g., "Rock & Roll"), the system can optimize low-level motor control to satisfy high-level semantic requests, bridging the gap between language and robotics.

Experimental Results

The framework was evaluated across diverse simulated environments (IsaacGym, Myosuite) and on real-world hardware (Xhand).

  • Data Efficiency: In fingertip reaching tasks, MoDex achieved superior performance using only 1% to 10% of the data required by model-free methods like Soft Actor-Critic (SAC).
  • High-DoF Success: On the 39-DoF Myohand—the most complex model tested—MoDex reached an 88% success rate in sequential reaching, whereas traditional synergy-based baselines failed entirely.
  • Real-World Robustness: The system successfully performed in-hand rotation of various objects, including cubes, cylinders, and irregular shapes like an orange, within just 30 minutes of real-world rollout.
  • Zero-Shot Transfer: Internal models learned for simple reaching were successfully applied to complex object manipulation without needing task-specific retraining.

Discussion and Limitations

While MoDex represents a leap in data-efficient dexterous control, several challenges remain. The current implementation relies on precise state tracking (e.g., joint positions or AprilTags), which is difficult to maintain in unstructured real-world environments. Furthermore, the use of simple MLP architectures for dynamics might be superseded by more expressive models, such as Diffusion Models, to better capture the multi-modal nature of contact physics. Future work will also explore the integration of tactile feedback to enhance the stability of in-hand manipulation for soft or slippery objects.

Summary updated

Abstract

Controlling hands in high-dimensional action space has been a longstanding challenge, yet humans naturally perform dexterous tasks with ease. In this paper, we draw inspiration from the concept of internal model exhibited in human behavior and reconsider dexterous hands as learnable systems. Specifically, we introduce MoDex, a framework that includes a couple of neural networks (NNs) capturing the dynamical characteristics of hands and a bidirectional planning approach, which demonstrates both training and planning efficiency. To show the versatility of MoDex, we further integrate it with an external model to manipulate in-hand objects and a large language model (LLM) to generate various gestures in both simulation and real world. Extensive experiments on different dexterous hands address the data efficiency in learning a new task and the transferability between different tasks.
BibTeX
@misc{2024modex,
  author = {Wenbo Ding (丁文伯) and Tong Wu and Shoujie Li and Chuqiao Lyu and Kit Wa Sou and Wang Sing Chan and Wenbo Ding},
  title = {{MoDex: Planning High-Dimensional Dexterous Control via Learning Neural Hand Models}},
  year = {2024},
  eprint = {2409.10983},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.10983},
}
Figure