DexHOT: Learning Hand-Object Configuration Transfer for Dexterous Manipulation

Zhewei Gui, Anthony Bolton, Jiyao Zhang, Zeyuan Chen, Qiyang Yan, Tianyu Wang, Hao Dong

Dexterous robot hands are marvels of engineering, but controlling them remains a massive challenge. Most current systems focus on moving an object to a target pose, letting the hand's configuration emerge as a byproduct. But as any human knows, how you hold a tool is just as important as where you move it.

TL;DR

DexHOT (Hand-Object Configuration Transfer) shifts the focus of dexterous manipulation from simple object-pose goals to explicit "hand-object configurations." By learning to transfer between stable, nearby configurations, a single policy can generalize to over 100 diverse objects and perform complex, multi-stage tasks like pouring water or driving screws on a real robot.

DexHOT learns a single policy to transfer between configurations that jointly specify hand articulation and object pose.

What Problem Does This Paper Address?

The core challenge in dexterous manipulation is that object-state goals (e.g., "rotate this mug 90 degrees") are underspecified. You can hold a mug by the rim, the handle, or the body; while the object's pose might be the same, each grasp offers different stability and enables different future actions.

Existing learning-based systems often leave these hand-object relations implicit. This makes long-horizon tasks—like picking up a tool and then reorienting it for use—extremely difficult because the initial grasp might not support the subsequent motion. DexHOT addresses this by making the joint configuration of the hand and object the explicit target of the control policy.

Methods

The authors introduce a framework that decomposes complex manipulation into a sequence of "local transfers."

  1. Feasible Configuration Synthesis: Using a tool called BODex, they generate a massive pool of over 1.1 million physically valid hand-object configurations across 120 different objects.
  2. Learning Local Transfers: Instead of training a policy to reach a distant goal in one go, they train a single recurrent policy to move between nearby configurations (within 20-40mm of each other). This "transfer primitive" is trained using Evolutionary Policy Optimization (EPO), a population-based reinforcement learning method that helps the agent discover diverse strategies like finger gaiting and pivoting.
  3. Hierarchical Chaining: To perform long-range tasks, a high-level planner (either a greedy search or a human demonstration) provides a sequence of configuration subgoals at 1Hz. The low-level DexHOT policy then tracks these subgoals at 30Hz in a closed loop.
The DexHOT pipeline: from grasp synthesis and local goal sampling to hierarchical chaining for complex tasks.

Datasets

The researchers introduced DexHOTBench, a comprehensive benchmark for evaluating hand-object configuration transfer.

  • Scale: 120 objects in total.
  • Splits: 40 objects for training, 40 unseen instances from training categories, and 40 objects from entirely unseen categories.
  • Diversity: Objects range from simple geometric shapes to complex real-world items like cameras, bananas, and power drills.
DexHOTBench includes 120 objects across seen and unseen categories to test generalization.

Key Insights and Ideas

  • Explicit is Better than Implicit: By explicitly targeting hand-object configurations, complex behaviors like finger gaiting (walking fingers across an object) and pivoting emerge naturally without needing specific reward functions for those skills.
  • Local to Global: The "aha moment" is that complex, long-horizon manipulation can be effectively modeled as a chain of simple, local transfers. This makes the learning problem much more tractable.
  • Population-Based Exploration is Crucial: Standard RL algorithms (like PPO) struggled to cover the vast space of possible hand-object relations. Using EPO allowed the system to maintain a diverse set of "exploration blocks," leading to much higher success rates.
  • Geometric Representation: Representing the configuration via 28 hand keypoints in the object's frame provides a robust, coordinate-invariant way to couple hand and object states.

Key Results

DexHOT demonstrates impressive generalization and real-world robustness:

  • Simulation: A single policy achieved an 80.5% success rate on seen objects and 69.4% on completely unseen object categories.
  • Real-Robot Transfer: Without any fine-tuning, the policy achieved 61.5% success across 200 transfers on a real Sharpa Wave hand.
  • Long-Horizon Tasks: When guided by human demonstrations, DexHOT achieved a 90% success rate across six challenging tasks, including barcode scanning, water pouring, and screw driving. It significantly outperformed baselines that relied only on object-pose goals.

Limitations and Future Work

While a major step forward, the authors note several areas for improvement:

  • Slippage: On very low-friction objects, the system can lose its grip before the closed-loop policy can compensate. Integrating tactile sensing could help.
  • Initialization: The system sometimes fails if the initial grasp (reproduced from a synthesized pose) is marginally unstable on the physical hardware.
  • Perception: Occasional tracking failures from the vision system (FoundationPose) can lead to task failure.
  • Future Directions: The team suggests exploring richer shape representations and improved dynamics modeling to handle more complex geometries and more agile maneuvers.

Summary updated

Abstract

Dexterous manipulation involves not only moving objects, but also arranging the hand around them to maintain useful contacts and enable subsequent actions. Yet many learning-based systems specify goals primarily through object poses or trajectories, leaving hand-object configurations to emerge implicitly. We introduce DexHOT, which formulates dexterous manipulation as hand-object configuration transfer: moving between feasible configurations that jointly specify hand articulation and object pose relative to the wrist. DexHOT learns a reusable relation-conditioned transfer primitive from synthetic feasible configuration pairs, without human reference motions, and composes long-horizon behaviors through closed-loop tracking of configuration subgoals. On DexHOTBench with 120 objects, a single policy achieves 80.5% transfer success on seen objects and 69.4% on unseen objects; on a real dexterous hand, it achieves 61.5% success over 200 real-robot transfers and 90% task success across six demonstration-guided tasks. These results show that explicit hand-object goals provide an expressive and composable interface for dexterous manipulation. More details and videos are available at https://dex-hot.github.io/.
BibTeX
@misc{gui2026dexhot,
  author = {Zhewei Gui and Anthony Bolton and Jiyao Zhang and Zeyuan Chen and Qiyang Yan and Tianyu Wang and Hao Dong},
  title = {{DexHOT: Learning Hand-Object Configuration Transfer for Dexterous Manipulation}},
  year = {2026},
  eprint = {2611.05119},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05119},
}
Figure