UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data

Dong Fang, Youjun Wu, Yuanxin Zhong, Rui Zhang, Yunlong Wang, Xiaosong Jia, Yu-Gang JiangarXiv 2026 · arXiv

TL;DR

UniDexTok introduces a unified framework for dexterous robotic hands by mapping diverse human and robot hand states into a shared 22-Degree-of-Freedom (DoF) semantic interface. By training a single, retargeting-free state tokenizer directly on real-world cross-embodiment data, it achieves sub-millimeter reconstruction accuracy and demonstrates powerful zero-shot and few-shot generalization to entirely unseen robotic hands.

What Problem Does This Paper Address?

While parallel grippers are simple enough to share a universal "open/close" state representation, dexterous robotic hands are a hardware wild west. Every manufacturer designs hands with different kinematics, joint definitions, and degrees of freedom (DoF). This hardware fragmentation means that data collected on one hand cannot easily be used to train a model for another.

To circumvent this, current Vision-Language-Action (VLA) and world models rely heavily on retargeting (mathematically warping human hand movements to fit a robot's geometry) or simulation. However, retargeting introduces geometric mismatches that alter the native state of the data, and simulation suffers from the notorious sim-to-real gap. This paper tackles the root problem: how do we build a unified, discrete state representation (a tokenizer) directly from heterogeneous, real-world dexterous hand data?

Figure 1: Overview of UniDexTok.

Methods

The authors introduce a two-part solution:

  1. Unified Dexterous Hand Model (UDHM): This is a standardized kinematic interface that maps both human and robot hand states into a shared 22-DoF vector. It assumes a rigid palm plane and models the fingers using anatomically motivated constraints (e.g., MCP abduction/adduction and flexion/extension).
Figure 2: UDHM kinematic parameterization. The model fits a palm plane, defines local motion axes, and reconstructs each finger chain with analytically defined forward kinematics.
  1. UniDexTok Tokenizer: Unlike prior methods that train separate tokenizers for each robot hand, UniDexTok shares a single encoder, Vector-Quantized (VQ) codebook, and decoder across all embodiments.
  2. Architecture: It utilizes a Diffusion Transformer (DiT)-style architecture adapted for kinematic states. To handle hand-specific variations, it uses Adaptive Layer Normalization (AdaLN) conditioned on a hand-type label.
  3. Factorized Codebook: To prevent "information collapse" (where different hand poses map to the same discrete token), the authors factorize the codebook. The encoder output is split into 8 channel groups, each quantized by a 32-entry sub-codebook, allowing the network to express $32^8$ unique token combinations using only 256 learned vectors.
UniDexTok architecture

Datasets

The model was trained and evaluated on real-world data split into two main categories:

  • Human Hand-Object Interaction Data: DexYCB, OakInk-v2, and EgoDex. Instead of retargeting, the authors mathematically reduced the standard 45-DoF MANO human hand representation to the 22 active UDHM coordinates.
  • Real-World Robot Hand Datasets: LET, Dexora, and LinkerHand (including L6, L10, and L20 variants, and the Robotera XHand1).

For evaluation of representation quality, the authors also constructed a 13-class gesture classification benchmark consisting of 130 hand-crafted real-world samples collected via a bare-hand teleoperation system.

Key Insights and Ideas

  • Human Hands as Valid Training Embodiments: Traditionally, human data is only used as a source to retarget from. This paper demonstrates that treating the human hand as just another dexterous embodiment during joint training significantly expands pose coverage and improves the reconstruction accuracy of the robot hands.
  • Semantic Insertion Beats Numerical Padding: When mapping a low-DoF robot hand to the 22-DoF UDHM space, placing the joint values into their exact semantic slots (e.g., matching the index pitch to the index pitch slot) and zero-padding the rest performs significantly better than simply appending zeros to the end of a shorter vector. This proves the model leverages joint-level semantic alignment.
  • Factorization Prevents Token Collapse: Standard single-codebook VQ-VAEs suffer from severe information collapse when forced to tokenize cross-embodiment data, often assigning identical discrete codes to visually distinct gestures. Factorizing the codebook completely mitigates this issue.

Key Results

  • Sub-Millimeter Accuracy: Compared to the recent UniHM baseline, UniDexTok reduced the Mean Per-Joint Angle Error (MPJAE) from 15.63° to 0.16° (a 98.98% reduction) and the Mean Per-Joint Position Error (MPJPE) from 18.51 mm to 0.18 mm (a 99.03% reduction).
  • Strong Generalization: When tested on an entirely unseen embodiment (the Inspire hand RH56E2), UniDexTok achieved low zero-shot joint angle errors (between 4.14° and 7.85°). Fine-tuning on just 4,528 frames (6.2% of the dataset) for 2 epochs slashed these errors below 2.08° across the board.
  • Downstream Discriminability: On the 13-class gesture benchmark, UniDexTok achieved 100% accuracy in linear probing and $k$-NN retrieval using both continuous and quantized features, whereas UniHM's performance dropped sharply to 84.62% after quantization due to token collapse.

Limitations and Future Work

  • Kinematic Assumptions: UDHM relies on a human-like kinematic structure. It may struggle to represent non-anthropomorphic grippers, soft robotic hands, or highly underactuated systems with strong mechanical couplings.
  • Hardware Constraints: The framework currently ignores hardware-specific physical properties like joint torque limits, tendon compliance, and actuator-level dynamics.
  • Lack of Environmental Context: The tokenizer focuses purely on the hand state. It does not explicitly model contact forces, tactile sensor signals, object geometry, or temporal dynamics—all of which are crucial for full closed-loop dexterous manipulation. The authors plan to extend UniDexTok to incorporate these richer object interactions in future iterations.

Summary updated

Abstract

Dexterous hands are essential for fine-grained manipulation, but their hardware designs vary substantially across embodiments. Differences in kinematics, joint definitions, and degrees of freedom make it difficult to define a shared state representation compared with parallel grippers. As a result, dexterous-hand data remains fragmented and difficult to use for joint training. In this work, we propose the Unified Dexterous Hand Model (UDHM), which maps human and robot hand states into a shared 22-DoF semantic interface. Based on UDHM, we introduce UniDexTok, a retargeting-free state tokenizer that learns embodiment-conditioned discrete tokens from standardized real joint states. UniDexTok provides a unified representation for heterogeneous dexterous hands without relying on retargeting or simulation data. Compared with the recent baseline UniHM, UniDexTok reduces MPJAE from 15.63 degrees to 0.16 degrees and MPJPE from 18.51 mm to 0.18 mm, corresponding to error reductions of 98.98% and 99.03%, respectively. These results improve reconstruction from centimeter-scale to sub-millimeter accuracy. Experiments further show that data from other embodiments improves target-embodiment reconstruction accuracy, demonstrating the benefit of cross-embodiment tokenization. UniDexTok also shows strong zero-shot and few-shot reconstruction ability when new dexterous hands are introduced.
BibTeX
@misc{fang2026unidextok,
  author = {Dong Fang and Youjun Wu and Yuanxin Zhong and Rui Zhang and Yunlong Wang and Xiaosong Jia and Yu-Gang Jiang},
  title = {{UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data}},
  year = {2026},
  eprint = {2606.10683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.10683},
}
Figure