TL;DR
DexFormer 2.0 is a single reinforcement learning policy capable of controlling 24 different arm-hand combinations for dexterous grasping. By mapping diverse robots to a canonical joint interface and using morphology-aware attention, it achieves zero-shot transfer to unseen robot embodiments and even parallel grippers, marking a significant step toward a "foundation policy" for robotic manipulation.
What Problem Does This Paper Address?
In traditional robotics, a control policy is typically a "specialist"—trained for one specific arm paired with one specific hand. If you swap the arm or change the hand, the kinematic chain, workspace, and joint dynamics change so fundamentally that the old policy becomes useless.
This specialization makes scaling incredibly expensive. As hardware diversifies, the number of required policies grows with the Cartesian product of available arms and hands. The core challenge is heterogeneity: different robots have different numbers of joints, varying axes of rotation, and unique physical limits. They lack a common "language" for observations and actions. DexFormer 2.0 asks: Can we build a single "generalist" policy that controls many combinations and generalizes to hardware it has never seen?
Methods
The authors introduce a morphology-aware recurrent policy architecture that treats robot joints as flexible tokens rather than a fixed vector.
- Canonical Joint Interface: Every robot is mapped to a set of 29 "slots" (7 for the arm, 22 for the hand). If a robot has fewer joints, "validity masks" tell the network which slots to ignore.
- Joint-Level Morphology Descriptors: Each joint is described by a 16-dimensional vector encoding its physical properties: its axis, attachment position, joint/velocity limits, semantic role (e.g., "thumb"), and its depth in the kinematic tree.
- MorphNode Encoder: This module uses MorphAttention, a spatial attention mechanism biased by the robot’s kinematic tree. It calculates "tree distances" between joints, allowing the model to understand that a finger joint is physically more related to its neighbor than to a joint on the shoulder.
- Temporal Context (Transformer-XL): Morphology alone doesn't tell the whole story—dynamics (how a joint actually responds to a command) must be inferred. The model maintains a 16-step memory to "feel out" the specific dynamics of the current embodiment during interaction.
- Shared Actor Head: A single network produces actions across all pairings, masked to ensure it only sends commands to active joints.
Datasets
- Embodiments: The study uses 4 robot arms (Flexiv, Franka, KUKA, xArm7) and 6 dexterous hands (Revo3, Sharpa, LEAP, Allegro, RAPID, Shadow), creating 24 possible combinations.
- Training/Test Split: 18 combinations were used for training; 6 were held out entirely to test zero-shot generalization.
- Objects: Training was conducted on 10 primitive shapes (cuboids, spheres, cones). Evaluation included 10 unseen objects from the YCB dataset (e.g., a soup can, an apple) to test object generalization.
- Zero-Shot Transfer: The policy was also tested on 4 entirely new hand types and 4 parallel grippers (via virtual-hand retargeting) that never appeared in the training set.
Key Insights and Ideas
- Memory is Mandatory: One of the biggest "aha" moments was that static morphology isn't enough for transfer. Ablations showed that removing the Transformer-XL memory halved the success rate on held-out pairs. The policy needs to "remember" the last few frames to infer the specific dynamics and contact responses of a new hand.
- The Kinematic Bias Matters: Treating the robot as a collection of joints is okay, but treating it as a tree is better. By biasing attention based on the kinematic structure, the model learns the physical relationships between limbs more effectively.
- Pairing Coverage over Samples: The researchers found that training on 18 combinations was vastly superior to training on 12, even when the total number of interaction samples was the same. Diversity in "pairing coverage" is the key to unlocking generalization.
Key Results
- High Success Rates: The policy achieved 89.01% stable success on seen arm-hand combinations and a impressive 51.18% on held-out (unseen) pairs.
- Object Generalization: When faced with both unseen robots and unseen YCB objects, the policy maintained a 48.85% success rate.
- Zero-Shot Hand Transfer: Without any fine-tuning, the policy could grasp using new hands like the Wuji 1 (27.18% mean SR) and even parallel grippers.
- Sim-to-Real: The authors distilled the simulation policy into a vision-based "student" and deployed it on a real Flexiv arm. It achieved a 68% success rate on a held-out arm-hand pair (Flexiv + Revo3) and 92% on a training pair (Flexiv + LEAP).
Limitations and Future Work
- The Generalization Gap: While 51% success on unseen robots is a breakthrough, it is still significantly lower than the 89% achieved on seen robots, suggesting there is still room to improve how we encode morphology.
- Privileged Information: The core policy relies on "object state" (exact positions and point clouds), which isn't directly available in the real world. This necessitates a distillation step to a vision-based student for deployment.
- Task Complexity: The current work focuses on the "reach and grasp" phase. Future iterations could expand this to complex in-hand manipulation or long-horizon tool use across different embodiments.
Summary updated