AdaMorph: Unified Motion Retargeting via Embodiment-Aware Adaptive Transformers

Haoyu Zhang, Shibo Jin, Lvsong Li, Jun Li, Liang Lin, Xiaodong He, Zecui Zeng

TL;DR

This paper introduces AdaMorph, a unified Transformer-based framework that retargets human motion to a diverse range of robot morphologies using a single model. By decoupling semantic motion intent from robot-specific kinematics and leveraging adaptive normalization techniques, AdaMorph achieves high-fidelity motion transfer and strong generalization capabilities without requiring embodiment-specific retraining.

What Problem Does This Paper Address?

Retargeting human motion to robots is crucial for enabling robots to perform human-like behaviors, but it's a significant challenge due to the vast differences in kinematics and dynamics between humans and various robot designs. Traditional methods often require training separate, specialized models for each robot, which is inefficient and fails to leverage the common semantic structure of human motion. This "one-robot-one-model" paradigm limits scalability and adaptability. The paper aims to solve this by developing a single, unified framework that can adapt human motion to a wide array of robot morphologies.

Methods

AdaMorph tackles motion retargeting as a conditional generation task. The core ideas include:

  • Canonical Base-Frame Representation: Human motion is first converted into a standardized representation using local velocities, angular velocities, and projected gravity in a base frame. This representation disentangles local motion dynamics from global posture, serving as a common interface for different embodiments.
  • Morphology-Agnostic Intent Encoder: A Transformer encoder processes the standardized human motion and a "Dynamic Human Prompt" (derived from SMPL shape parameters) to generate a morphology-agnostic latent "intent" representation. This intent captures the semantic meaning of the motion, independent of the specific human's body shape.
  • Dual-Pathway Embodiment Prompting: To bridge the gap between the generic intent and specific robot kinematics, AdaMorph uses two prompting mechanisms:
  • Token-Level Cross-Attention: Learnable "Static Robot Prompts" are used in cross-attention layers within the decoder to inject robot-specific kinematic details.
  • Layer-Wise AdaLN Modulation: These robot prompts are also used to modulate the decoder's feature space via Adaptive Layer Normalization (AdaLN). This dynamically adjusts the generative process to match the target robot's kinematics.
  • Physics-Constrained Optimization: The training objective incorporates differentiable physics constraints. This includes feature-space reconstruction, orientation consistency (minimizing geodesic distance on SO(3)), and trajectory consistency (matching predicted displacements). A curriculum learning schedule is used to stabilize training.
  • Embodiment-Specific Output Adapters: Lightweight MLPs are used to map the shared latent representation to the robot-specific output space, handling the varying number of degrees of freedom (DoF) for different robots.

Datasets

The primary dataset used is AMASS [1], a large-scale collection of human motion capture data. To generate ground-truth robot motions for training, the authors utilized an optimization-based method called General Motion Retargeting [14]. The processed dataset consists of approximately 30 million paired motion segments from 12 distinct humanoid robot embodiments.

Key Insights and Ideas

  • Decoupling Intent from Execution: The core insight is that by separating the semantic intent of a motion from the specific kinematic execution, a single model can learn to control diverse robots. This is achieved through the morphology-agnostic encoder and the embodiment-aware prompting.
  • AdaLN for Style Transfer in Motion: Adapting AdaLN from image generation to motion retargeting is a key innovation. It allows the robot's embodiment to act as a "style" that globally modulates the motion decoder's behavior, effectively switching its generative dynamics.
  • Learned Robot Prompts Capture Topology: The learnable "Static Robot Prompts" are not just arbitrary identifiers. Analysis shows they form a structured latent space that reflects the morphological similarities between robots, indicating the model learns meaningful topological semantics.
  • Physics-Informed Training for Robustness: Incorporating differentiable physics constraints during training is crucial for ensuring physically plausible and consistent motion, preventing common issues like drift.

Key Results

  • Unified Control: AdaMorph successfully retargeted human motion to 12 distinct humanoid robot embodiments using a single trained model.
  • High-Fidelity Motion: Qualitative results (Figure 2) show robots faithfully reproducing source human behaviors, preserving dynamic essence and rhythmic alignment (Figure 4 shows high PCC scores for root velocity and whole-body activity consistency, often >0.85).
  • Zero-Shot Generalization: The model demonstrated strong performance on unseen, out-of-distribution motion sequences (ethnic dance movements), retargeting them without any fine-tuning (Figure 5).
  • Learned Representation Analysis: Cosine similarity matrices and t-SNE visualizations of robot prompts revealed clear block-diagonal structures and semantic clustering, indicating the model learns meaningful morphological relationships between robots.

Limitations and Future Work

  • Limitations: The paper acknowledges that the current framework focuses on retargeting to a specific set of humanoid robots. While it generalizes well, extending it to vastly different robot types (e.g., quadrupedal, aerial) might require further investigation. The current work is primarily focused on motion generation, not direct physics-based control in a real-world environment.
  • Future Work: The authors suggest extending the framework to physics-based control for robust sim-to-real transfer. Further exploration into handling more diverse robot morphologies and potentially incorporating real-time adaptation based on sensory feedback could also be promising directions.

Summary updated

Abstract

Retargeting human motion to heterogeneous robots is a fundamental challenge in robotics, primarily due to the severe kinematic and dynamic discrepancies between varying embodiments. Existing solutions typically resort to training embodiment-specific models, which scales poorly and fails to exploit shared motion semantics. To address this, we present AdaMorph, a unified neural retargeting framework that enables a single model to adapt human motion to diverse robot morphologies. Our approach treats retargeting as a conditional generation task. We map human motion into a morphology-agnostic latent intent space and utilize a dual-purpose prompting mechanism to condition the generation. Instead of simple input concatenation, we leverage Adaptive Layer Normalization (AdaLN) to dynamically modulate the decoder's feature space based on embodiment constraints. Furthermore, we enforce physical plausibility through a curriculum-based training objective that ensures orientation and trajectory consistency via integration. Experimental results on 12 distinct humanoid robots demonstrate that AdaMorph effectively unifies control across heterogeneous topologies, exhibiting strong zero-shot generalization to unseen complex motions while preserving the dynamic essence of the source behaviors.
BibTeX
@misc{zhang2026adamorph,
  author = {Haoyu Zhang and Shibo Jin and Lvsong Li and Jun Li and Liang Lin and Xiaodong He and Zecui Zeng},
  title = {{AdaMorph: Unified Motion Retargeting via Embodiment-Aware Adaptive Transformers}},
  year = {2026},
  eprint = {2601.07284},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.07284},
}
Figure