M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking

Zuxing Lu, Ziang Zheng, Yao Lyu, Jingyu Liu, Feihong Zhang, Song Lu, Xin Yuan, Changyin Sun, Xingxing Zuo, Shengbo Eben Li

TL;DR

This paper introduces M3imic, a novel framework for humanoid robots that learns a single, versatile controller capable of mimicking diverse human motions from various input types (joint angles, human poses, end-effector poses). By unifying these heterogeneous references into a shared latent space, M3imic achieves impressive sim-to-real transfer and robust performance across different tasks without task-specific retraining.

What Problem Does This Paper Address?

Humanoid robots are envisioned to assist humans in a wide array of tasks, from walking and running to complex manipulation. To achieve this, they need to be able to accurately mimic human movements. However, human motion data comes in various forms: dense robot joint trajectories, full-body human poses, and sparser end-effector poses. Existing methods struggle to handle this "multi-modal" nature of motion references, often requiring separate controllers for different input types or complex pre-processing steps like inverse kinematics. This mismatch in representation and granularity poses a significant challenge for building a truly versatile and general-purpose humanoid controller.

Methods

M3imic tackles this challenge with a unified, end-to-end reinforcement learning framework. The core innovation lies in its multi-modal command encoder, which uses modality-specific encoders to map diverse motion references (robot joint angles, human body poses, and end-effector poses) into a shared, low-dimensional latent space. This shared representation allows a single policy to learn from and act upon these different modalities.

Key methodological components include:

  • Unified Latent Space: Instead of relying on explicit conversions (like inverse kinematics), M3imic learns an implicit representation that bridges the gap between different motion modalities.
  • Asymmetric Actor-Critic Architecture: This design improves learning efficiency by allowing the actor (policy) to use partial observations while the critic (value function) has access to more privileged information from the simulator.
  • Multi-Modal Autoencoder Loss: Beyond reinforcement learning, the framework incorporates an autoencoder loss to ensure that the latent representations from different modalities are consistent and can reconstruct the original inputs.
  • Curriculum Learning Strategy: To handle the complexity of large-scale, diverse motion datasets, M3imic employs an adaptive sampling strategy that gradually shifts focus from easier to more difficult motion segments, stabilizing training.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Training Datasets: LAFAN1 and 100STYLE are used for training, providing diverse and dynamic human motions.
  • Test Dataset: The OMOMO dataset is used for evaluation, featuring complex manipulation tasks.

Key Insights and Ideas

  • Unified Representation is Key: The most significant insight is that a single, shared latent space can effectively bridge the gap between fundamentally different motion representations (dense joint angles vs. sparse end-effector poses), enabling a versatile controller.
  • Modality Trade-offs: The research highlights a fidelity-robustness trade-off between different input modalities. Dense robot joint inputs lead to higher tracking fidelity but less flexibility, while sparse end-effector inputs offer better robustness to distribution shifts.
  • Curriculum Learning for Stability: Adaptive sampling based on segment difficulty is crucial for stabilizing training on large, heterogeneous motion datasets, preventing the policy from getting stuck on easier motions.
  • Sim-to-Real Generalization: The framework demonstrates exceptional generalization capabilities, achieving high success rates on unseen real-world tasks without any task-specific fine-tuning.

Key Results

M3imic achieves state-of-the-art performance across various metrics:

  • Simulation: A peak success rate of 98.42% on the unseen OMOMO test dataset.
  • Tracking Accuracy: Outperforms strong baselines like TWIST2 and ExBody2 in terms of root-relative mean per-keypoint position error (Empkpe) and mean per-joint angle error (Empjae). For instance, it achieved 46.05 mm Empkpe on the training data, a significant reduction compared to baselines.
  • Real-World Deployment: Demonstrates robust performance in real-world experiments on the Unitree G1 robot, successfully executing diverse motions like dancing, running, and walking, as well as teleoperation tasks. The end-effector policy achieved comparable performance to the robot-joint policy in real-world scenarios, with Empkpe of 43.22 mm.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the end-effector policy offers robustness, there's a trade-off with tracking fidelity. The current work primarily focuses on robustness to input domain differences rather than entirely unseen motion categories.
  • Future Work: The authors propose integrating high-level perception and planning with the learned versatile controller to form a complete hierarchical control framework. They also suggest exploring more diverse datasets to further enhance generalization.

Summary updated

BibTeX
@article{lu2026m,
  author = {Zuxing Lu and Ziang Zheng and Yao Lyu and Jingyu Liu and Feihong Zhang and Song Lu and Xin Yuan and Changyin Sun and Xingxing Zuo and Shengbo Eben Li},
  title = {{M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking}},
  journal = {IEEE Robotics and Automation Letters},
  year = {2026},
  url = {https://openreview.net/forum?id=F0lJv4dNNh},
}
Figure