Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots

Alan FernOpenReview 2026 · OpenReview

TL;DR

This paper introduces the Masked Humanoid Controller (MHC), a novel reinforcement learning-based controller that unifies diverse humanoid robot behaviors (walking, standing, mimicry) under a single, flexible interface. By allowing users to specify target trajectories for only parts of the robot's state, the MHC enables intuitive control via various inputs like joystick commands or motion capture, demonstrating robust performance on both simulation and the real-world Digit V3 robot.

What Problem Does This Paper Address?

Controlling humanoid robots for complex, real-world tasks is incredibly challenging. Current approaches often rely on separate, meticulously engineered controllers for distinct skills like walking, manipulation, or mimicking human motion. This fragmentation leads to a lack of versatility, making it difficult to combine skills or adapt to new tasks without significant re-engineering. A key hurdle is the absence of a unified way to command these diverse behaviors. High-level systems might want to issue commands through joystick inputs, motion capture data, or even video demonstrations, but integrating these disparate inputs into a single, coherent whole-body control strategy is a major open problem. This paper aims to bridge this gap by developing a single controller capable of understanding and executing a wide range of partially specified commands.

Methods

The core innovation is the Masked Humanoid Controller (MHC). Instead of requiring a complete specification of the robot's future state, the MHC accepts "masked target trajectories." This means users can specify desired movements for only certain parts of the robot (e.g., just the torso's velocity or a specific arm's trajectory) while leaving other parts unspecified. The MHC then learns to "fill in the blanks" to generate a complete, stable, and coherent whole-body motion.

The MHC is trained using reinforcement learning (RL) in simulation (MuJoCo) on a Digit V3 humanoid model. Key methodological components include:

  • Motion Data Generation: A diverse dataset of human motions is created by retargeting motion capture data (AMASS, Reallusion), video demonstrations, and optimized trajectories to the Digit's kinematics using Inverse Kinematics (IK).
  • Unified Interface: The masked trajectory input allows for flexible command modalities, accommodating joystick inputs (partial body control), motion capture (full body mimicry), and combinations thereof.
  • Curriculum Learning: Training progresses through three stages:
  • Locomotion: Focuses on basic balance and walking with partial directives (root motion only).
  • Stability: Enhances robustness to perturbations with continuous forces.
  • Whole-Body Directives: Trains the MHC to handle both fully and partially specified directives, integrating locomotion with upper-body control.
  • Domain Randomization: Physical parameters of the simulation are randomized during training to improve sim-to-real transfer.
  • Reward Design: A carefully crafted reward function encourages command following, natural body postures, smooth joint motion, and crucially, tracks the unmasked parts of the directive.

Datasets

  • Training Data: A combined dataset of human motion capture (AMASS, Reallusion), video demonstrations, and optimized trajectories. This data is retargeted to the Digit V3 humanoid's kinematic model.
  • Evaluation Data: The paper defines three sets (setA, setB, setC) derived from the training data, used to evaluate generalization capabilities. These sets vary in their composition of motion sources and difficulty.

Key Insights and Ideas

  • The Power of Masking: The core insight is that specifying what not to control is as important as specifying what to control. This masking mechanism elegantly unifies disparate control inputs into a single framework.
  • Curriculum is Crucial: The staged curriculum is vital for learning complex behaviors. Starting with basic locomotion and gradually introducing perturbations and full-body directives leads to more stable and robust learning than training on all tasks simultaneously.
  • Unified Controller for Versatility: A single learned controller can indeed handle a wide spectrum of tasks, from simple walking to complex loco-manipulation, eliminating the need for complex switching logic between specialized controllers.
  • Sim-to-Real Transfer is Achievable: Despite the inherent challenges, the MHC demonstrates promising sim-to-real transfer capabilities, performing real-world tasks like boxing and box manipulation.

Key Results

  • Locomotion Performance: The MHC shows competitive performance against a state-of-the-art standing and walking controller (SaW) in terms of root drift under various forces and walking speeds.
  • Mimicry Performance: The MHC significantly outperforms baseline controllers (OffsetBlind, OffsetAware) in whole-body imitation tasks, achieving a substantially lower failure rate (0.45% vs. 4.8% and 49.2%).
  • Generalization: Models trained on different subsets of the data (setA, setB, setC) show good generalization to unseen motions, with performance generally improving as the training dataset size increases.
  • Real-World Demonstration: The MHC successfully executed tasks on the Digit V3 robot, including locomotion directives, boxing, and box manipulation, with a failure rate of 5% and a mean per-joint position error of 0.117m.

Limitations and Future Work

  • Sim-to-Real Gaps: The authors acknowledge remaining sim-to-real discrepancies, particularly for motions requiring wide foot placements or extended single-leg balancing. Incorporating more real-world data into the training process is suggested.
  • Simulation Bottlenecks: The reliance on MuJoCo limits the scale and speed of training. Exploring higher-throughput simulators like Isaac Sim (once Digit V3 is supported) is a future direction.
  • Masking Variety: Expanding the variety of masking patterns used during training could further enhance versatility.
  • Inconsistent Inputs: The current framework does not handle inconsistent or impossible input directives (e.g., conflicting root velocity and joint commands), which is a target for future work.

Summary updated

Abstract

A major challenge in humanoid robotics is designing a unified interface for commanding diverse whole-body behaviors, from precise footstep sequences to partial-body mimicry and joystick teleoperation. We introduce the Masked Humanoid Controller (MHC), a learned whole-body controller that exposes a simple yet expressive interface: the specification of masked target trajectories over selected subsets of the robot’s state variables. This unified abstraction allows high-level systems to issue commands in a flexible format that accommodates multi-modal inputs such as optimized trajectories, motion capture clips, re-targeted video, and real-time joystick signals. The MHC is trained in simulation using a curriculum that spans this full range of modalities, enabling robust execution of partially specified behaviors while maintaining balance and disturbance rejection. We demonstrate the MHC both in simulation and on the real-world Digit V3 humanoid, showing that a single learned controller is capable of executing such diverse whole-body commands in the real world through a common representational interface.
BibTeX
@misc{fern2026learning,
  author = {Alan Fern},
  title = {{Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots}},
  year = {2026},
  url = {https://openreview.net/forum?id=4SyO68HJqM},
  howpublished = {OpenReview},
}
Figure