Hierarchical World Models as Visual Whole-Body Humanoid Controllers

Nicklas Hansen, Jyothir S V, Vlad Sobal, Yann LeCun, Xiaolong Wang, Hao Su

TL;DR

Puppeteer is a hierarchical world model that enables 56-DoF humanoids to perform complex tasks like jumping over gaps using only visual observations. By decoupling low-level physics from high-level task planning through a MoCap-pretrained tracking agent, it achieves state-of-the-art performance while producing remarkably natural, human-like motions that are preferred by humans 97% of the time.

What Problem Does This Paper Address?

Whole-body humanoid control is often considered a "grand challenge" in robotics. It requires managing a high-dimensional action space (56 joints in this study) while maintaining stability in a bipedal morphology that is inherently prone to falling.

When you add visual perception to the mix—requiring the agent to perceive terrain and obstacles from pixels while simultaneously coordinating its limbs—the problem becomes exponentially harder. Traditional Reinforcement Learning (RL) often results in "reward hacking," where agents find mathematically optimal but physically nonsensical ways to move, such as "rolling" or sliding across the floor. To avoid this, researchers usually rely on simplifying assumptions like privileged information (exact coordinates of obstacles) or pre-defined "skill primitives." The authors of this paper sought a purely data-driven approach that produces natural movement without manual engineering.

Methods

The authors propose Puppeteer, a two-stage hierarchical framework built on the TD-MPC2 algorithm. TD-MPC2 is a model-based RL method that learns a latent world model and selects actions by planning (using Model Predictive Control) within that model.

The hierarchy consists of two distinct agents:

  1. Low-level Tracking Agent: This agent is pretrained on a large dataset of human Motion Capture (MoCap) data. It learns to map "commands"—specifically, the desired 3D positions of the head, hands, and feet—into joint torques. This effectively bakes "human-like" physics into the controller.
  2. High-level Puppeteer Agent: This agent is trained on specific downstream tasks (e.g., navigating a corridor with hurdles). It takes RGB camera images and proprioceptive data as input and outputs the 3D end-effector targets for the tracking agent to follow.
Figure 2. Approach. We pretrain a tracking agent (world model) on human MoCap data using RL; this agent takes proprioceptive information qt and an abstract reference motion (command) ct as input, and synthesizes H low-level actions that tracks the reference motion. We then train a high- level puppeteering agent on downstream tasks via online interaction; this agent takes both state qt and visual information vt as input, and outputs commands for the tracking agent to execute.

A key innovation here is the interface: instead of using abstract embeddings, the high-level agent outputs geometric locations. This makes the low-level tracking agent highly reusable across entirely different tasks.

Datasets

  • Pretraining: The authors used the CMU MoCap dataset (via the MoCapAct subset), which includes over 800 clips of human movement retargeted to a 56-DoF humanoid model.
  • Evaluation: The paper introduces a new, challenging benchmark suite of 8 tasks for a 56-DoF humanoid:
  • Visual Tasks: Corridor, Hurdles, Walls, Gaps, and Stairs.
  • Non-visual Tasks: Stand, Walk, and Run.
The 8 tasks in the proposed benchmark suite.

Key Insights and Ideas

  • MoCap as a Structural Prior: Pretraining on MoCap doesn't just teach the robot how to move; it constrains the solution space. By forcing the agent to move through an end-effector interface learned from humans, the "unnatural" solutions (like rolling) are effectively filtered out.
  • The Necessity of Planning: The authors found that planning at both levels of the hierarchy is critical. Model-free methods (like SAC) or models that don't use online planning (like DreamerV3) failed to achieve meaningful performance on the visual tasks.
  • Zero-Shot Generalization: Because the model learns a robust understanding of physics and geometry, it can generalize to environments it never saw during training. For example, an agent trained on 0.4m gaps could successfully navigate gaps up to 1.2m wide without further training.

Key Results

  • Performance: Puppeteer matches or exceeds the performance of flat, non-hierarchical state-of-the-art models (TD-MPC2) across most tasks, while being significantly more sample-efficient than prior hierarchical methods.
  • Naturalness: In a user study with 51 participants, 97.8% of responses preferred Puppeteer’s motions over the baseline. While the baseline TD-MPC2 often learned to "roll" up stairs to maximize speed, Puppeteer learned a natural walking gait.
  • Quantitative Metrics: Puppeteer maintained a human-like torso height (approx. 96cm) compared to the baseline (approx. 86cm), which tended to crouch or deform its posture to optimize for reward.
Figure 1. Visual whole-body control for humanoids. We present Puppeteer, a hierarchical world model for humanoid control with visual observations. Our method produces natural and human-like motions without any reward design or skill primitives, and traverses challenging terrain.

Limitations and Future Work

  • Visio-Locomotive Focus: The current tasks are primarily about moving through space. The authors acknowledge that future work should explore manipulation—using the humanoid's hands to interact with and move objects.
  • Hierarchy Depth: The system currently uses a two-level hierarchy. More complex tasks might benefit from deeper hierarchies or temporal abstraction.
  • High-Level Pretraining: While the low-level agent is pretrained, the high-level agent is currently trained from scratch for each task. Pretraining the high-level agent on a diverse set of navigation tasks could further improve efficiency and generalization.

Summary updated

Abstract

Whole-body control for humanoids is challenging due to the high-dimensional nature of the problem, coupled with the inherent instability of a bipedal morphology. Learning from visual observations further exacerbates this difficulty. In this work, we explore highly data-driven approaches to visual whole-body humanoid control based on reinforcement learning, without any simplifying assumptions, reward design, or skill primitives. Specifically, we propose a hierarchical world model in which a high-level agent generates commands based on visual observations for a low-level agent to execute, both of which are trained with rewards. Our approach produces highly performant control policies in 8 tasks with a simulated 56-DoF humanoid, while synthesizing motions that are broadly preferred by humans.
BibTeX
@inproceedings{hansen2025hierarchical,
  author = {Nicklas Hansen and Jyothir S V and Vlad Sobal and Yann LeCun and Xiaolong Wang and Hao Su},
  title = {{Hierarchical World Models as Visual Whole-Body Humanoid Controllers}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=7wuJMvK639},
}
Figure