Hierarchical World Models as Visual Whole-Body Humanoid Controllers

Nicklas Hansen, Jyothir S. V, Vlad Sobal, Yann LeCun, Xiaolong Wang, Hao Su

Abstract

Whole-body control for humanoids is challenging due to the high-dimensional nature of the problem, coupled with the inherent instability of a bipedal morphology. Learning from visual observations further exacerbates this difficulty. In this work, we explore highly data-driven approaches to visual whole-body humanoid control based on reinforcement learning, without any simplifying assumptions, reward design, or skill primitives. Specifically, we propose a hierarchical world model in which a high-level agent generates commands based on visual observations for a low-level agent to execute, both of which are trained with rewards. Our approach produces highly performant control policies in 8 tasks with a simulated 56-DoF humanoid, while synthesizing motions that are broadly preferred by humans. Code and videos: https://nicklashansen.com/rlpuppeteer

Introduction

Controlling a humanoid robot is a "grand challenge" in robotics due to the high-dimensional action space and the inherent instability of bipedal movement. Traditionally, researchers have relied on privileged information (e.g., exact coordinates of obstacles) or pre-defined skill primitives to simplify the problem. While these methods work, they are often brittle and produce robotic, unnatural movements.

End-to-end reinforcement learning (RL) from pixels to joint torques is theoretically possible but often results in "reward hacking," where agents adopt efficient but bizarre behaviors rather than human-like movement. This paper introduces Puppeteer, a hierarchical framework that enables a 56-degree-of-freedom (DoF) humanoid to perform complex, vision-based tasks while maintaining natural motion.

Methods

The authors propose Puppeteer, a hierarchical framework consisting of two distinct agents, both implemented using the TD-MPC2 model-based reinforcement learning algorithm.

The Puppeteer hierarchical framework decouples high-level visual planning from low-level tracking.

Hierarchical Architecture

  1. Low-Level Tracking Agent: This agent is pre-trained on the CMU Motion Capture (MoCap) dataset. It acts as a "physics engine" for human movement, translating abstract commands (desired 3D positions of the head, hands, and feet) into joint-level torques. Because it is trained on human data, it inherently prefers human-like movements.
  2. High-Level Puppeteer Agent: This agent handles perception and strategy. It takes raw RGB camera images and the robot's internal state as input, then "plays" the low-level agent like a puppet by outputting the 3D target coordinates for the tracking agent to follow.

Planning with Termination

A key technical contribution is the addition of a termination prediction head to the world model. By learning to predict the probability of a fall during its internal "imagination" (planning) phase, the agent can proactively avoid actions that lead to instability.

Experimental Setup

The system was evaluated on a benchmark of 8 tasks for a 56-DoF humanoid, including 5 visual tasks and 3 proprioceptive tasks.

The Puppeteer task suite includes proprioceptive locomotion and visual navigation tasks.
  • Datasets: The CMU MoCap (specifically the MoCapAct subset) was used to pre-train the low-level tracking agent.
  • Tasks: Visual tasks included navigating a corridor, jumping hurdles, weaving through walls, crossing gaps, and climbing stairs. Proprioceptive tasks included standing, walking, and running.

Key Results

  • Performance and Generalization: Puppeteer matched or exceeded the performance of strong baselines like DreamerV3 and non-hierarchical TD-MPC2. It also demonstrated zero-shot generalization; for example, an agent trained on 0.4m gaps could successfully navigate gaps up to 1.2m wide.
  • Human Preference: In a user study with 51 participants, 97.8% preferred Puppeteer's motions over standard RL agents, describing them as significantly more natural and human-like.
  • Efficiency: While standard end-to-end RL often requires hundreds of millions of interactions, Puppeteer's high-level agent can learn new visual tasks in just 3 million steps, thanks to the pre-trained low-level controller.

Limitations and Future Work

The current work primarily focuses on visio-locomotive tasks. Future research should explore whole-body manipulation (e.g., carrying objects) to test the hierarchy's ability to coordinate limbs for tasks beyond balance. Additionally, transitioning these models to physical hardware remains a challenge due to the "sim-to-real" gap in contact physics.

Summary updated

Abstract

Whole-body control for humanoids is challenging due to the high-dimensional nature of the problem, coupled with the inherent instability of a bipedal morphology. Learning from visual observations further exacerbates this difficulty. In this work, we explore highly data-driven approaches to visual whole-body humanoid control based on reinforcement learning, without any simplifying assumptions, reward design, or skill primitives. Specifically, we propose a hierarchical world model in which a high-level agent generates commands based on visual observations for a low-level agent to execute, both of which are trained with rewards. Our approach produces highly performant control policies in 8 tasks with a simulated 56-DoF humanoid, while synthesizing motions that are broadly preferred by humans. Code and videos: https://nicklashansen.com/rlpuppeteer
BibTeX
@inproceedings{hansen2025hierarchical,
  author = {Nicklas Hansen and Jyothir S. V and Vlad Sobal and Yann LeCun and Xiaolong Wang and Hao Su},
  title = {{Hierarchical World Models as Visual Whole-Body Humanoid Controllers}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=7wuJMvK639},
}
Figure