Abstract
Whole-body control for humanoids is challenging due to the high-dimensional nature of the problem, coupled with the inherent instability of a bipedal morphology. Learning from visual observations further exacerbates this difficulty. In this work, we explore highly data-driven approaches to visual whole-body humanoid control based on reinforcement learning, without any simplifying assumptions, reward design, or skill primitives. Specifically, we propose a hierarchical world model in which a high-level agent generates commands based on visual observations for a low-level agent to execute, both of which are trained with rewards. Our approach produces highly performant control policies in 8 tasks with a simulated 56-DoF humanoid, while synthesizing motions that are broadly preferred by humans. Code and videos: https://nicklashansen.com/rlpuppeteer
Introduction
Controlling a humanoid robot is a "grand challenge" in robotics due to the high-dimensional action space and the inherent instability of bipedal movement. Traditionally, researchers have relied on privileged information (e.g., exact coordinates of obstacles) or pre-defined skill primitives to simplify the problem. While these methods work, they are often brittle and produce robotic, unnatural movements.
End-to-end reinforcement learning (RL) from pixels to joint torques is theoretically possible but often results in "reward hacking," where agents adopt efficient but bizarre behaviors rather than human-like movement. This paper introduces Puppeteer, a hierarchical framework that enables a 56-degree-of-freedom (DoF) humanoid to perform complex, vision-based tasks while maintaining natural motion.
Methods
The authors propose Puppeteer, a hierarchical framework consisting of two distinct agents, both implemented using the TD-MPC2 model-based reinforcement learning algorithm.
Hierarchical Architecture
- Low-Level Tracking Agent: This agent is pre-trained on the CMU Motion Capture (MoCap) dataset. It acts as a "physics engine" for human movement, translating abstract commands (desired 3D positions of the head, hands, and feet) into joint-level torques. Because it is trained on human data, it inherently prefers human-like movements.
- High-Level Puppeteer Agent: This agent handles perception and strategy. It takes raw RGB camera images and the robot's internal state as input, then "plays" the low-level agent like a puppet by outputting the 3D target coordinates for the tracking agent to follow.
Planning with Termination
A key technical contribution is the addition of a termination prediction head to the world model. By learning to predict the probability of a fall during its internal "imagination" (planning) phase, the agent can proactively avoid actions that lead to instability.
Experimental Setup
The system was evaluated on a benchmark of 8 tasks for a 56-DoF humanoid, including 5 visual tasks and 3 proprioceptive tasks.
- Datasets: The CMU MoCap (specifically the MoCapAct subset) was used to pre-train the low-level tracking agent.
- Tasks: Visual tasks included navigating a corridor, jumping hurdles, weaving through walls, crossing gaps, and climbing stairs. Proprioceptive tasks included standing, walking, and running.
Key Results
- Performance and Generalization: Puppeteer matched or exceeded the performance of strong baselines like DreamerV3 and non-hierarchical TD-MPC2. It also demonstrated zero-shot generalization; for example, an agent trained on 0.4m gaps could successfully navigate gaps up to 1.2m wide.
- Human Preference: In a user study with 51 participants, 97.8% preferred Puppeteer's motions over standard RL agents, describing them as significantly more natural and human-like.
- Efficiency: While standard end-to-end RL often requires hundreds of millions of interactions, Puppeteer's high-level agent can learn new visual tasks in just 3 million steps, thanks to the pre-trained low-level controller.
Limitations and Future Work
The current work primarily focuses on visio-locomotive tasks. Future research should explore whole-body manipulation (e.g., carrying objects) to test the hierarchy's ability to coordinate limbs for tasks beyond balance. Additionally, transitioning these models to physical hardware remains a challenge due to the "sim-to-real" gap in contact physics.
Summary updated