TL;DR
Puppeteer is a hierarchical world model that enables 56-DoF humanoids to perform complex tasks like jumping over gaps using only visual observations. By decoupling low-level physics from high-level task planning through a MoCap-pretrained tracking agent, it achieves state-of-the-art performance while producing remarkably natural, human-like motions that are preferred by humans 97% of the time.
What Problem Does This Paper Address?
Whole-body humanoid control is often considered a "grand challenge" in robotics. It requires managing a high-dimensional action space (56 joints in this study) while maintaining stability in a bipedal morphology that is inherently prone to falling.
When you add visual perception to the mix—requiring the agent to perceive terrain and obstacles from pixels while simultaneously coordinating its limbs—the problem becomes exponentially harder. Traditional Reinforcement Learning (RL) often results in "reward hacking," where agents find mathematically optimal but physically nonsensical ways to move, such as "rolling" or sliding across the floor. To avoid this, researchers usually rely on simplifying assumptions like privileged information (exact coordinates of obstacles) or pre-defined "skill primitives." The authors of this paper sought a purely data-driven approach that produces natural movement without manual engineering.
Methods
The authors propose Puppeteer, a two-stage hierarchical framework built on the TD-MPC2 algorithm. TD-MPC2 is a model-based RL method that learns a latent world model and selects actions by planning (using Model Predictive Control) within that model.
The hierarchy consists of two distinct agents:
- Low-level Tracking Agent: This agent is pretrained on a large dataset of human Motion Capture (MoCap) data. It learns to map "commands"—specifically, the desired 3D positions of the head, hands, and feet—into joint torques. This effectively bakes "human-like" physics into the controller.
- High-level Puppeteer Agent: This agent is trained on specific downstream tasks (e.g., navigating a corridor with hurdles). It takes RGB camera images and proprioceptive data as input and outputs the 3D end-effector targets for the tracking agent to follow.
A key innovation here is the interface: instead of using abstract embeddings, the high-level agent outputs geometric locations. This makes the low-level tracking agent highly reusable across entirely different tasks.
Datasets
- Pretraining: The authors used the CMU MoCap dataset (via the MoCapAct subset), which includes over 800 clips of human movement retargeted to a 56-DoF humanoid model.
- Evaluation: The paper introduces a new, challenging benchmark suite of 8 tasks for a 56-DoF humanoid:
- Visual Tasks: Corridor, Hurdles, Walls, Gaps, and Stairs.
- Non-visual Tasks: Stand, Walk, and Run.
Key Insights and Ideas
- MoCap as a Structural Prior: Pretraining on MoCap doesn't just teach the robot how to move; it constrains the solution space. By forcing the agent to move through an end-effector interface learned from humans, the "unnatural" solutions (like rolling) are effectively filtered out.
- The Necessity of Planning: The authors found that planning at both levels of the hierarchy is critical. Model-free methods (like SAC) or models that don't use online planning (like DreamerV3) failed to achieve meaningful performance on the visual tasks.
- Zero-Shot Generalization: Because the model learns a robust understanding of physics and geometry, it can generalize to environments it never saw during training. For example, an agent trained on 0.4m gaps could successfully navigate gaps up to 1.2m wide without further training.
Key Results
- Performance: Puppeteer matches or exceeds the performance of flat, non-hierarchical state-of-the-art models (TD-MPC2) across most tasks, while being significantly more sample-efficient than prior hierarchical methods.
- Naturalness: In a user study with 51 participants, 97.8% of responses preferred Puppeteer’s motions over the baseline. While the baseline TD-MPC2 often learned to "roll" up stairs to maximize speed, Puppeteer learned a natural walking gait.
- Quantitative Metrics: Puppeteer maintained a human-like torso height (approx. 96cm) compared to the baseline (approx. 86cm), which tended to crouch or deform its posture to optimize for reward.
Limitations and Future Work
- Visio-Locomotive Focus: The current tasks are primarily about moving through space. The authors acknowledge that future work should explore manipulation—using the humanoid's hands to interact with and move objects.
- Hierarchy Depth: The system currently uses a two-level hierarchy. More complex tasks might benefit from deeper hierarchies or temporal abstraction.
- High-Level Pretraining: While the low-level agent is pretrained, the high-level agent is currently trained from scratch for each task. Pretraining the high-level agent on a diverse set of navigation tasks could further improve efficiency and generalization.
Summary updated