TL;DR
This paper introduces a modular control framework that sequences high-level loco-manipulation skills through a single, shared whole-body controller (WBC) to achieve long-horizon humanoid robot autonomy. To counter the distribution shifts that occur when chaining independently learned skills, the authors propose a rollout-based data aggregation technique that incrementally robustifies the shared WBC. The approach is validated on "Humanoid Hanoi," a newly introduced long-horizon box rearrangement benchmark, demonstrating successful multi-minute autonomous execution on the Digit V3 humanoid robot.
What Problem Does This Paper Address?
Executing long-horizon tasks with humanoid robots—such as rearranging boxes over several minutes—requires a seamless blend of locomotion and manipulation skills. While breaking these tasks down into modular, independently trained skills (like walking, picking, and placing) makes intuitive sense, naively chaining them together is notoriously brittle.
Two main issues arise:
- Low-Level Instability: Switching between skill-specific low-level controllers at skill boundaries alters closed-loop dynamics, often causing the robot to lose balance.
- Distribution Shift: Composing skills introduces novel robot states and high-level commands that the low-level controller never encountered during its isolated training phase. Small errors accumulate over time, pushing the robot into unrecoverable configurations.
The authors tackle these challenges by proposing an architecture that keeps the low-level whole-body controller fixed and shared across all skills, treating robustness as an incremental maintenance problem of this shared controller.
Methods
The control architecture is split into a hierarchical structure:
- High-Level Skills: A library of reusable skills (
GoTo,Pickup,GoTo-with-box, andPlace) implemented as LSTM policies trained via Reinforcement Learning (PPO). Instead of directly commanding joint torques, these skills output masked motion directives. This masking mechanism allows a skill to constrain only relevant parts of the body (e.g., arm positions for a pickup) while leaving other parts unconstrained. - Shared Whole-Body Controller (WBC): An always-on, task-agnostic Masked Humanoid Controller (MHC) takes the robot's proprioceptive state and the masked motion directives to output joint-level PD targets at 50 Hz. Because the WBC is shared, switching skills merely changes the high-level command stream rather than the underlying control laws.
- WBC Coverage Expansion (Data Aggregation): To make the shared WBC robust to skill-induced distribution shifts, the authors implement a rollout-based data aggregation procedure. They run the high-level skills with the initial WBC in simulation, collect the closed-loop directive sequences from successful rollouts, add these sequences to the WBC's reference trajectory library, and continue training the WBC under domain randomization.
Datasets
- AMASS Dataset: A large database of human motion capture data used for the initial pre-training of the Masked Humanoid Controller.
- Humanoid Hanoi Benchmark: A novel, long-horizon box rearrangement benchmark introduced in this paper. Inspired by the classic Tower of Hanoi puzzle, it requires a humanoid robot to move three boxes of varying sizes and masses between three tower locations while respecting stacking constraints. A full solution requires sequencing 7 moves, totaling 28 individual skill executions over an extended duration.
Key Insights and Ideas
- Robustness as WBC Maintenance: Instead of modifying low-level control on a per-skill basis (e.g., via residual policies or per-skill fine-tuning), the authors show that it is better to treat robustness as a data-coverage problem for a single, shared WBC.
- The Power of Masked Directives: Utilizing a masked interface allows the WBC to autonomously complete unspecified degrees of freedom (like leg joints and torso pitch) in a dynamically consistent manner, ensuring the robot maintains balance regardless of the active manipulation skill.
- Closed-Loop Data Aggregation Wins: Training the WBC on actual closed-loop trajectories generated by the high-level skills exposes the controller to realistic tracking errors and deployment distributions, which significantly eases sim-to-real transfer.
Key Results
The authors compared their Extended (data aggregation) approach against a Base frozen WBC, a Finetune baseline, and a Residual policy baseline.
- Simulation Performance: In evaluations featuring heavy domain randomization (DR), the Extended approach achieved a 49% complete-task success rate on the Humanoid Hanoi benchmark, substantially outperforming Base (25%), Finetune (21%), and Residual (19%).
- Hardware Deployment: The framework successfully transferred to the physical Digit V3 humanoid robot. Across three workspace configurations, Digit achieved an overall 40% full-task success rate (6 out of 15 trials), executing fully autonomous, continuous rearrangement sequences lasting over 5 minutes per successful trial.
Limitations and Future Work
Despite the promising results, the authors noted several failure modes that highlight areas for future research:
- Perception Vulnerability: The real-world system relied on AprilTags for localization and box tracking. Occlusions or lighting changes frequently caused tag loss, leading to failed grasps or placements.
- Object-Unaware Locomotion: The
GoTolocomotion skills were blind to the environment's obstacles. After placing a box, the robot would occasionally collide with the newly formed stack when walking backward or turning in place. - Accumulated Placement Errors: Small errors in base positioning before entering the
Placeskill directly translated into box alignment offsets, occasionally causing stacks to destabilize and collapse in later stages of the puzzle.
Summary updated