TL;DR
HEAVE-HO is a decentralized framework that enables teams of humanoid robots to collaborate on complex, long-horizon tasks—like carrying a heavy table through a narrow doorway—without any network communication. By combining a vision-language planner with speech-based synchronization and a robust low-level residual policy, the system scales seamlessly from pairs to teams of ten and even allows humans to join the robot squad.
What Problem Does This Paper Address?
While individual humanoid robots are becoming increasingly capable at locomotion and simple manipulation, many real-world tasks remain physically impossible for a single machine. Moving a couch, handling large crates, or navigating a construction site often requires a team.
However, multi-robot collaboration is notoriously difficult due to three main challenges:
- Tight Physical Coupling: When robots hold the same object, the forces they exert are interdependent. A wrong move by one can trip the other.
- Long-Horizon Planning: Tasks like "move this box to the truck" require a sequence of coordinated skills (lift, turn, walk, handover) that must be visually grounded in the environment.
- Synchronization: In a decentralized setup, robots must start and stop actions at the exact same time to maintain stability. Traditional solutions rely on high-bandwidth WiFi/network communication, which can be unreliable or laggy in the field.
HEAVE-HO tackles these by removing the "network crutch," opting instead for a decentralized approach where robots coordinate through local observations and audible speech.
Methods
The authors propose a hierarchical framework that splits the problem into high-level planning and low-level control.
High-Level: Vision-Language Planning & Speech Sync A "leader" robot (or a human) uses a fine-tuned vision-language model (Qwen3-VL-2B) to observe the scene and decide on the next team skill (e.g., "step left"). Instead of sending a packet over WiFi, the leader speaks the command. Follower robots use on-board microphones to recognize the command. To ensure they all start at once, they follow a shared "skill schedule"—a coarse timing grid that absorbs the small variations in acoustic delay.
Low-Level: Multi-Agent Residual Post-Training Rather than training a collaborative controller from scratch, the authors use a "residual" approach. They take a frozen foundation model for humanoid tracking (SONIC) and add a small, trainable "residual policy" on top. This residual layer is trained via Multi-Agent Reinforcement Learning (MAPPO) to handle the specific forces and contact requirements of shared loads. Crucially, at execution time, each robot only sees its own proprioception and the object's basic specs—it has no direct access to its partners' internal states.
Datasets
The researchers introduced HH-BENCH, a new benchmark specifically designed for multi-humanoid collaboration. It consists of 12 diverse tasks:
- Lift and Carry: Navigating narrow doors, L-shaped corridors, and stairs.
- Diverse Interaction: Shoulder-carrying (palanquin/rebar), pulling fire hoses, and pushing heavy millstones.
- Handover: Passing objects between robots at different heights (e.g., from a truck bed).
The vision-language planner was trained on a mix of 24 real-world episodes per task and 1,000 simulated episodes. Reference motions for the low-level policy were derived from human videos and procedural inverse kinematics.
Key Insights and Ideas
- Speech as a Universal Interface: Using audible commands isn't just a gimmick; it's a robust synchronization primitive that naturally allows for heterogeneous teams. Because the "API" is just spoken language, a Unitree G1 robot can lead an EngineAI PM01 robot, or a human can step in and take over the leader role seamlessly.
- The Power of Residuals: By freezing a foundation locomotion model, the learning process becomes much more stable. The residual policy doesn't have to learn "how to walk"; it only has to learn "how to walk while someone else is pulling on the table."
- Zero-Shot Scaling: Because the low-level policy is egocentric (it only cares about the robot's own relationship to the object), a policy trained on just two robots generalizes to teams of four, eight, or ten without any additional training.
Key Results
- Simulation Success: HEAVE-HO achieved a 94.4% success rate across HH-BENCH tasks in simulation, dwarfing baselines like CHORUS (12%) and Multi-HDMI (28.5%).
- Real-World Robustness: In 100 real-world trials, the system succeeded 92% of the time. It successfully navigated "signature" challenges like carrying a table through a door narrower than the table's length (requiring a coordinated rotation).
- Efficiency: The fine-tuned planner is roughly 120x faster than using zero-shot GPT-4 (Astra) through an API, with a per-decision latency of just 52.5 ms.
- Scalability: The authors demonstrated successful table-carrying with up to 10 robots in simulation and 4 robots in the real world using the exact same policy.
Limitations and Future Work
Despite its success, HEAVE-HO has a few "human-like" limitations:
- Acoustic Environment: The speech-based sync requires a reasonably quiet environment. In a very noisy factory, the followers might "mishear" the leader.
- Demonstration Heavy: Training the planner for entirely new tasks still requires a significant number of teleoperated demonstrations, which is labor-intensive.
- Fixed Roles: Currently, robots are assigned a fixed "slot" around an object (e.g., "front-left"). Future work could involve dynamic role assignment where robots decide where to grab based on the situation.
Overall, HEAVE-HO represents a significant step toward making humanoid teams a practical reality for logistics and construction, proving that sometimes, the best way for robots to talk to each other is just to speak up.
Summary updated