TL;DR
HALO is a human-aligned data collection system that enables high-quality demonstrations for whole-body mobile manipulation by bridging the "perception" and "mobility" gaps found in existing interfaces. By allowing operators to physically guide robot arms while walking naturally on a low-resistance platform, HALO speeds up data collection by up to 2.9× and enables robots to learn complex, sub-millimeter precision tasks like zip-tie fastening.
What Problem Does This Paper Address?
Training robots for whole-body mobile manipulation—tasks that require the simultaneous coordination of two arms and a moving base—is notoriously difficult. The bottleneck is often the data collection interface.
Existing methods generally suffer from two major issues:
- The Perception Gap: In leader-follower systems (like Mobile ALOHA), the operator stands behind the robot. Their viewpoint and hand positions are spatially separated from the robot's end-effectors, making fine-grained manipulation nearly impossible.
- The Mobility Gap: Powered robot bases are heavy and have high inertia. Operators often have to "fight" the base or use unintuitive joysticks, which disrupts the natural flow of human locomotion and coordination.
As shown in the figure below, HALO aims to eliminate these gaps by putting the operator in the "shoes" of the robot.
Methods
HALO introduces a hardware-software co-design to make robot teaching feel natural:
- Human-Aligned Hardware: The system uses kinesthetic teaching, where the operator directly moves the robot's arms. To facilitate this, the authors redesigned the standard ALOHA end-effectors to be more compact and ergonomic, increasing the available pitch angle by 40° and providing better physical feedback.
- Low-Resistance Mobility: During data collection, the heavy powered base is replaced with a lightweight, passive platform. This reduces the force required to move the system from 47.3N to just 2.3N, allowing the operator to walk and turn naturally.
- Deployment-Aligned Post-Processing: Because the operator's hands are visible during kinesthetic teaching, HALO uses a vision pipeline (incorporating SAM2 and DiffuEraser) to mask and inpaint the demonstrator's hands. This ensures the trained policy doesn't rely on "seeing" human hands to function.
- Base Motion Alignment: The system calibrates the passive platform's encoder data to match the specific dynamics and latencies of the actual powered base used during deployment.
- Policy Learning: The authors fine-tuned a pretrained Vision-Language-Action (VLA) model ($\pi_{0.5}$) using Asynchronous Real-Time Action Chunking (RTC), which helps maintain smooth, continuous motion during execution.
Datasets
The researchers evaluated HALO on several challenging real-world tasks designed to test different aspects of mobile manipulation:
- Push Chair: Requires rapid, coordinated base and bimanual movement.
- Disassemble Lego: Tests precise bimanual coordination with randomized configurations.
- Zip-Tie Fastening: A high-dexterity task requiring sub-millimeter accuracy.
- Long-Horizon Box Manipulation: A six-stage task involving retrieving a box, loading objects, transporting it to a conveyor, and returning to the start.
Key Insights and Ideas
- Adapt the Robot to the Human: The core philosophy is that the interface should adapt to human capabilities rather than forcing the human to adapt to the robot's constraints.
- Direct Feedback is King: Kinesthetic teaching provides immediate force and visual feedback that VR or leader-follower arms cannot match, which is critical for tasks involving forceful contact or high precision.
- Inpainting as a Bridge: By using advanced video inpainting, researchers can collect "dirty" data (with the human in the loop) and clean it to look like autonomous execution, effectively bridging the embodiment gap.
- Velocity-Based Control for Generalization: Using direct base velocity instead of global coordinates encourages the policy to learn reactive behaviors, allowing it to generalize to new environments and large starting position variations (up to 2-3 meters).
Key Results
- Efficiency: HALO achieved a 2.9× speedup in demonstration collection for the "Push Chair" task compared to Mobile ALOHA.
- Dexterity: The system enabled a 30% success rate on Zip-Tie Fastening, a task where both Mobile ALOHA and VR teleoperation failed completely (0% success).
- Coordination: On the complex six-stage long-horizon task, HALO-trained policies achieved a 73.3% success rate.
- Robustness: The "Push Chair" policy generalized to unseen backgrounds and different chair-table configurations, largely due to the reactive nature of the learned velocity-based control.
Limitations and Future Work
- Visual Artifacts: While the inpainting pipeline is effective, it can still leave minor visual artifacts that might affect policy robustness in highly cluttered scenes.
- Active Perception: The current setup uses fixed cameras. The authors suggest that future work could incorporate "active perception," where the robot moves its head/cameras to mimic human gaze patterns during complex tasks.
- Inference Latency: While asynchronous execution (RTC) helps, very high-speed maneuvers are still sensitive to the latency of the underlying VLA models.
Summary updated