HALO: Human-Aligned Data Collection for WhoLe-body MObile Manipulation

Shuai Yang, Fangyi Xu, Yishu Shen, Kecheng Zheng, Yinghao XuarXiv 2026 · arXiv

TL;DR

HALO is a human-aligned data collection system that enables high-quality demonstrations for whole-body mobile manipulation by bridging the "perception" and "mobility" gaps found in existing interfaces. By allowing operators to physically guide robot arms while walking naturally on a low-resistance platform, HALO speeds up data collection by up to 2.9× and enables robots to learn complex, sub-millimeter precision tasks like zip-tie fastening.

What Problem Does This Paper Address?

Training robots for whole-body mobile manipulation—tasks that require the simultaneous coordination of two arms and a moving base—is notoriously difficult. The bottleneck is often the data collection interface.

Existing methods generally suffer from two major issues:

  1. The Perception Gap: In leader-follower systems (like Mobile ALOHA), the operator stands behind the robot. Their viewpoint and hand positions are spatially separated from the robot's end-effectors, making fine-grained manipulation nearly impossible.
  2. The Mobility Gap: Powered robot bases are heavy and have high inertia. Operators often have to "fight" the base or use unintuitive joysticks, which disrupts the natural flow of human locomotion and coordination.

As shown in the figure below, HALO aims to eliminate these gaps by putting the operator in the "shoes" of the robot.

Figure 1: Interface comparison. Mobile ALOHA exhibits perception and mobility gaps, while our interface aligns human–robot perception and preserves natural operator mobility.

Methods

HALO introduces a hardware-software co-design to make robot teaching feel natural:

  • Human-Aligned Hardware: The system uses kinesthetic teaching, where the operator directly moves the robot's arms. To facilitate this, the authors redesigned the standard ALOHA end-effectors to be more compact and ergonomic, increasing the available pitch angle by 40° and providing better physical feedback.
  • Low-Resistance Mobility: During data collection, the heavy powered base is replaced with a lightweight, passive platform. This reduces the force required to move the system from 47.3N to just 2.3N, allowing the operator to walk and turn naturally.
  • Deployment-Aligned Post-Processing: Because the operator's hands are visible during kinesthetic teaching, HALO uses a vision pipeline (incorporating SAM2 and DiffuEraser) to mask and inpaint the demonstrator's hands. This ensures the trained policy doesn't rely on "seeing" human hands to function.
  • Base Motion Alignment: The system calibrates the passive platform's encoder data to match the specific dynamics and latencies of the actual powered base used during deployment.
  • Policy Learning: The authors fine-tuned a pretrained Vision-Language-Action (VLA) model ($\pi_{0.5}$) using Asynchronous Real-Time Action Chunking (RTC), which helps maintain smooth, continuous motion during execution.

Datasets

The researchers evaluated HALO on several challenging real-world tasks designed to test different aspects of mobile manipulation:

  • Push Chair: Requires rapid, coordinated base and bimanual movement.
  • Disassemble Lego: Tests precise bimanual coordination with randomized configurations.
  • Zip-Tie Fastening: A high-dexterity task requiring sub-millimeter accuracy.
  • Long-Horizon Box Manipulation: A six-stage task involving retrieving a box, loading objects, transporting it to a conveyor, and returning to the start.

Key Insights and Ideas

  • Adapt the Robot to the Human: The core philosophy is that the interface should adapt to human capabilities rather than forcing the human to adapt to the robot's constraints.
  • Direct Feedback is King: Kinesthetic teaching provides immediate force and visual feedback that VR or leader-follower arms cannot match, which is critical for tasks involving forceful contact or high precision.
  • Inpainting as a Bridge: By using advanced video inpainting, researchers can collect "dirty" data (with the human in the loop) and clean it to look like autonomous execution, effectively bridging the embodiment gap.
  • Velocity-Based Control for Generalization: Using direct base velocity instead of global coordinates encourages the policy to learn reactive behaviors, allowing it to generalize to new environments and large starting position variations (up to 2-3 meters).
Figure 2: HALO overview and learned capabilities. (a) Human-aligned data collection with natural locomotion and direct bimanual kinesthetic teaching. (b) Learned policies exhibit precise manipulation, coordinated whole-body control, long-horizon mobile manipulation, and navigation.

Key Results

  • Efficiency: HALO achieved a 2.9× speedup in demonstration collection for the "Push Chair" task compared to Mobile ALOHA.
  • Dexterity: The system enabled a 30% success rate on Zip-Tie Fastening, a task where both Mobile ALOHA and VR teleoperation failed completely (0% success).
  • Coordination: On the complex six-stage long-horizon task, HALO-trained policies achieved a 73.3% success rate.
  • Robustness: The "Push Chair" policy generalized to unseen backgrounds and different chair-table configurations, largely due to the reactive nature of the learned velocity-based control.

Limitations and Future Work

  • Visual Artifacts: While the inpainting pipeline is effective, it can still leave minor visual artifacts that might affect policy robustness in highly cluttered scenes.
  • Active Perception: The current setup uses fixed cameras. The authors suggest that future work could incorporate "active perception," where the robot moves its head/cameras to mimic human gaze patterns during complex tasks.
  • Inference Latency: While asynchronous execution (RTC) helps, very high-speed maneuvers are still sensitive to the latency of the underlying VLA models.

Summary updated

Abstract

Learning whole-body mobile manipulation requires demonstrations that coordinate bimanual actions with locomotion across extended workspaces. However, existing interfaces trade off direct robot control, perception and action alignment, and natural operator mobility, making such demonstrations difficult to collect. We introduce HALO, a human-aligned data collection system for whole-body mobile manipulation. The system aligns the operator's perception and action space with the robot while preserving direct physical feedback and natural locomotion. Operators guide the robot end-effectors through kinesthetic teaching and move a low-resistance platform without the deployed mobile base while performing bimanual manipulation. This interface enables more natural whole-body demonstrations, but the close human-robot coupling introduces discrepancies between the demonstration and deployment settings. We address these discrepancies with deployment-aligned demonstration processing, which calibrates and temporally aligns platform motion with the deployed base response and masks and inpaints demonstrator hands in recorded observations. Compared with Mobile ALOHA and VR teleoperation, HALO enables up to 2.9x faster demonstration collection while maintaining comparable or higher success rates. Policies trained with HALO perform precise manipulation tasks, including zip-tie fastening, and achieve a 73.3% success rate on a six-stage long-horizon mobile manipulation task involving repeated coordinated base and arm motion. Project page: https://robbyant-research.github.io/Lingbot-ALOHA/
BibTeX
@misc{yang2026halo,
  author = {Shuai Yang and Fangyi Xu and Yishu Shen and Kecheng Zheng and Yinghao Xu},
  title = {{HALO: Human-Aligned Data Collection for WhoLe-body MObile Manipulation}},
  year = {2026},
  eprint = {2611.16439},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16439},
}
Figure