Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos

Hao Luo, Yicheng Feng, Wanpeng Zhang, Sipeng Zheng, Ye Wang, Haoqi Yuan, Jiazheng Liu, Chaoyi Xu, Qin Jin, Zongqing LuarXiv 2025 · DOI · arXiv

Introduction

Robotics currently faces a significant data bottleneck. While Large Language Models (LLMs) have scaled rapidly by leveraging vast internet datasets, embodied AI lacks a comparable repository of high-quality robotic demonstrations. This scarcity is particularly acute in dexterous manipulation—tasks requiring fine-grained finger control—where teleoperating multi-fingered hands is complex and synthetic data often fails to bridge the sim-to-real gap.

Being-H0 addresses this challenge by treating the human hand as a "foundation manipulator." By pretraining a Vision-Language-Action (VLA) model on massive datasets of human videos, the researchers demonstrate that robots can learn sophisticated motion priors from human behavior, significantly reducing the need for expensive robot-specific data.

Methodology: Physical Instruction Tuning

The Being-H0 framework utilizes a three-stage "Physical Instruction Tuning" paradigm to bridge the gap between human video and robotic execution:

  1. VLA Pretraining: Using an InternVL3 backbone, the model is trained to predict 3D human hand motions from video and textual instructions. The model treats 3D motion as a "foreign language," predicting it autoregressively within a transformer architecture.
  2. Physical Space Alignment: To ensure spatial consistency across diverse camera viewpoints found in web videos, the framework employs a "weak-perspective" projection. This maps visual data into a consistent 3D coordinate system, allowing the model to learn spatial reasoning rather than simple 2D pixel patterns.
  3. Post-training Adaptation: The pretrained model is fine-tuned on a limited set of robot-specific data. Despite the morphological differences between human and robotic hands, the underlying motion logic—such as reaching, grasping, and orienting—transfers effectively.

A critical technical innovation is Part-Level Motion Tokenization. Rather than quantizing the entire hand as a single unit, Being-H0 separates the wrist (global position) from the fingers (fine articulation). Utilizing Grouped Residual Quantization (GRQ), the model achieves millimeter-level reconstruction accuracy, which is essential for high-precision tasks like threading or handling small objects.

The UniHand Dataset

To facilitate this large-scale pretraining, the researchers curated UniHand, a comprehensive dataset comprising over 150 million instruction-following samples. UniHand aggregates 11 distinct data sources, including:

  • Motion Capture: High-precision 3D data from sources like OAKINK2 and ARCTIC.
  • VR-Recorded Data: Egocentric data captured via devices like the Apple Vision Pro (e.g., EgoDex).
  • Pseudo-Annotated Videos: "In-the-wild" RGB videos where 3D labels were generated using state-of-the-art pose predictors.

For the primary experimental phase, a balanced subset of 2.5 million samples (UniHand-2.5M) was utilized.

Key Insights and Results

Scaling Laws in Robotics

The research demonstrates a clear scaling law for robotic manipulation. As the model size increases (from 1B to 14B parameters) and the training data expands, the model’s ability to follow complex instructions and maintain physical plausibility improves significantly. By tokenizing 3D MANO hand parameters into discrete codes, robotic actions are integrated directly into the transformer, allowing the model to reason about physical actions similarly to how LLMs reason about text.

Real-World Performance

In hardware evaluations using a Franka arm equipped with an Inspire dexterous hand, Being-H0 outperformed established baselines such as GR00T N1.5 and InternVL3. Notable achievements include:

  • Complex Tasks: A 100% success rate in "Pour-Cup" tasks and a 75% success rate in "Unfold-Clothes," a task where many baselines failed to even initiate a successful grasp.
  • Data Efficiency: Being-H0 demonstrated remarkable sample efficiency, with models trained on only 25% of available robot data matching or exceeding the performance of non-pretrained baselines trained on 100% of the data.
  • Generalization: The model exhibited zero-shot generalization, successfully performing pick-and-place tasks with novel objects in cluttered environments, suggesting a robust understanding of scene geometry rather than simple trajectory memorization.

Limitations and Future Directions

While Being-H0 represents a significant step forward, several challenges remain. The current model focuses primarily on hand motion and does not explicitly model the 6D pose or geometry of the objects being manipulated. Additionally, the transfer from human motion to robotic action currently relies on a relatively simple MLP-based projection; future work may explore more sophisticated strategies like diffusion policies. Finally, the integration of tactile and force feedback remains a critical frontier for achieving truly human-level dexterity.

Summary updated

Abstract

We introduce Being-H0, a dexterous Vision-Language-Action model (VLA) trained on large-scale human videos. Existing VLAs struggle with complex manipulation tasks requiring high dexterity and generalize poorly to novel scenarios and tasks, primarily due to their reliance on synthetic data with significant sim-to-real gaps or teleoperated demonstrations lacking scale and diversity. To address this data bottleneck, we propose leveraging human hands as a foundation manipulator, capitalizing on the rich dexterity and scalability present in web data. Our approach centers on physical instruction tuning, a novel training paradigm that combines large-scale VLA pretraining from human videos, physical space alignment for 3D reasoning, and post-training adaptation for robotic tasks. Additionally, we introduce a part-level motion tokenization method which achieves millimeter-level reconstruction accuracy to model precise hand trajectories for action learning. To support our proposed paradigm, we further develop a comprehensive data curation pipeline that integrates heterogeneous sources -- including motion capture, VR, and RGB-only videos -- into a large-scale dataset with millions of motion-based instructional instances. We empirically show the excellence of Being-H0 in hand motion generation and instruction following, and it also scales well with model and data sizes. Importantly, we observe the expected gains of Being-H0 in real-world robotic manipulation as physical instruction tuning is applied. More details are available at https://beingbeyond.github.io/Being-H0.
BibTeX
@misc{luo2025being,
  author = {Hao Luo and Yicheng Feng and Wanpeng Zhang and Sipeng Zheng and Ye Wang and Haoqi Yuan and Jiazheng Liu and Chaoyi Xu and Qin Jin and Zongqing Lu},
  title = {{Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos}},
  year = {2025},
  eprint = {2507.15597},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.15597},
}
Figure