SHAPE: Discrete Supervision and Parallel Continuous Streams for Learning Dexterous Manipulation from Egocentric Human Data

Yaping Li, Zhihong Liu, Jiacheng Li, Yi Yang, Wenchao Yue, Mingzhe Ye, Xiangcheng Liu, Yiyang Chen, Haotian He, Qi Zhang, Haonan Wang, Dahua Lin, Zhijie DengarXiv 2026 · arXiv

TL;DR

SHAPE is a new vision-language-action (VLA) model that bridges the "action gap" between human videos and robot execution for dexterous manipulation. By using discrete action tokens as an auxiliary training task and providing dual executable output streams (joint and end-effector), SHAPE achieves state-of-the-art performance on complex bimanual tasks and can even learn new skills directly from human demonstrations without any task-specific robot data.

What Problem Does This Paper Address?

Training robots to perform dexterous, multi-fingered manipulation is notoriously difficult and expensive. While we can collect thousands of hours of egocentric human videos (like someone cooking or cleaning), humans and robots have different "bodies" (morphology and kinematics).

This creates two major hurdles:

  1. The Action Gap: Human videos provide visual cues and retargeted hand poses, but they lack the specific joint commands and actuator signals a robot needs.
  2. Deployment Dilemma: Should a robot follow precise joint-level commands (good for accuracy) or high-level end-effector (EEF) targets (better for transferring skills from humans)?

Current models often struggle to effectively "digest" human data to improve robot performance, or they force a choice between precision and transferability.

Methods

The authors introduce SHAPE (Supervision from Human Action tokens with Parallel Executable streams). Its architecture consists of:

  • A Vision-Language Backbone: Based on Qwen3-VL-4B, it processes images and instructions.
  • Auxiliary Discrete Supervision: The backbone is trained to predict discrete "FAST" tokens of action chunks. This provides a direct training signal to the backbone, helping it learn action-informative representations from human data where joint labels are missing.
  • Dual-Stream Action Expert: A 0.6B parameter transformer using flow-matching to output two parallel, executable streams:
  • Joint Stream: Direct commands for the robot's 22-DoF hands and 7-DoF arms.
  • EEF Stream: High-level wrist and fingertip targets, which are then converted to joint commands via Inverse Kinematics (IK).
Overview of the SHAPE architecture and training process.

Datasets

The model was pretrained on a massive mix of data:

  • Human Data: 2.8k hours of egocentric recordings covering 320 everyday tasks.
  • Robot Data: 228 hours of bimanual teleoperation data (105 tasks) on a robot with two 7-DoF arms and two 22-DoF dexterous hands.
  • Evaluation Benchmarks: Real-robot testing on 6 tasks with robot demonstrations (e.g., inserting a USB, opening a cabinet) and 3 "human-to-robot" (H2R) tasks with zero robot demonstrations. They also evaluated on 24 tasks in the RoboCasa simulation environment.

Key Insights and Ideas

  • Discrete Tokens are the "Secret Sauce": Continuous regression gradients often get "lost" in the action decoder. Predicting discrete tokens forces the backbone to organize its internal representations around task-relevant actions, significantly boosting the benefit of human pretraining.
  • The Best of Both Worlds: The authors found that the Joint Stream is superior for fine-grained control when robot data is available, while the EEF Stream is far better at transferring skills from human videos because it abstracts away the robot's specific joint configuration.
  • Zero-Robot-Data Learning: Because SHAPE learns a shared representation of "action" across humans and robots, it can watch a human perform a new task and then execute it on the robot using the EEF stream, even if it has never seen a robot demonstration for that specific task.
Analysis showing how discrete tokens help the backbone learn action-informative features.

Key Results

  • Real-Robot Performance: On six complex bimanual tasks, SHAPE achieved a score of 0.80, outperforming the previous state-of-the-art $\pi_{0.5}$ (0.63) and NVIDIA’s GR00T N1.7 (0.51).
  • Human-to-Robot Transfer: In tasks with zero robot demonstrations, SHAPE achieved a 35% success rate, whereas models trained only on robot data failed completely (0% success).
  • Simulation Success: In the RoboCasa benchmark, SHAPE reached a 72.2% overall success rate, significantly higher than baselines.
Success rates for human-to-robot transfer across different tasks.

Limitations and Future Work

While SHAPE is a significant step forward, some challenges remain:

  • Precision Bottlenecks: Extremely fine-grained tasks, like millimeter-tolerance connector insertion, still have low completion rates (around 10-15%).
  • Transfer Success Rate: While 35% success on zero-robot-data tasks is a breakthrough, it is still far from the reliability needed for production environments.
  • Scaling: The authors suggest that scaling to even larger human video corpora and a wider variety of robot embodiments is the next logical step to improve generalization.

Summary updated

Abstract

Egocentric human recordings are far cheaper than dexterous robot demonstrations, promising stronger pretraining and direct skill learning without task-specific robot data. Yet humans and robots differ in morphology and kinematics, so human recordings lack the joint positions and actuator commands that supervise robot policies, providing only retargeted end-effector (EEF) actions. This raises two questions: how human data should supervise the policy, and how a dexterous robot should read out actions at deployment. We address both in SHAPE (Supervision from Human Action tokens with Parallel Executable streams), a vision-language-action (VLA) model whose backbone predicts discrete action tokens of EEF chunks as an auxiliary task, while a shared flow-matching expert outputs two executable streams, EEF and joint. On a bimanual robot with 22-DoF hands, a matched study shows that discrete supervision helps learn from human data, driving backbone gradients and shaping action-informative representations across domains. Under standard control pipelines at deployment, direct joint readout excels at fine-grained control with robot demonstrations, scoring 0.80 across six tasks against 0.63 for π0.5, whereas IK-based EEF readout excels at human-to-robot transfer. Pretrained with human data and discrete tokens, SHAPE learns new tasks from human demonstrations without task-specific robot data.
BibTeX
@misc{li2026shape,
  author = {Yaping Li and Zhihong Liu and Jiacheng Li and Yi Yang and Wenchao Yue and Mingzhe Ye and Xiangcheng Liu and Yiyang Chen and Haotian He and Qi Zhang and Haonan Wang and Dahua Lin and Zhijie Deng},
  title = {{SHAPE: Discrete Supervision and Parallel Continuous Streams for Learning Dexterous Manipulation from Egocentric Human Data}},
  year = {2026},
  eprint = {2611.24585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24585},
}
Figure