TL;DR
SHAPE is a new vision-language-action (VLA) model that bridges the "action gap" between human videos and robot execution for dexterous manipulation. By using discrete action tokens as an auxiliary training task and providing dual executable output streams (joint and end-effector), SHAPE achieves state-of-the-art performance on complex bimanual tasks and can even learn new skills directly from human demonstrations without any task-specific robot data.
What Problem Does This Paper Address?
Training robots to perform dexterous, multi-fingered manipulation is notoriously difficult and expensive. While we can collect thousands of hours of egocentric human videos (like someone cooking or cleaning), humans and robots have different "bodies" (morphology and kinematics).
This creates two major hurdles:
- The Action Gap: Human videos provide visual cues and retargeted hand poses, but they lack the specific joint commands and actuator signals a robot needs.
- Deployment Dilemma: Should a robot follow precise joint-level commands (good for accuracy) or high-level end-effector (EEF) targets (better for transferring skills from humans)?
Current models often struggle to effectively "digest" human data to improve robot performance, or they force a choice between precision and transferability.
Methods
The authors introduce SHAPE (Supervision from Human Action tokens with Parallel Executable streams). Its architecture consists of:
- A Vision-Language Backbone: Based on Qwen3-VL-4B, it processes images and instructions.
- Auxiliary Discrete Supervision: The backbone is trained to predict discrete "FAST" tokens of action chunks. This provides a direct training signal to the backbone, helping it learn action-informative representations from human data where joint labels are missing.
- Dual-Stream Action Expert: A 0.6B parameter transformer using flow-matching to output two parallel, executable streams:
- Joint Stream: Direct commands for the robot's 22-DoF hands and 7-DoF arms.
- EEF Stream: High-level wrist and fingertip targets, which are then converted to joint commands via Inverse Kinematics (IK).
Datasets
The model was pretrained on a massive mix of data:
- Human Data: 2.8k hours of egocentric recordings covering 320 everyday tasks.
- Robot Data: 228 hours of bimanual teleoperation data (105 tasks) on a robot with two 7-DoF arms and two 22-DoF dexterous hands.
- Evaluation Benchmarks: Real-robot testing on 6 tasks with robot demonstrations (e.g., inserting a USB, opening a cabinet) and 3 "human-to-robot" (H2R) tasks with zero robot demonstrations. They also evaluated on 24 tasks in the RoboCasa simulation environment.
Key Insights and Ideas
- Discrete Tokens are the "Secret Sauce": Continuous regression gradients often get "lost" in the action decoder. Predicting discrete tokens forces the backbone to organize its internal representations around task-relevant actions, significantly boosting the benefit of human pretraining.
- The Best of Both Worlds: The authors found that the Joint Stream is superior for fine-grained control when robot data is available, while the EEF Stream is far better at transferring skills from human videos because it abstracts away the robot's specific joint configuration.
- Zero-Robot-Data Learning: Because SHAPE learns a shared representation of "action" across humans and robots, it can watch a human perform a new task and then execute it on the robot using the EEF stream, even if it has never seen a robot demonstration for that specific task.
Key Results
- Real-Robot Performance: On six complex bimanual tasks, SHAPE achieved a score of 0.80, outperforming the previous state-of-the-art $\pi_{0.5}$ (0.63) and NVIDIA’s GR00T N1.7 (0.51).
- Human-to-Robot Transfer: In tasks with zero robot demonstrations, SHAPE achieved a 35% success rate, whereas models trained only on robot data failed completely (0% success).
- Simulation Success: In the RoboCasa benchmark, SHAPE reached a 72.2% overall success rate, significantly higher than baselines.
Limitations and Future Work
While SHAPE is a significant step forward, some challenges remain:
- Precision Bottlenecks: Extremely fine-grained tasks, like millimeter-tolerance connector insertion, still have low completion rates (around 10-15%).
- Transfer Success Rate: While 35% success on zero-robot-data tasks is a breakthrough, it is still far from the reliability needed for production environments.
- Scaling: The authors suggest that scaling to even larger human video corpora and a wider variety of robot embodiments is the next logical step to improve generalization.
Summary updated