METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model

Yankai Fu, Ning Chen, Junkai Zhao, Shaozhe Shan, Guocai Yao, Pengwei Wang, Zhongyuan Wang, Shanghang ZhangarXiv 2025 · DOI · arXiv

Introduction

Dexterous manipulation—using multi-fingered hands to perform complex tasks like opening drawers or pouring drinks—remains a significant challenge in robotics. The primary hurdle is the scarcity of action-annotated data for complex hands, as teleoperating high-degree-of-freedom (DoF) robot hands is costly and slow. METIS addresses this by leveraging the abundance of human egocentric (first-person) videos to learn general motion priors, bridging the "embodiment gap" between human and robotic hands.

Methods: Integrated Reasoning and Acting

METIS is built upon a 7-billion parameter Large Language Model (LLaMA-2) and introduces three core innovations:

  1. Motion-aware Dynamics: To ensure stable motor control, METIS discretizes motion into tokens. It uses a VQ-VAE for visual dynamics and a Residual Quantization (RQ-VAE) for motion dynamics, capturing hierarchical patterns from coarse wrist movements to fine fingertip adjustments.
  2. Unified Action Space: By representing actions as 3D wrist poses and 3D fingertip positions, METIS creates an embodiment-agnostic representation. This allows the model to learn from both human and robotic data seamlessly.
  3. Chain-of-Thought Reasoning: The model generates high-level subtask descriptions (e.g., "Open the drawer") before outputting the specific motion tokens required for execution, integrating high-level reasoning with low-level control.
The METIS VLA Model architecture, illustrating the flow from visual and text inputs through the VLM to reasoning and motion-aware dynamics for action execution.

The EgoAtlas Dataset

The foundation of METIS is EgoAtlas, a massive dataset comprising 343,000 trajectories and nearly 90 million image-action pairs. It integrates existing human and robot datasets (such as ARCTIC, H2O, and EgoDex) and is supplemented by 10,000 high-fidelity human trajectories collected using a custom wearable system for precise finger and wrist tracking. This unified dataset allows the model to learn the intent of a movement regardless of whether the actor is a human or a specific robot model.

Key Insights

  • Embodiment Agnosticism: Focusing on fingertip positions in 3D space rather than joint angles allows the model to generalize across different robot hands. This is why METIS can be trained on humans and deployed on robots with different finger counts or lengths.
  • Sample Efficiency: Pretraining on human data significantly reduces the amount of robot-specific data needed. METIS achieved a 50% success rate on complex tasks with only 10% of the task-specific data, while models without human pretraining failed entirely.
  • Discretization Benefits: Turning complex 3D motion into discrete tokens allows the model to leverage the powerful pattern-recognition capabilities of pre-trained LLMs, treating "moving a finger" like "choosing a word."

Experimental Results

METIS was evaluated on a Unitree G1 humanoid robot across six real-world tasks, demonstrating state-of-the-art performance. The model achieved a 95% success rate in closing a laptop and 85% in pick-and-place tasks. Furthermore, it maintained high performance in out-of-distribution (OOD) scenarios, including cluttered backgrounds, changing lighting conditions, and novel objects.

Performance of METIS across diverse dexterous tasks and its ability to generalize to unseen backgrounds, lighting, objects, and even new robot embodiments.

Limitations and Future Work

While METIS shows remarkable promise, it currently relies solely on egocentric views, which can lead to occlusions where the robot's own arm or the object itself blocks the camera. Future work will explore integrating third-person perspectives to provide a more comprehensive view of the workspace. Additionally, expanding the EgoAtlas dataset to include the vast amount of third-person human activity data available online could further enhance the model's physical "common sense."

Summary updated

Abstract

Building a generalist robot that can perceive, reason, and act across diverse tasks remains an open challenge, especially for dexterous manipulation. A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills, as teleoperation is difficult and costly. Human data, with its vast scale and diverse manipulation behaviors, provides rich priors for learning robotic actions. While prior works have explored leveraging human demonstrations, they are often constrained by limited scenarios and a large visual gap between human and robots. To eliminate these limitations, we propose METIS, a vision-language-action (VLA) model for dexterous manipulation pretrained on multi-source egocentric datasets. We first construct EgoAtlas, which integrates large-scale human and robotic data from multiple sources, all unified under a consistent action space. We further extract motion-aware dynamics, a compact and discretized motion representation, which provides efficient and expressive supervision for VLA training. Built upon them, METIS integrates reasoning and acting into a unified framework, enabling effective deployment to downstream dexterous manipulation tasks. Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in six real-world tasks. Experimental results also highlight the superior generalization and robustness to out-of-distribution scenarios. These findings emphasize METIS as a promising step toward a generalist model for dexterous manipulation.
BibTeX
@misc{fu2025metis,
  author = {Yankai Fu and Ning Chen and Junkai Zhao and Shaozhe Shan and Guocai Yao and Pengwei Wang and Zhongyuan Wang and Shanghang Zhang},
  title = {{METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model}},
  year = {2025},
  eprint = {2511.17366},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.17366},
}
Figure