Introduction
Dexterous manipulation—using multi-fingered hands to perform complex tasks like opening drawers or pouring drinks—remains a significant challenge in robotics. The primary hurdle is the scarcity of action-annotated data for complex hands, as teleoperating high-degree-of-freedom (DoF) robot hands is costly and slow. METIS addresses this by leveraging the abundance of human egocentric (first-person) videos to learn general motion priors, bridging the "embodiment gap" between human and robotic hands.
Methods: Integrated Reasoning and Acting
METIS is built upon a 7-billion parameter Large Language Model (LLaMA-2) and introduces three core innovations:
- Motion-aware Dynamics: To ensure stable motor control, METIS discretizes motion into tokens. It uses a VQ-VAE for visual dynamics and a Residual Quantization (RQ-VAE) for motion dynamics, capturing hierarchical patterns from coarse wrist movements to fine fingertip adjustments.
- Unified Action Space: By representing actions as 3D wrist poses and 3D fingertip positions, METIS creates an embodiment-agnostic representation. This allows the model to learn from both human and robotic data seamlessly.
- Chain-of-Thought Reasoning: The model generates high-level subtask descriptions (e.g., "Open the drawer") before outputting the specific motion tokens required for execution, integrating high-level reasoning with low-level control.
The EgoAtlas Dataset
The foundation of METIS is EgoAtlas, a massive dataset comprising 343,000 trajectories and nearly 90 million image-action pairs. It integrates existing human and robot datasets (such as ARCTIC, H2O, and EgoDex) and is supplemented by 10,000 high-fidelity human trajectories collected using a custom wearable system for precise finger and wrist tracking. This unified dataset allows the model to learn the intent of a movement regardless of whether the actor is a human or a specific robot model.
Key Insights
- Embodiment Agnosticism: Focusing on fingertip positions in 3D space rather than joint angles allows the model to generalize across different robot hands. This is why METIS can be trained on humans and deployed on robots with different finger counts or lengths.
- Sample Efficiency: Pretraining on human data significantly reduces the amount of robot-specific data needed. METIS achieved a 50% success rate on complex tasks with only 10% of the task-specific data, while models without human pretraining failed entirely.
- Discretization Benefits: Turning complex 3D motion into discrete tokens allows the model to leverage the powerful pattern-recognition capabilities of pre-trained LLMs, treating "moving a finger" like "choosing a word."
Experimental Results
METIS was evaluated on a Unitree G1 humanoid robot across six real-world tasks, demonstrating state-of-the-art performance. The model achieved a 95% success rate in closing a laptop and 85% in pick-and-place tasks. Furthermore, it maintained high performance in out-of-distribution (OOD) scenarios, including cluttered backgrounds, changing lighting conditions, and novel objects.
Limitations and Future Work
While METIS shows remarkable promise, it currently relies solely on egocentric views, which can lead to occlusions where the robot's own arm or the object itself blocks the camera. Future work will explore integrating third-person perspectives to provide a more comprehensive view of the workspace. Additionally, expanding the EgoAtlas dataset to include the vast amount of third-person human activity data available online could further enhance the model's physical "common sense."
Summary updated