Abstract
Learning robot manipulation directly from raw visual input is a significant challenge due to the high cost of data collection and the complexity of physical interactions. While pre-trained visual models from the computer vision community offer a starting point, they often lack the specific temporal and interaction-related knowledge necessary for robotics. This paper introduces Vi-PRoM (Visual Pre-training for Robot Manipulation), a two-stage pre-training scheme, and EgoNet, a massive egocentric video dataset. By combining contrastive learning with temporal and semantic supervision, the proposed approach significantly outperforms existing state-of-the-art models in both simulated benchmarks and real-world robotic tasks.
Introduction
In recent years, the field of robot manipulation has moved toward learning policies directly from pixels. However, training these models from scratch requires an enormous amount of interaction data, which is expensive and difficult to scale. While the computer vision community has successfully leveraged models pre-trained on datasets like ImageNet, these models are often optimized for static object recognition rather than the dynamic, first-person interactions required by robots.
Researchers from ByteDance, Southeast University, and Tsinghua University address this gap by systematically benchmarking datasets, architectures, and training objectives. Their goal is to identify the optimal "recipe" for visual pre-training tailored specifically for robotic manipulation, leading to the development of the Vi-PRoM framework.
The Vi-PRoM Methodology
The proposed Vi-PRoM framework utilizes a two-stage pre-training strategy designed to instill both general visual understanding and specific robotic "intuition."
Stage 1: Contrastive Self-Supervised Learning
The first stage employs the MoCo-v3 framework for contrastive learning. By pulling similar images together and pushing dissimilar ones away in a high-dimensional feature space, the model learns to distinguish between various scenes and objects without requiring manual labels. This provides a robust foundation for general visual representation.
Stage 2: Multi-Task Supervised Fine-Tuning
To inject task-relevant intelligence, the authors introduce two auxiliary tasks that do not require human-annotated labels:
- Visual Semantics: Using a frozen ResNet-101 model to generate "pseudo-labels," the model is taught to recognize object categories within the EgoNet dataset, grounding the visual features in semantic meaning.
- Temporal Dynamics: The model is tasked with predicting the original sequence of scrambled video frames. This forces the architecture to understand the physical "flow" of actions and the temporal progression of tasks.
For the final policy learning, these pre-trained features are integrated into Behavior Cloning (BC) and Proximal Policy Optimization (PPO) pipelines to translate visual inputs into motor commands.
The EgoNet Dataset
A core contribution of this work is EgoNet, a benchmark dataset curated specifically for robotic pre-training. Derived from the Ego4D dataset, EgoNet contains approximately 503,000 video clips (totaling 1.5 million frames). Unlike ImageNet, which consists of third-person static images, EgoNet focuses on egocentric (first-person) views of human-object interactions. This perspective closely mimics the visual input a robot receives during manipulation tasks, providing more relevant data for learning how to interact with the environment.
Key Insights and Experimental Findings
The researchers' systematic benchmarking revealed several critical insights for the future of robotic vision:
- The Superiority of Egocentric Data: Pre-training on EgoNet proved far more effective than pre-training on ImageNet. The interaction-related knowledge inherent in human video clips translates more effectively to robotic tasks than static images of unrelated categories.
- Architecture Efficiency: Despite the industry-wide shift toward Vision Transformers (ViTs), the study found that a standard ResNet-50 architecture remains the "sweet spot" for manipulation tasks. Larger models, such as ResNet-101, did not yield significant performance gains, suggesting that current robotic tasks may not yet benefit from the increased capacity of massive ViTs.
- Contrastive vs. Masked Modeling: Contrastive learning objectives outperformed Masked Image Modeling (MIM). The authors suggest that for robotics, understanding visual semantics and object relationships is more critical than the low-level structural reconstruction emphasized by MIM.
- The Importance of Time: Explicitly training for temporal dynamics was a major factor in success, particularly in multi-stage environments like the Franka Kitchen, where the order of operations is vital.
Results
Vi-PRoM demonstrated superior performance across multiple benchmarks:
- Franka Kitchen: Using Behavior Cloning, Vi-PRoM achieved a 43.8% success rate, significantly outperforming the previous state-of-the-art model, R3M (27.4%).
- MetaWorld: In Reinforcement Learning (PPO) tests, Vi-PRoM reached a 46.8% success rate, compared to 38.6% for R3M.
- Real-World Deployment: The model was successfully validated on a physical 6-DoF robot arm. It performed tasks such as opening and closing drawers and doors, proving that the features learned from human videos and simulation are robust enough for real-world application.
Conclusion and Future Work
This research provides a clear roadmap for visual pre-training in robotics, highlighting the importance of egocentric data and multi-stage training objectives. While Vi-PRoM represents a significant step forward, the authors acknowledge the ongoing challenge of the "sim-to-real" gap and the need for even larger-scale datasets. Future work will likely focus on scaling pre-training to even larger raw video collections and developing more standardized real-world benchmarks to move beyond the limitations of current simulators.
Summary updated