Introduction
In computer vision and natural language processing, the standard approach is to start with a powerful pre-trained model (like a ResNet or BERT) and adapt it to a specific task. However, in the world of robotic control, the dominant paradigm remains "tabula-rasa" learning: training a visuo-motor policy entirely from scratch using only data from the specific environment where the robot operates.
This "from scratch" approach is incredibly data-hungry, often requiring millions or billions of interactions. This paper asks a fundamental question: Why haven't pre-trained vision models found the same success in control as they have in other AI fields? Is the visual domain gap too large, or are we simply missing the right way to utilize these models?
The authors demonstrate that frozen vision models pre-trained on standard computer vision datasets (like ImageNet) can indeed serve as highly effective, "off-the-shelf" perception modules for complex robotic control tasks. By strategically combining features from different layers of a self-supervised model, these representations can match or even outperform ground-truth state information, challenging the long-standing tabula-rasa paradigm in reinforcement and imitation learning.
Methods: The PVR Paradigm
The authors propose a paradigm shift: instead of training the vision encoder as part of the policy, they use a Pre-trained Visual Representation (PVR).
Key aspects of their approach include:
- Frozen Encoders: The vision model is never updated during policy training. This simplifies the architecture and makes training significantly more stable.
- Model Selection: The study evaluates several architectures, including standard Supervised Learning (SL) ResNets, Self-Supervised Learning (SSL) models like MoCo, and multi-modal models like CLIP.
- Full-Hierarchy Features: A major contribution is the investigation of feature hierarchies. Instead of relying solely on the final layer of a network (which captures high-level semantics), the authors experiment with concatenating features from multiple intermediate layers (e.g., layers 3, 4, and 5 of a ResNet-50).
- Policy Training: Policies are trained using Imitation Learning (Behavioral Cloning) on expert trajectories, which is more data-efficient than Reinforcement Learning for evaluating a large number of models.
Datasets and Evaluation Domains
To ensure the findings are robust, the study spans a diverse set of benchmarks:
- Pre-training Datasets: ImageNet (object-centric) and Places (scene-centric).
- Evaluation Domains:
- Habitat: Indoor navigation using photorealistic scans of real-world homes.
- DeepMind Control (DMC) Suite: Classic locomotion tasks like "Cheetah-Run" and "Walker-Walk."
- Adroit: Highly complex dexterous manipulation using a 28-degree-of-freedom anthropomorphic hand.
- Franka Kitchen: Multi-stage manipulation tasks involving kitchen objects.
Key Insights
- SSL Outperforms Supervised Learning: Self-supervised models (specifically MoCo) consistently provide better features for control than supervised models. This suggests that the "instance discrimination" task in SSL captures more useful spatial information than standard category labels.
- The Power of the Crop: In SSL pre-training, crop augmentations are the most critical factor. They provide translation and occlusion invariance, helping the model focus on the relative displacement between objects and the agent—a vital cue for control.
- Layer Specialization: There is a clear "Goldilocks" effect for layers. Fine-grained physics tasks (like those in MuJoCo) benefit from earlier layers that preserve spatial detail. Semantic tasks (like navigation in Habitat) prefer later layers that understand "what" and "where" objects are.
- Universal Representations: By concatenating features from multiple layers (the "Full-Hierarchy" model), the authors created a single PVR that works across all domains. This suggests it is possible to build "near-universal" perception modules for robotics.
Key Results
The results serve as a striking validation of the pre-training paradigm for control:
- Beating Ground-Truth: In the Franka Kitchen domain, the PVR actually outperformed ground-truth state features (87% success vs. 71%).
- Matching the "Oracle": In Habitat and DMC, the PVR matched the performance of policies trained with perfect state information.
- Failure of "From Scratch": Across every single test, the PVR significantly outperformed the traditional approach of training a vision model from scratch with the same amount of data.
Limitations and Future Work
While the results are highly promising, the authors note several areas for further exploration:
- Freezing vs. Fine-tuning: This study focused exclusively on frozen models to isolate the quality of the representation. Future work could investigate if "on-the-fly" fine-tuning provides additional performance gains.
- Reinforcement Learning: Due to the computational cost of evaluating dozens of models, the authors primarily used Imitation Learning. Validating these insights in a pure RL setting is a logical next step.
- Real-World Deployment: While Habitat uses real-world scans, all evaluations were performed in simulation. Testing the "sim-to-real" transfer of these PVRs on physical hardware remains the ultimate test for their utility.
Summary updated