Introduction
In an ideal simulation, a reinforcement learning (RL) agent knows the exact coordinates of every object in its environment. In the real world, however, a robot only has access to its cameras and joint sensors. This "pixel-to-action" challenge is notoriously difficult for two main reasons:
- High Dimensionality: Images contain thousands of pixels, most of which are irrelevant to the task at hand (like the color of the background wallpaper).
- Non-stationarity: In standard "end-to-end" RL, the robot tries to learn a visual representation while simultaneously learning a policy. This creates a "moving target" problem where visual features change as the robot explores, often leading to unstable training and extreme sample inefficiency.
Existing solutions, such as Variational Autoencoders (VAEs) or contrastive learning, often produce brittle representations that fail if lighting changes slightly or if the robot moves to a new environment.
RRL (Resnet as representation for Reinforcement Learning) demonstrates a surprisingly simple alternative: a frozen ResNet-34 pre-trained on ImageNet provides a remarkably effective visual representation for complex robotic tasks. By decoupling "learning to see" from "learning to act," RRL allows robots to learn contact-rich manipulation directly from pixels with a sample efficiency that rivals methods using perfect state information.
The RRL Approach
The authors propose a refreshingly straightforward architecture. Instead of training a custom visual encoder from scratch, they take a standard ResNet-34 pre-trained on the ImageNet dataset.
They remove the final classification layer, leaving a 512-dimensional feature vector, and—crucially—they freeze all the weights. The robot’s "state" at any given time is simply the concatenation of these frozen visual features and its own internal proprioceptive data (joint positions). To train the policy, they use DAPG (Demo Augmented Policy Gradient), which leverages a small number of human demonstrations (about 25) to "warm up" the agent before letting it learn through trial and error.
Datasets and Evaluation
The visual encoder is pre-trained on the standard ImageNet dataset (1.28 million images across 1,000 classes). For evaluation, the primary benchmark is the ADROIT manipulation suite, which features high-dimensional, contact-rich tasks:
- Hammer: Picking up a hammer and hitting a nail.
- Door: Unlatching and opening a door.
- Pen: Rotating a pen to a target orientation in-hand.
- Relocate: Picking up a ball and moving it to a target location.
The authors also tested RRL on the DeepMind Control Suite (DMControl), which contains simpler, planar tasks like Cartpole and Cheetah, to establish a baseline comparison.
Key Insights and Ideas
- The "Generalization" Surprise: It is non-obvious that a network trained to distinguish between "Golden Retrievers" and "School Buses" would be effective at helping a robot hand pick up a pen. However, the authors find that the visual features learned from the vast diversity of ImageNet are general enough to cover the visual distributions a robot might encounter.
- Stability through Freezing: By freezing the encoder, the authors eliminate the non-stationarity problem. The visual "language" the robot uses to describe the world stays consistent throughout the entire training process.
- Diversity over Architecture: The authors tested various architectures (ShuffleNet, MobileNet, ResNet-50). They found that while larger models help slightly, the real power comes from the diversity of the pre-training data, not the specific neural wiring.
- Robustness to Distractors: Because ResNet was trained on real-world photos with varying backgrounds and lighting, RRL-based policies are significantly more robust to visual "noise" (like changing the color of an object or shifting lights) than methods that learn representations from scratch in a simulator.
Key Results
- Performance: On the complex ADROIT tasks, RRL achieved performance nearly identical to "Oracle" agents that had access to the true underlying state (exact coordinates).
- Efficiency: RRL was roughly 5x more compute-efficient than FERM, a state-of-the-art method that attempts to learn representations and policies simultaneously.
- The Complexity Paradox: Interestingly, on simple tasks (DMControl), RRL performed worse than task-specific methods. This suggests that for easy tasks, "general" vision is overkill and adds unnecessary noise, but for "real-world" complexity, it becomes a superpower.
Limitations and Future Work
While highly effective, RRL has limitations. ResNet is trained on static images, whereas embodied agents might benefit more from representations that understand temporal dynamics or "interactivity." Additionally, while ImageNet is diverse, it is human-centric; there may still be a "domain gap" between third-person photos and the specific egocentric viewpoints of a robot.
Ultimately, the success of RRL hints at a future where robotics relies on a "standard" visual backbone—similar to how NLP utilizes BERT or GPT—allowing researchers to focus entirely on the logic of movement rather than the mechanics of sight.
Summary updated