RRL: Resnet as representation for Reinforcement Learning

Rutav M. Shah, Vikash Kumar

Introduction

In an ideal simulation, a reinforcement learning (RL) agent knows the exact coordinates of every object in its environment. In the real world, however, a robot only has access to its cameras and joint sensors. This "pixel-to-action" challenge is notoriously difficult for two main reasons:

  1. High Dimensionality: Images contain thousands of pixels, most of which are irrelevant to the task at hand (like the color of the background wallpaper).
  2. Non-stationarity: In standard "end-to-end" RL, the robot tries to learn a visual representation while simultaneously learning a policy. This creates a "moving target" problem where visual features change as the robot explores, often leading to unstable training and extreme sample inefficiency.

Existing solutions, such as Variational Autoencoders (VAEs) or contrastive learning, often produce brittle representations that fail if lighting changes slightly or if the robot moves to a new environment.

RRL (Resnet as representation for Reinforcement Learning) demonstrates a surprisingly simple alternative: a frozen ResNet-34 pre-trained on ImageNet provides a remarkably effective visual representation for complex robotic tasks. By decoupling "learning to see" from "learning to act," RRL allows robots to learn contact-rich manipulation directly from pixels with a sample efficiency that rivals methods using perfect state information.

The RRL Approach

The authors propose a refreshingly straightforward architecture. Instead of training a custom visual encoder from scratch, they take a standard ResNet-34 pre-trained on the ImageNet dataset.

The RRL framework: Images are processed by a frozen ResNet pre-trained on ImageNet, whose features are combined with proprioceptive data to train an RL policy.

They remove the final classification layer, leaving a 512-dimensional feature vector, and—crucially—they freeze all the weights. The robot’s "state" at any given time is simply the concatenation of these frozen visual features and its own internal proprioceptive data (joint positions). To train the policy, they use DAPG (Demo Augmented Policy Gradient), which leverages a small number of human demonstrations (about 25) to "warm up" the agent before letting it learn through trial and error.

Datasets and Evaluation

The visual encoder is pre-trained on the standard ImageNet dataset (1.28 million images across 1,000 classes). For evaluation, the primary benchmark is the ADROIT manipulation suite, which features high-dimensional, contact-rich tasks:

  • Hammer: Picking up a hammer and hitting a nail.
  • Door: Unlatching and opening a door.
  • Pen: Rotating a pen to a target orientation in-hand.
  • Relocate: Picking up a ball and moving it to a target location.

The authors also tested RRL on the DeepMind Control Suite (DMControl), which contains simpler, planar tasks like Cartpole and Cheetah, to establish a baseline comparison.

Key Insights and Ideas

  • The "Generalization" Surprise: It is non-obvious that a network trained to distinguish between "Golden Retrievers" and "School Buses" would be effective at helping a robot hand pick up a pen. However, the authors find that the visual features learned from the vast diversity of ImageNet are general enough to cover the visual distributions a robot might encounter.
Feature activation maps demonstrating that representations learned on ImageNet successfully highlight relevant objects in complex robotic manipulation tasks.
  • Stability through Freezing: By freezing the encoder, the authors eliminate the non-stationarity problem. The visual "language" the robot uses to describe the world stays consistent throughout the entire training process.
  • Diversity over Architecture: The authors tested various architectures (ShuffleNet, MobileNet, ResNet-50). They found that while larger models help slightly, the real power comes from the diversity of the pre-training data, not the specific neural wiring.
  • Robustness to Distractors: Because ResNet was trained on real-world photos with varying backgrounds and lighting, RRL-based policies are significantly more robust to visual "noise" (like changing the color of an object or shifting lights) than methods that learn representations from scratch in a simulator.

Key Results

  • Performance: On the complex ADROIT tasks, RRL achieved performance nearly identical to "Oracle" agents that had access to the true underlying state (exact coordinates).
  • Efficiency: RRL was roughly 5x more compute-efficient than FERM, a state-of-the-art method that attempts to learn representations and policies simultaneously.
  • The Complexity Paradox: Interestingly, on simple tasks (DMControl), RRL performed worse than task-specific methods. This suggests that for easy tasks, "general" vision is overkill and adds unnecessary noise, but for "real-world" complexity, it becomes a superpower.

Limitations and Future Work

While highly effective, RRL has limitations. ResNet is trained on static images, whereas embodied agents might benefit more from representations that understand temporal dynamics or "interactivity." Additionally, while ImageNet is diverse, it is human-centric; there may still be a "domain gap" between third-person photos and the specific egocentric viewpoints of a robot.

Ultimately, the success of RRL hints at a future where robotics relies on a "standard" visual backbone—similar to how NLP utilizes BERT or GPT—allowing researchers to focus entirely on the logic of movement rather than the mechanics of sight.

Summary updated

Abstract

Generalist robots capable of performing dexterous, contact-rich manipulation tasks will enhance productivity and provide care in un-instrumented settings like homes. Such tasks warrant operations in real-world only using the robot's proprioceptive sensor such as onboard cameras, joint encoders, etc which can be challenging for policy learning owing to the high dimensionality and partial observability issues. We propose RRL: Resnet as representation for Reinforcement Learning -- a straightforward yet effective approach that can learn complex behaviors directly from proprioceptive inputs. RRL fuses features extracted from pre-trained Resnet into the standard reinforcement learning pipeline and delivers results comparable to learning directly from the state. In a simulated dexterous manipulation benchmark, where the state of the art methods fails to make significant progress, RRL delivers contact rich behaviors. The appeal of RRL lies in its simplicity in bringing together progress from the fields of Representation Learning, Imitation Learning, and Reinforcement Learning. Its effectiveness in learning behaviors directly from visual inputs with performance and sample efficiency matching learning directly from the state, even in complex high dimensional domains, is far from obvious.
BibTeX
@inproceedings{shah2021rrl,
  author = {Rutav M. Shah and Vikash Kumar},
  title = {{RRL: Resnet as representation for Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure