RRL: Resnet as representation for Reinforcement Learning

Rutav Shah, Vikash Kumar

TL;DR

This paper introduces RRL, a simple yet effective method that leverages pre-trained ResNet features for reinforcement learning (RL) in complex robotic manipulation tasks. RRL demonstrates that using general-purpose visual representations allows robots to learn sophisticated behaviors directly from raw sensor inputs, matching or exceeding state-of-the-art methods even in challenging, high-dimensional scenarios.

What Problem Does This Paper Address?

Developing robots capable of performing dexterous, contact-rich manipulation tasks in un-instrumented environments (like homes) is a significant challenge. These tasks require robots to operate using only their proprioceptive sensors (cameras, joint encoders), which generate high-dimensional and partially observable data. Traditional RL methods struggle with this complexity, often requiring extensive manual engineering, task-specific representations, or operating in simplified, low-dimensional settings. The core problem is learning robust and efficient policies for complex real-world manipulation tasks directly from raw sensory data without relying on perfect state information or extensive environment instrumentation.

Methods

RRL (ResNet as Representation for Reinforcement Learning) is a straightforward approach that integrates pre-trained ResNet features into a standard RL pipeline. The key idea is to use a ResNet model, pre-trained on a massive dataset like ImageNet, as a fixed feature extractor. This extractor transforms raw visual inputs into a compact, low-dimensional representation. These features are then combined with other proprioceptive sensor data (like joint encoder readings) and fed into a standard RL algorithm (specifically, Natural Policy Gradient, NPG, augmented with demonstrations). This decouples representation learning from policy learning, allowing the policy to benefit from general, robust visual features without the instability of learning representations and policies simultaneously.

Datasets

  • ImageNet: Used for pre-training the ResNet feature extractor.
  • ADROIT Manipulation Suite: A benchmark consisting of complex, high-dimensional dexterous manipulation tasks (e.g., object relocation, in-hand manipulation, tool use). This suite is highlighted as being more representative of real-world challenges than simpler benchmarks.
  • MJRL Environments: Includes lower-dimensional tasks like Reacher and Peg Insertion, used for comparison.
  • DMControl Suite: A standard benchmark for visual RL, used to highlight the limitations of task-specific representations learned on simpler datasets.

Key Insights and Ideas

  • General Representations for Specific Tasks: Pre-training a feature extractor on a vast and diverse dataset (like ImageNet) yields representations that are surprisingly effective for downstream RL tasks, even with significant domain shifts. This suggests that "task-agnostic" representations can be highly beneficial.
  • Robustness to Distractors: Using pre-trained ResNet features makes the learned policies more robust to visual distractors (lighting changes, clutter, object color variations) compared to task-specific representations.
  • Simplicity and Scalability: RRL's strength lies in its simplicity. By leveraging existing, powerful pre-trained models, it avoids complex representation learning pipelines and can be easily integrated with various RL algorithms.
  • Bridging Representation and Reinforcement Learning: RRL effectively combines progress from Representation Learning, Imitation Learning, and Reinforcement Learning, demonstrating a path towards more generalizable robotic agents.
  • Benchmark Limitations: The paper argues that simpler benchmarks like DMControl might not adequately capture the complexity of real-world manipulation, leading to task-specific representations that fail to generalize.

Key Results

  • Competitive Performance on Complex Tasks: RRL achieved performance comparable to or exceeding state-of-the-art methods (including demonstration-assisted ones like DAPG) on the challenging ADROIT manipulation suite, often approaching the performance of methods using privileged state information.
  • Efficiency: RRL was found to be significantly more compute-efficient than competing methods like FERM.
  • Robustness: RRL policies showed resilience to visual distractors and even noisy proprioceptive inputs.
  • Generalization: Different pre-trained feature extractors (ShuffleNet, MobileNet) also performed well, reinforcing the idea that the diversity of the pre-training dataset is key, not necessarily the specific architecture.
  • Limitations on Simple Tasks: On simpler, low-dimensional tasks like Reacher, RRL initially struggled compared to methods that learned task-specific representations, suggesting that for very simple tasks, specialized representations might be more efficient.

Limitations and Future Work

  • Data Mismatch: The paper acknowledges a potential mismatch between the human-centric environments used for pre-training ResNet and the specific morphologies and operational modes of robots.
  • Static Scene Bias: Current pre-trained models are often trained on static scenes, while embodied agents require features for dynamic and interactive movements.
  • Future Work: The authors suggest exploring shared representational spaces for robotics, leveraging data from various sources, and developing hardware-accelerated solutions for efficient feature compression and low-latency transmission. They also highlight the need for more realistic benchmarks that capture the complexity of real-world tasks.

Summary updated

Abstract

Generalist robots capable of performing dexterous, contact-rich manipulation tasks will enhance productivity and provide care in un-instrumented settings like homes. Such tasks warrant operations in real-world only using the robot's proprioceptive sensor such as onboard cameras, joint encoders, etc which can be challenging for policy learning owing to the high dimensionality and partial observability issues. We propose RRL: Resnet as representation for Reinforcement Learning -- a straightforward yet effective approach that can learn complex behaviors directly from proprioceptive inputs. RRL fuses features extracted from pre-trained Resnet into the standard reinforcement learning pipeline and delivers results comparable to learning directly from the state. In a simulated dexterous manipulation benchmark, where the state of the art methods fails to make significant progress, RRL delivers contact rich behaviors. The appeal of RRL lies in its simplicity in bringing together progress from the fields of Representation Learning, Imitation Learning, and Reinforcement Learning. Its effectiveness in learning behaviors directly from visual inputs with performance and sample efficiency matching learning directly from the state, even in complex high dimensional domains, is far from obvious.
BibTeX
@inproceedings{shah2021rrl,
  author = {Rutav Shah and Vikash Kumar},
  title = {{RRL: Resnet as representation for Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure