The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agents

Andrea Dittadi, Frederik Träuble, Manuel Wüthrich, Felix Widmaier, Peter Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf, Sebastian Bauer

Abstract

Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pretrained representations that will lead to agents that generalize.

Introduction

Reinforcement learning (RL) agents often exhibit extreme brittleness when faced with out-of-distribution (OOD) scenarios. An agent trained to manipulate a specific object in a controlled simulation may fail entirely if the object's color changes, lighting shifts, or if the policy is deployed on a physical robot. This lack of robustness stems from the difficulty of learning concise, generalizable representations of the environment directly from high-dimensional visual input.

This study investigates whether pretraining visual representations—teaching a model "what the world looks like" before teaching it "how to act"—can mitigate these issues. Specifically, we explore which properties of these representations facilitate OOD generalization and whether we can predict an agent's success using computationally inexpensive proxy tasks.

Experimental Methodology

The research utilizes a massive two-stage pipeline to evaluate the relationship between representation properties and RL performance across the TriFinger and CausalWorld platforms.

Overview of the experimental pipeline: 1) Representation pretraining, 2) Downstream RL training with frozen encoders, and 3) Evaluation across various OOD settings.

1. Representation Pretraining

We trained 240 distinct Variational Autoencoders (VAEs) and $\beta$-VAEs on a dataset of robotic images. These models learn to compress high-dimensional visual data into low-dimensional latent vectors ($z$).

2. Downstream RL Training

Using the Soft Actor Critic (SAC) algorithm, we trained over 11,000 RL policies. During this phase, the visual encoder from the pretraining stage remained frozen. The RL agent learned to map the latent features ($z_t$) and robot state variables ($x_t$) to actions ($a_t$).

3. The Proxy Task: Generalization Score (GS)

To circumvent the high computational cost of training thousands of RL agents for every test case, we developed a "Generalization Score." This score is derived from a simple Multi-Layer Perceptron (MLP) trained to predict ground-truth factors (e.g., object coordinates) from the representation. We then tested the correlation between this MLP's generalization ability and the full RL agent's OOD performance.

Key Insights and Results

The Predictive Power of Proxy Tasks

A primary finding is that the Generalization Score (GS) serves as a reliable predictor for RL performance across diverse OOD scenarios. This suggests that researchers can select optimal representations using simple supervised learning tasks rather than exhaustive RL training, significantly reducing compute requirements.

Disentanglement vs. Information Content

While "disentangled" representations—where individual latent dimensions correspond to specific physical properties—are often hypothesized to improve generalization, our results suggest their impact is secondary. Once the total information content of a representation is accounted for, disentanglement itself does not significantly enhance RL robustness.

Robustness and Sim-to-Real Transfer

The agents demonstrated remarkable zero-shot sim-to-real generalization. Policies trained exclusively in the CausalWorld simulator successfully operated on the physical TriFinger robotic platform without fine-tuning or traditional domain randomization. Furthermore, agents trained only on cubes generalized well to novel shapes like spheres, maintaining a 95% success rate on familiar colors and nearly 70% on novel color-shape combinations.

Conclusion

This work demonstrates that pretrained representations are a viable path toward robust, generalizable robotic agents. By identifying that simple proxy tasks can predict downstream RL success, we provide a framework for more efficient model selection. Future research should investigate whether these findings extend beyond VAEs to contrastive learning or masked autoencoders, and explore the effects of fine-tuning encoders during the RL phase.

Summary updated

Abstract

Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pretrained representations that will lead to agents that generalize.
BibTeX
@inproceedings{dittadi2022role,
  author = {Andrea Dittadi and Frederik Träuble and Manuel Wüthrich and Felix Widmaier and Peter Gehler and Ole Winther and Francesco Locatello and Olivier Bachem and Bernhard Schölkopf and Sebastian Bauer},
  title = {{The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agents}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=c2Ft8FYUVp},
}
Figure