Representation Learning for Out-Of-Distribution Generalization in Reinforcement Learning

Andrea Dittadi, Frederik Träuble, Manuel Wüthrich, Felix Widmaier, Peter V. Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf, Stefan Bauer

Abstract

Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pretrained representations that will lead to agents that generalize.

Introduction

Deep Reinforcement Learning (RL) has achieved remarkable success in controlled environments, yet it remains notoriously brittle when faced with distribution shifts. An agent trained to manipulate a red cube in a simulator often fails entirely when the cube's color changes or when the agent is deployed on a physical robotic arm. This challenge, known as Out-Of-Distribution (OOD) generalization, is a primary bottleneck for real-world robotics.

While the community has long hypothesized that "good" representations—such as those derived from Variational Autoencoders (VAEs)—can enhance generalization, empirical evidence has been limited. Most evaluations rely on proxy tasks like reconstruction quality rather than actual performance in complex control tasks. This study addresses this gap through a massive-scale systematic evaluation of representation learning for robotic RL.

Experimental Methodology

The researchers employed a decoupled learning framework using the Trifinger robot platform. The process involved two distinct stages:

  1. Representation Learning: 240 different $\beta$-VAEs were trained on a dataset of 1 million images. These models compress high-dimensional camera observations into low-dimensional latent vectors. Crucially, these encoders remained frozen during the subsequent RL phase.
  2. Policy Learning: Soft Actor-Critic (SAC) was used to train RL policies. The agent's state was defined by the VAE's latent vector combined with raw robot variables (joint angles and velocities).

To ensure statistical rigor, the study involved training 11,520 policies across two tasks: ReachCube and Pushing. The evaluation spanned multiple levels of distribution shift, from simple color changes to full sim-to-real transfer.

Experimental evaluation settings: (a) in-distribution training, (b) OOD1 (new colors for encoder), (c) OOD2-sim (new colors for both), and (d) OOD2-real (sim-to-real transfer).

Key Insights on Representation Properties

Disentanglement vs. Informativeness

In representation learning, "disentanglement"—the separation of independent factors like color and position into distinct latent dimensions—is often prioritized. However, this study finds that once informativeness (the amount of relevant information captured) is accounted for, disentanglement provides negligible additional benefit for RL generalization.

The Role of Regularization and Noise

Adding L1 regularization to the policy network significantly improved OOD performance. By enforcing sparsity, the agent is forced to ignore latent "noise" and focus on the most critical features. Furthermore, introducing additive input noise during VAE training proved essential for robust sim-to-real transfer, acting as a form of implicit domain randomization.

Results and Sim-to-Real Transfer

The study achieved successful zero-shot transfer to a physical robot. Policies trained exclusively on simulated red cubes successfully reached and pushed green and blue cubes in the real world, achieving success rates of approximately 80% for reaching tasks.

A critical finding is the predictive power of proxy tasks. Performance on a simple downstream proxy task in simulation was found to correlate strongly with real-world robotic performance. This suggests that researchers can perform effective model selection in simulation to identify the representations most likely to succeed in the real world, bypassing the need for exhaustive physical testing.

Conclusion

By moving beyond anecdotal results to a systematic characterization of over 11,000 policies, this work demonstrates that specific simulation-based metrics can reliably predict OOD and sim-to-real performance. While the study utilized frozen encoders and static backgrounds, the findings provide a robust foundation for future work in end-to-end learning and more complex, dynamic environments. The results suggest that the path to generalizable robotics lies not just in the scale of data, but in the careful selection and regularization of the underlying world representations.

Summary updated

Abstract

Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pretrained representations that will lead to agents that generalize.
BibTeX
@inproceedings{dittadi2022representation,
  author = {Andrea Dittadi and Frederik Träuble and Manuel Wüthrich and Felix Widmaier and Peter V. Gehler and Ole Winther and Francesco Locatello and Olivier Bachem and Bernhard Schölkopf and Stefan Bauer},
  title = {{Representation Learning for Out-Of-Distribution Generalization in Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=c2Ft8FYUVp},
}
Figure