Abstract
Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pretrained representations that will lead to agents that generalize.
Introduction
Reinforcement learning (RL) agents often exhibit extreme brittleness when faced with out-of-distribution (OOD) scenarios. An agent trained to manipulate a specific object in a controlled simulation may fail entirely if the object's color changes, lighting shifts, or if the policy is deployed on a physical robot. This lack of robustness stems from the difficulty of learning concise, generalizable representations of the environment directly from high-dimensional visual input.
This study investigates whether pretraining visual representations—teaching a model "what the world looks like" before teaching it "how to act"—can mitigate these issues. Specifically, we explore which properties of these representations facilitate OOD generalization and whether we can predict an agent's success using computationally inexpensive proxy tasks.
Experimental Methodology
The research utilizes a massive two-stage pipeline to evaluate the relationship between representation properties and RL performance across the TriFinger and CausalWorld platforms.
1. Representation Pretraining
We trained 240 distinct Variational Autoencoders (VAEs) and $\beta$-VAEs on a dataset of robotic images. These models learn to compress high-dimensional visual data into low-dimensional latent vectors ($z$).
2. Downstream RL Training
Using the Soft Actor Critic (SAC) algorithm, we trained over 11,000 RL policies. During this phase, the visual encoder from the pretraining stage remained frozen. The RL agent learned to map the latent features ($z_t$) and robot state variables ($x_t$) to actions ($a_t$).
3. The Proxy Task: Generalization Score (GS)
To circumvent the high computational cost of training thousands of RL agents for every test case, we developed a "Generalization Score." This score is derived from a simple Multi-Layer Perceptron (MLP) trained to predict ground-truth factors (e.g., object coordinates) from the representation. We then tested the correlation between this MLP's generalization ability and the full RL agent's OOD performance.
Key Insights and Results
The Predictive Power of Proxy Tasks
A primary finding is that the Generalization Score (GS) serves as a reliable predictor for RL performance across diverse OOD scenarios. This suggests that researchers can select optimal representations using simple supervised learning tasks rather than exhaustive RL training, significantly reducing compute requirements.
Disentanglement vs. Information Content
While "disentangled" representations—where individual latent dimensions correspond to specific physical properties—are often hypothesized to improve generalization, our results suggest their impact is secondary. Once the total information content of a representation is accounted for, disentanglement itself does not significantly enhance RL robustness.
Robustness and Sim-to-Real Transfer
The agents demonstrated remarkable zero-shot sim-to-real generalization. Policies trained exclusively in the CausalWorld simulator successfully operated on the physical TriFinger robotic platform without fine-tuning or traditional domain randomization. Furthermore, agents trained only on cubes generalized well to novel shapes like spheres, maintaining a 95% success rate on familiar colors and nearly 70% on novel color-shape combinations.
Conclusion
This work demonstrates that pretrained representations are a viable path toward robust, generalizable robotic agents. By identifying that simple proxy tasks can predict downstream RL success, we provide a framework for more efficient model selection. Future research should investigate whether these findings extend beyond VAEs to contrastive learning or masked autoencoders, and explore the effects of fine-tuning encoders during the RL phase.
Summary updated