Abstract
Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pretrained representations that will lead to agents that generalize.
Summary (TL;DR)
By training over 10,000 RL policies across 240 different visual representations, the authors show that a simple, cheap proxy task can accurately predict how well an RL agent will generalize to new environments. They also demonstrate that robust pretrained representations allow for zero-shot sim-to-real transfer without the need for traditional domain randomization.
Introduction
Reinforcement Learning (RL) agents are notoriously brittle. An agent trained to move a red cube in a simulator might completely fail if the cube turns blue or if it is deployed on a real-world robot. This "out-of-distribution" (OOD) generalization is a major hurdle for robotics. In this study, researchers conduct a massive empirical investigation to understand how pretrained visual representations—the "eyes" of the agent—influence its ability to handle these shifts.
What Problem Does This Paper Address?
The standard RL pipeline often involves learning a task from scratch using raw pixels. This is sample-inefficient and leads to agents that overfit to specific visual features of their training environment.
A promising alternative is to use pretrained representations: first, train an encoder (like a VAE) on a large dataset of images to learn a low-dimensional "latent space," then train the RL agent using these latent vectors instead of raw pixels. However, it hasn't been clear which properties of these representations (e.g., disentanglement, informativeness, or reconstruction accuracy) actually help an agent generalize when the environment changes.
Methodology
The authors used a robotic manipulation setup (the TriFinger platform) involving a robotic finger and a cube. Their experimental pipeline followed three steps:
- Pretrain Representations: They trained 240 different Variational Autoencoders (VAEs), varying hyperparameters like the $\beta$ factor (which controls disentanglement) and latent space size.
- Train RL Policies: They trained over 11,000 policies using Soft Actor-Critic (SAC) for two tasks: reaching an object and pushing an object. Crucially, the pretrained encoder was kept frozen during RL training.
- Evaluate Generalization: They tested the agents in four scenarios: in-distribution, OOD1 (colors seen by encoder but not agent), OOD2-sim (unseen colors and shapes), and OOD2-real (zero-shot transfer to a physical robot).
Key Insights and Results
- The Power of Proxy Tasks: The most significant finding is that you don't need to train a full RL agent to know if a representation is good. A simple "Generalization Score" (GS)—measured by how well a tiny MLP can predict ground-truth factors (like cube position) from the representation in OOD settings—is a highly reliable predictor of RL performance.
- Disentanglement vs. Informativeness: While "disentangled" representations are popular, the authors found they don't inherently improve RL generalization. Their perceived benefit usually stems from being more "informative" overall. Once you control for informativeness, the advantage of disentanglement vanishes.
- Sim-to-Real Success: Many agents showed surprising robustness to the sim-to-real gap. By selecting representations with high proxy task scores, the authors achieved an 80% success rate on the physical TriFinger robot, despite being trained entirely in simulation without domain randomization or fine-tuning.
Conclusion
This research demonstrates that robust pretrained representations allow for zero-shot sim-to-real transfer and that simple proxy tasks can predict downstream RL performance. While the study focused on VAEs and frozen encoders, it provides a strong foundation for using pretrained visual models to build more generalizable robotic agents.
Summary updated