Reinforcement learning (RL) agents can achieve superhuman performance on specific tasks, but they are notoriously brittle. An agent trained to navigate a specific environment will often fail completely if a background color changes or an obstacle moves by just a few pixels. This paper, a contribution to ICLR 2021 from the Google Research Brain Team, tackles this "holy grail" of RL: generalization.
To address this, the authors introduce Policy Similarity Embeddings (PSEs), a method that forces an RL agent to learn representations based on behavioral similarity rather than just visual features. By ensuring that states requiring similar optimal actions are mapped close together in embedding space, PSEs significantly improve an agent's ability to generalize to unseen environments.
The Challenge of Generalization
In supervised learning, massive datasets help models generalize. In RL, agents often train on a limited set of environments and tend to "overfit" to specific visual textures or configurations.
Current solutions usually borrow from computer vision—such as data augmentation (cropping, rotating) or regularization (dropout). However, these methods ignore the fundamental sequential nature of RL. The authors argue that a truly robust representation should be based on the mechanics of the task. For example, in a jumping game, the agent's representation should prioritize its distance to the obstacle, regardless of whether the floor is blue or the background is a video of a busy street.
Methodology: Learning Behavioral Similarity
The paper introduces two core components to achieve this robust representation:
- Policy Similarity Metric (PSM): This is a novel way to measure how "similar" two states are. Inspired by bisimulation metrics, PSM defines two states as similar if the optimal policy would take similar actions in those states and in all future states resulting from those actions. Crucially, PSM is reward-agnostic. It focuses on behavior rather than just chasing a scalar reward signal, making it much more robust to superficial environment changes.
- Contrastive Metric Embeddings (CMEs): To utilize this metric, the authors adapt contrastive learning (specifically the SimCLR framework). They train a neural network to produce embeddings where the distance between two states in the embedding space matches their PSM distance.
By combining these, the authors create PSEs. During training, the agent performs its normal RL task while simultaneously minimizing a contrastive loss that organizes its internal "map" of the world based on behavioral similarity.
Datasets and Evaluation
The authors evaluated PSEs on three distinct benchmarks designed to test generalization:
- Jumping Task from Pixels: A custom environment where a block must jump over an obstacle. Generalization is tested by changing floor heights and obstacle positions not seen during training.
- LQR with Spurious Correlations: A classic control task (Linear Quadratic Regulator) modified with high-dimensional "distractor" features that are correlated with actions during training but change during testing.
- Distracting DM Control Suite: A standard set of continuous control tasks (like "Cheetah Run") overlaid with dynamic, high-variance video backgrounds that make visual generalization extremely difficult.
Key Insights and Results
The core "aha moment" of this research is that while pixels change, the optimal behavior often remains the same across semantically similar tasks. Mapping states to actions provides a "ground truth" for similarity that is more stable than raw visual input. Furthermore, because PSEs are orthogonal to standard data augmentation, combining them yields the best results.
Key performance highlights include:
- Jumping Task: On the most difficult "Narrow" grid (which tests extrapolation to unseen obstacle placements), the combination of RandConv augmentation + PSEs solved 52.4% of test tasks, compared to only 33.7% for RandConv alone.
- LQR: PSM achieved near-optimal generalization (error of 0.03), while other popular out-of-distribution methods like IPO or Weight Sparsity failed significantly (errors > 24.0).
- DM Control Suite: In the "Distracting" suite, adding PSEs to the top-performing DrQ baseline led to substantial gains across almost all tasks. In "Ball in Cup Catch" and "Cheetah Run," it nearly doubled the baseline score in some configurations.
Limitations and Future Work
While highly effective, the approach has limitations. The current formulation of PSM works best when you have access to (or can approximate) the optimal policy during training; in sparse-reward environments where the agent never finds the optimal path, calculating PSM is difficult. Additionally, computing the PSM involves dynamic programming, which can be computationally expensive for very large state spaces. Finally, while the theory covers stochasticity, empirical evaluations focused largely on deterministic transitions, leaving room for future exploration in noisy environments.
Overall, this paper provides a principled, theoretically grounded way to move RL representations away from "pixel-pushing" and toward a deeper, more generalizable understanding of task dynamics.
Summary updated