Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, Marc G. Bellemare

Reinforcement learning (RL) agents can achieve superhuman performance on specific tasks, but they are notoriously brittle. An agent trained to navigate a specific environment will often fail completely if a background color changes or an obstacle moves by just a few pixels. This paper, a contribution to ICLR 2021 from the Google Research Brain Team, tackles this "holy grail" of RL: generalization.

To address this, the authors introduce Policy Similarity Embeddings (PSEs), a method that forces an RL agent to learn representations based on behavioral similarity rather than just visual features. By ensuring that states requiring similar optimal actions are mapped close together in embedding space, PSEs significantly improve an agent's ability to generalize to unseen environments.

The Challenge of Generalization

In supervised learning, massive datasets help models generalize. In RL, agents often train on a limited set of environments and tend to "overfit" to specific visual textures or configurations.

Visual differences between training and testing environments in the Jumping Task.

Current solutions usually borrow from computer vision—such as data augmentation (cropping, rotating) or regularization (dropout). However, these methods ignore the fundamental sequential nature of RL. The authors argue that a truly robust representation should be based on the mechanics of the task. For example, in a jumping game, the agent's representation should prioritize its distance to the obstacle, regardless of whether the floor is blue or the background is a video of a busy street.

Methodology: Learning Behavioral Similarity

The paper introduces two core components to achieve this robust representation:

  1. Policy Similarity Metric (PSM): This is a novel way to measure how "similar" two states are. Inspired by bisimulation metrics, PSM defines two states as similar if the optimal policy would take similar actions in those states and in all future states resulting from those actions. Crucially, PSM is reward-agnostic. It focuses on behavior rather than just chasing a scalar reward signal, making it much more robust to superficial environment changes.
  2. Contrastive Metric Embeddings (CMEs): To utilize this metric, the authors adapt contrastive learning (specifically the SimCLR framework). They train a neural network to produce embeddings where the distance between two states in the embedding space matches their PSM distance.

By combining these, the authors create PSEs. During training, the agent performs its normal RL task while simultaneously minimizing a contrastive loss that organizes its internal "map" of the world based on behavioral similarity.

Datasets and Evaluation

The authors evaluated PSEs on three distinct benchmarks designed to test generalization:

  • Jumping Task from Pixels: A custom environment where a block must jump over an obstacle. Generalization is tested by changing floor heights and obstacle positions not seen during training.
  • LQR with Spurious Correlations: A classic control task (Linear Quadratic Regulator) modified with high-dimensional "distractor" features that are correlated with actions during training but change during testing.
  • Distracting DM Control Suite: A standard set of continuous control tasks (like "Cheetah Run") overlaid with dynamic, high-variance video backgrounds that make visual generalization extremely difficult.
Grid configurations for the Jumping Task, including the difficult 'Narrow' grid used to test extrapolation.

Key Insights and Results

The core "aha moment" of this research is that while pixels change, the optimal behavior often remains the same across semantically similar tasks. Mapping states to actions provides a "ground truth" for similarity that is more stable than raw visual input. Furthermore, because PSEs are orthogonal to standard data augmentation, combining them yields the best results.

Key performance highlights include:

  • Jumping Task: On the most difficult "Narrow" grid (which tests extrapolation to unseen obstacle placements), the combination of RandConv augmentation + PSEs solved 52.4% of test tasks, compared to only 33.7% for RandConv alone.
  • LQR: PSM achieved near-optimal generalization (error of 0.03), while other popular out-of-distribution methods like IPO or Weight Sparsity failed significantly (errors > 24.0).
  • DM Control Suite: In the "Distracting" suite, adding PSEs to the top-performing DrQ baseline led to substantial gains across almost all tasks. In "Ball in Cup Catch" and "Cheetah Run," it nearly doubled the baseline score in some configurations.

Limitations and Future Work

While highly effective, the approach has limitations. The current formulation of PSM works best when you have access to (or can approximate) the optimal policy during training; in sparse-reward environments where the agent never finds the optimal path, calculating PSM is difficult. Additionally, computing the PSM involves dynamic programming, which can be computationally expensive for very large state spaces. Finally, while the theory covers stochasticity, empirical evaluations focused largely on deterministic transitions, leaving room for future exploration in noisy environments.

Overall, this paper provides a principled, theoretically grounded way to move RL representations away from "pixel-pushing" and toward a deeper, more generalizable understanding of task dynamics.

Summary updated

Abstract

Reinforcement learning methods trained on few environments rarely learn policies that generalize to unseen environments. To improve generalization, we incorporate the inherent sequential structure in reinforcement learning into the representation learning process. This approach is orthogonal to recent approaches, which rarely exploit this structure explicitly. Specifically, we introduce a theoretically motivated policy similarity metric (PSM) for measuring behavioral similarity between states. PSM assigns high similarity to states for which the optimal policies in those states as well as in future states are similar. We also present a contrastive representation learning procedure to embed any state similarity metric, which we instantiate with PSM to obtain policy similarity embeddings (PSEs). We demonstrate that PSEs improve generalization on diverse benchmarks, including LQR with spurious correlations, a jumping task from pixels, and Distracting DM Control Suite.
BibTeX
@inproceedings{agarwal2021contrastive,
  author = {Rishabh Agarwal and Marlos C. Machado and Pablo Samuel Castro and Marc G. Bellemare},
  title = {{Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=He-ZyHSN6lq},
}
Figure