Abstract
Reinforcement learning methods trained on few environments rarely learn policies that generalize to unseen environments. To improve generalization, we incorporate the inherent sequential structure in reinforcement learning into the representation learning process. This approach is orthogonal to recent approaches, which rarely exploit this structure explicitly. Specifically, we introduce a theoretically motivated policy similarity metric (PSM) for measuring behavioral similarity between states. PSM assigns high similarity to states for which the optimal policies in those states as well as in future states are similar. We also present a contrastive representation learning procedure to embed any state similarity metric, which we instantiate with PSM to obtain policy similarity embeddings (PSEs). We demonstrate that PSEs improve generalization on diverse benchmarks, including LQR with spurious correlations, a jumping task from pixels, and Distracting DM Control Suite.
Introduction
Deep Reinforcement Learning (RL) agents often suffer from severe overfitting to their training environments. An agent trained to navigate a specific obstacle course frequently fails when faced with minor variations, such as shifted obstacle positions or altered background textures. While the computer vision community has successfully leveraged task-agnostic data augmentation (e.g., random cropping) to improve generalization, these methods do not explicitly account for the sequential decision-making nature of RL. In RL, the most critical feature of a state is often the sequence of actions required to achieve a goal. This work addresses the challenge of incorporating this behavioral structure directly into the agent's internal representation.
Methodology
We propose a two-step framework to learn representations that are robust to environmental distractors by focusing on long-term behavior.
Policy Similarity Metric (PSM)
We define a novel metric to quantify the similarity between two states based on the optimal policy. Unlike reward-based metrics like bisimulation, PSM considers two states similar if the optimal actions in those states—and all subsequent states reachable from them—are identical. This reward-agnostic approach ensures the metric focuses purely on the behavioral requirements of the task.
Contrastive Metric Embeddings (CMEs)
To implement this metric within a neural network, we utilize a contrastive learning objective. The agent is trained to map "behaviorally similar" states closer together in the embedding space while pushing "dissimilar" states apart. The resulting Policy Similarity Embeddings (PSEs) serve as a powerful auxiliary loss that can be integrated with standard RL algorithms like Soft Actor-Critic (SAC) or Deep Q-Networks (DQN).
Experimental Setup
The efficacy of PSEs was evaluated across three benchmarks designed to test generalization capabilities:
- Jumping Task from Pixels: A pixel-based environment where a block must jump over obstacles. Generalization is tested by introducing obstacle positions and floor heights not encountered during training.
- LQR with Spurious Correlations: A control task modified with 10,000 high-dimensional distractor features that correlate with optimal actions during training but change during testing.
- Distracting DM Control Suite: A standard benchmark where backgrounds are replaced with dynamic video sequences to distract the agent.
Results and Discussion
Our experiments reveal that behavior-based invariants are superior for generalization. In the Jumping Task, PSEs combined with data augmentation achieved an 87% success rate on test tasks, significantly outperforming the 50.7% achieved by data augmentation alone.
In the LQR task, PSEs demonstrated remarkable robustness, achieving near-optimal performance (0.02 error) despite 10,000 distractor features, whereas all baseline methods failed (errors > 24.0). Furthermore, on the Distracting DM Control Suite, PSEs consistently outperformed state-of-the-art methods like DrQ, particularly in tasks with complex video backgrounds like Finger Spin and Walker Walk.
A key insight from our "Jumping Task with Colors" experiment showed that while standard data augmentation can wash out task-relevant information (like color-coded obstacles), PSEs correctly identify that different visual cues requiring different behaviors must be represented distinctly.
Limitations and Future Work
While PSEs show great promise, they currently require access to an optimal policy ($\pi^*$) to compute the ground-truth PSM. In our evaluations, we utilized expert trajectories or learned proxies. Developing methods to learn these embeddings through online interaction without an expert remains a primary area for future research. Additionally, the computational cost of calculating the 1-Wasserstein distance in the metric's recursive definition can be high for large state spaces. Future work will also explore the performance of PSEs in highly stochastic environments, as the current study focused primarily on deterministic settings.
Summary updated