Introduction
In Deep Reinforcement Learning (RL), agents frequently learn from high-dimensional, noisy inputs like raw pixels. A fundamental challenge is representation learning: transforming these complex inputs into compact vectors that capture only the task-relevant information. This paper introduces the Reducing Approximation Gap (RAP) distance, a novel behavioral metric designed to help RL agents learn superior state representations. By bridging the gap between theoretical "bisimulation" metrics and practical deep learning implementations, RAP enables agents to ignore task-irrelevant visual noise (such as moving backgrounds) and achieve state-of-the-art performance in complex, distracted environments.
The Representation Learning Challenge
A popular approach to representation learning relies on behavioral metrics like bisimulation. The core principle is that two states should be close in the embedding space if they yield similar rewards and lead to similar future states. However, theoretical "gold standard" metrics are computationally intractable because they require calculating the 1-Wasserstein distance between probability distributions.
Existing practical approximations suffer from three major "gaps":
- Loss Function Mismatch: To simplify calculations, previous methods introduce biases that loosen the theoretical bounds on the value function.
- Dynamics Relaxation: Replacing complex distance measures with simpler ones often breaks the mathematical guarantees of the original metric.
- The $L_1/L_2$ Limitation: Standard distances (like Euclidean distance) force the distance between a state and itself to be zero. However, in stochastic environments, a state’s "behavioral" self-distance is often non-zero due to inherent randomness in rewards and transitions. Forcing this to zero in the embedding space degrades the representation.
Methodology: The RAP Distance
To address these limitations simultaneously, the authors propose the RAP distance, which introduces several key innovations:
- Expected State Comparison: Instead of comparing full, expensive transition distributions, RAP compares the expected next states. The authors prove that this approach still provides a valid contraction mapping and a unique fixed point.
- Variance-Aware Loss: To avoid loss function mismatch, the authors derive a new objective that moves the expectation operator outside the absolute value. By incorporating the reward variance into the loss, they allow for more accurate sampling without introducing bias.
- Non-Zero Self-Distance: RAP adopts a specific distance formulation that combines $L_2$ norms and angular distance. This allows the model to represent the inherent "behavioral noise" of a state without artificially pushing similar states apart in the embedding space.
Implementation: The RAP method is built on top of the Soft Actor-Critic (SAC) algorithm. As shown in the architecture diagram, it learns a reward approximator and a dynamics model on the embedding space to provide the necessary signals for the RAP loss, which is optimized jointly with the standard SAC actor and critic losses.
Experimental Evaluation
The authors evaluated RAP across three challenging benchmarks designed to test robustness against visual distractions and environmental variations:
- Distracting DeepMind Control Suite (DMC): Standard control tasks (e.g., Cheetah-run, Walker-walk) overlaid with "natural video" backgrounds from the Kinetics dataset to act as visual noise.
- Robosuite: A robotic simulation environment with randomized colors, lighting, and camera positions.
- CARLA: A high-fidelity autonomous driving simulator where agents trained in one weather condition were tested across others (e.g., rain, sunlight) to evaluate generalization.
Key Insights and Results
- The "Aha" Moment on Self-Distance: The paper highlights that in RL, due to stochastic policies and rewards, a state is not "behaviorally identical" to itself across different time steps. By allowing non-zero self-distance, the embedding space becomes significantly more robust.
- Bridging Theory and Practice: The core contribution is a mathematical proof that RAP maintains the theoretical "Value Difference Bound." If two states are close in RAP distance, their optimal values are guaranteed to be close.
- Superior Robustness: In the Distracting DMC benchmark, RAP significantly outperformed previous behavioral methods like DBC and MICo, and even beat DrQ (a leading data-augmentation method) in learning speed and final score on most tasks.
- Robosuite Dominance: On the "Door Opening" task, RAP achieved a score of ~102, while the next best method (DrQ) only reached ~14. In the "Two Arm Peg-In-Hole" task, it outperformed the runner-up by 1.4x.
- Real-World Generalization: In CARLA, RAP demonstrated smoother training curves and higher final scores than baselines, proving its capability to handle complex, multi-camera inputs for autonomous driving.
Limitations and Future Work
While highly effective, RAP has a few areas for further exploration:
- Computational Overhead: RAP requires training auxiliary models (reward and dynamics predictors), increasing the per-step computational cost compared to vanilla RL.
- Hyperparameter Sensitivity: The distance function uses a hyperparameter ($k$) to balance the $L_2$ and angular components, which may require tuning for entirely new domains.
- Future Directions: The authors suggest applying these robust metrics to more complex, non-stationary environments where the rules of the world change over time.
Summary updated