Overview
Evaluating learned reward functions is notoriously difficult because standard metrics often test them on physically impossible transitions where they haven't been trained. This paper introduces Dynamics-Aware Reward Distance (DARD), a new pseudometric that uses environment dynamics to compare rewards only on "realizable" transitions, making it a much more reliable predictor of actual policy performance than previous state-of-the-art methods.
The Challenge of Evaluating Learned Rewards
In Reinforcement Learning (RL), we often learn reward functions from human demonstrations or preferences. But how do we know if a learned reward is actually good?
The traditional "rollout method" involves training a full policy on the learned reward and seeing how it performs on the ground-truth task. This is computationally expensive, potentially dangerous in the real world, and makes it hard to tell if a failure is due to a bad reward function or a bad optimization algorithm.
To fix this, researchers previously proposed EPIC (Equivalent-Policy Invariant Comparison). EPIC compares reward functions directly without training a policy. However, EPIC has a major flaw: it evaluates rewards by sampling transitions between any two states, even if the transition is physically impossible (like a robot teleporting across a room). Since learned reward functions (usually neural networks) are only reliable on data similar to their training set, these "out-of-distribution" queries lead to inaccurate comparisons.
Methods: Dynamics-Aware Reward Distance (DARD)
The authors propose DARD to address the shortcomings of EPIC. Like EPIC, DARD is a pseudometric designed to be invariant to reward shaping—the practice of adding mathematical "hints" to a reward function that change its density but not the optimal policy.
The key innovation in DARD is the Dynamics-Aware Transformation. Instead of averaging reward values over all possible state-state pairs (as EPIC does), DARD uses a transition model of the environment to sample only transitions that are approximately physically realizable.
How it works: 1. Transformation: It transforms the reward function into a canonical form using expectations over a transition model $T(s'|s, a)$. 2. Comparison: It calculates the Pearson distance between these transformed rewards. 3. Flexibility: DARD can work with a known ground-truth simulator or a learned transition model (DARD-L), making it applicable even when the exact physics are unknown.
Experimental Setup
The researchers evaluated DARD across several simulated environments:
- Bouncing Balls: A custom multi-agent environment where an agent navigates a field of bouncing obstacles. This environment is specifically designed to trigger "teleportation" issues in EPIC.
- Reacher: A standard robotic manipulation task from OpenAI Gym/MuJoCo.
- Point Maze: Used specifically to analyze the limitations of the method.
They tested these using various reward models, including ground-truth, shaped, and models learned via regression or human-like preferences.
Key Insights and Ideas
- The OOD Trap: Learned reward functions are "brittle" outside their training distribution. EPIC’s biggest strength (being dynamics-invariant) is its biggest weakness when applied to learned models because it forces the model to "hallucinate" rewards for impossible physics.
- Local vs. Global Sampling: EPIC uses global sampling (any $s$ to any $s'$), while DARD uses local sampling (where can I actually go from $s$ given action $a$?). This keeps the evaluation "in-distribution."
- Predictive Power: DARD's distance values correlate much more strongly with the actual return of a policy trained on that reward. If DARD says two rewards are similar, the policies they produce are likely to perform similarly.
- Robustness to Model Error: Surprisingly, DARD-L (using a learned, imperfect transition model) performs almost as well as DARD using ground-truth physics.
Key Results
Across the tested environments, DARD demonstrated a superior ability to evaluate reward functions accurately compared to baseline methods.
- Predicting Performance: In the Reacher environment, EPIC assigned a lower distance to a poor "Preferences" model than to a high-performing "Regression" model. DARD correctly identified the Regression model as being closer to the ground truth.
- Sanity Checks: On "Feasibility" models (which return noise on impossible transitions), EPIC failed completely (assigning high distance), while DARD correctly recognized them as equivalent to the ground truth.
- Correlation: DARD showed a consistent ordering of reward models that matched the ordering of their actual episode returns, a property EPIC lacked when dealing with learned functions.
Limitations and Future Work
- The "Smoothness" Issue: DARD can struggle in environments with very small timesteps and very smooth reward functions (like "negative distance to goal"). If all nearby transitions have nearly identical rewards, DARD becomes sensitive to minor network noise.
- Dynamics Dependency: Unlike EPIC, DARD requires a transition model. While the paper shows learned models work well, the method's accuracy is still tied to the quality of that model.
- High-Dimensional Scaling: Future work is needed to see how DARD handles high-dimensional visual inputs (like pixels) where learning a perfect transition model is significantly harder.
Summary updated