Dynamics-Aware Comparison of Learned Reward Functions

Blake Wulfe, Ashwin Balakrishna, Logan Ellis, Jean Mercat, Rowan McAllister, Adrien Gaidon

Introduction

Evaluating learned reward functions is a critical challenge in Reinforcement Learning (RL). Often, rewards are learned from human demonstrations or preferences, but verifying their quality is difficult because different reward functions can lead to identical optimal behaviors—a phenomenon known as reward shaping.

The traditional approach to evaluation, the "rollout method," involves training a policy on the learned reward and measuring its performance on the ground-truth task. However, this process is computationally expensive and conflates errors in the reward function with the limitations of the RL algorithm itself. To address this, metrics like EPIC (Equivalent Policy Invariant Comparison) were developed to compare rewards directly. While EPIC is invariant to potential-based shaping, it fails when applied to learned rewards because it evaluates them on physically impossible "teleportation" transitions that the reward model never encountered during training.

Dynamics-Aware Reward Distance (DARD)

This paper introduces Dynamics-Aware Reward Distance (DARD), a pseudometric designed to evaluate reward functions while remaining robust to the distribution of realistic transitions. Unlike EPIC, which samples transitions randomly from the entire state space, DARD incorporates a transition model to ensure rewards are only compared on physically reachable states.

Comparison of canonicalization methods: EPIC (left) uses global sampling, while DARD (right) uses dynamics-aware sampling to stay on the data manifold.

DARD operates through three main steps:

  1. Dynamics-Aware Transformation: It uses an approximate transition model $T(s'|s,a)$ to evaluate the reward only on realistic transitions.
  2. Canonical Form: The reward is transformed into a canonical form that strips away potential-based shaping while remaining within the training data distribution.
  3. Correlation Analysis: It calculates the Pearson correlation between these transformed rewards over a dataset of realistic transitions.

DARD can utilize either a ground-truth simulator or a learned transition model (DARD-L), making it versatile for environments where the underlying physics are not perfectly known.

Key Results and Insights

The authors evaluated DARD on simulated physical domains, including a multi-agent "Bouncing Balls" navigation task and a robotic "Reacher" arm. A key finding is that learned reward functions are highly brittle; they produce nonsensical noise on out-of-distribution (OOD) transitions. EPIC's reliance on global sampling makes it susceptible to this noise, often marking high-quality rewards as "distant" from the ground truth.

As shown in the table below, DARD correctly identifies "Shaping" and "Feasibility" rewards as being functionally identical to the ground truth (distance of 0.00), whereas EPIC fails the feasibility test by assigning a high distance (e.g., 529) due to OOD noise.

Quantitative comparison of reward distances and their correlation with actual episode returns.

Key Takeaways:

  • Predictive Power: DARD is significantly more predictive of downstream policy performance than EPIC, especially for rewards learned via regression or preferences.
  • Model Robustness: DARD remains effective even with imperfect, learned transition models, as long as they capture the general neighborhood of possible next states.
  • Local vs. Global Invariance: The research demonstrates that global invariance across all imaginary transitions is unnecessary; invariance across the transitions an agent actually experiences is sufficient for reliable evaluation.

Limitations and Future Work

While DARD excels in physical environments, it can struggle with extremely smooth reward functions (like negative distance to a goal) in environments with very small timesteps. In such cases, the dynamics-aware normalization can become sensitive to minor noise. Future work involves scaling DARD to high-dimensional visual inputs, which will require the integration of robust visual transition models. Ultimately, DARD provides a path toward benchmarking reward learning algorithms without the overhead of full policy training.

Summary updated

BibTeX
@inproceedings{wulfe2022dynamics,
  author = {Blake Wulfe and Ashwin Balakrishna and Logan Ellis and Jean Mercat and Rowan McAllister and Adrien Gaidon},
  title = {{Dynamics-Aware Comparison of Learned Reward Functions}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=CALFyKVs87},
}
Figure