Reward Canonicalization for Behavioral Equivalence

21 concepts10 papers1 proposal

In reinforcement learning (RL), the reward function is the source of truth, dictating what an agent should do. When we design, learn, or evaluate reward functions—such as in Inverse Reinforcement Learning (IRL) or Reinforcement Learning from Human Feedback (RLHF)—we frequently need to answer a fundamental question: How different are two reward functions?

The intuitive approach is to treat reward functions as vectors and compute a standard geometric distance, like the Euclidean ($L^2$) distance. However, as Erik Jenner et al. demonstrate, standard geometric metrics fail catastrophically when applied to rewards. Two reward functions can look completely different under an $L^2$ metric while inducing the exact same optimal behavior. Conversely, two rewards can look geometrically identical but lead to wildly different policies.

To compare rewards reliably, we must look past their surface-level numerical values and focus on their behavioral equivalence classes. This is achieved through reward canonicalization: transforming reward functions into a standardized, invariant form that strips away behaviorally irrelevant variations.

The Illusion of Distance in Reward Space

Why do standard distance metrics fail? The core issue is that the mapping from reward functions to optimal policies is many-to-one. Multiple distinct numerical representations of a task yield identical behavior.

As Kareem Amin et al. note, the simplest form of this redundancy is a constant offset. If you add a constant $+10$ to every state transition, the relative preference between any two actions remains unchanged, meaning the optimal policy remains identical.

A more complex and pervasive form of redundancy is potential-based reward shaping. In reward shaping, we modify a reward function by adding an auxiliary term:

$$R_{\text{shaped}}(s, a, s') = R(s, a, s') + \gamma \Phi(s') - \Phi(s)$$

where $\Phi(s)$ is a potential function over states and $\gamma$ is the discount factor. This transformation acts like a conservative force field in physics. It provides intermediate "hints" that can help an RL agent learn faster, but because the potentials cancel out over any closed loop or trajectory, it does not alter the optimal policy.

Because of these transformations, evaluating reward functions based on raw values is highly misleading. As argued by Joar Max Viktor Skalse et al., robust evaluation metrics must focus on the relative ordering of policies induced by a reward, rather than the geometric distance between reward vectors.

The Mechanics of Reward Canonicalization

To compare two reward functions meaningfully without the prohibitive expense of training an RL agent from scratch for every comparison, we must isolate their true behavioral meaning. The solution is to map every reward function to a unique representative within its behavioral equivalence class—a process called canonicalization.

Raw Reward R₁ ───► [ Strip Offsets & Potential Shaping ] ───► Canonical Reward R*_₁
                                                                     │
                                                             Distance Metric (e.g., EPIC)
                                                                     ▲
Raw Reward R₂ ───► [ Strip Offsets & Potential Shaping ] ───► Canonical Reward R*_₂

The pioneering framework for this is EPIC (Equivalent-Policy Invariant Comparison), introduced by Adam Gleave et al.. EPIC achieves invariance to potential-based shaping by mathematically removing the potential components. It does this by taking expectations over a distribution of states, effectively "averaging out" the impact of any state-dependent potential shifts.

Once the rewards are transformed into their canonical forms ($R^*_1$ and $R^*_2$), the authors compute the correlation between these canonical rewards over a coverage dataset of transitions. Because this correlation step is invariant to positive linear rescaling, EPIC can determine if two rewards represent the same underlying preferences, regardless of differences in scale, offset, or shaping.

Grounding in Environment Dynamics

While EPIC operates over a general distribution of transitions, it can sometimes evaluate reward functions on physically impossible transitions (e.g., teleporting across a gridworld). To address this, Blake Wulfe et al. introduced DARD (Dynamics-Aware Reward Distance).

DARD refines the canonicalization step by explicitly incorporating the environment's transition dynamics. By taking expectations over the actual next-state distributions dictated by the environment, DARD ensures that rewards are only compared on "realizable" transitions. This dynamics-aware canonicalization makes the metric a much more reliable predictor of how a reward function will actually perform when deployed in a specific physical environment.

From Distance to Performance Guarantees

Building upon these insights, Erik Jenner et al. formalized a unified, three-step pipeline for comparing reward functions:

  1. Canonicalization: Transform the reward to achieve invariance under specific equivalence relations (like potential shaping).
  2. Normalization: Scale the transformed reward to prevent arbitrary magnitude differences from distorting the distance.
  3. Distance Measurement: Apply a bounded norm to compute the final distance.

This structured approach culminated in the STARC (STAndardised Reward Comparison) framework by Joar Max Viktor Skalse et al.. STARC refines the canonicalization process to remove components that do not influence policy ordering, such as redistribution across transitions. Crucially, STARC bridges the gap between abstract reward distances and practical performance: it provides "tight" theoretical bounds on worst-case regret. If the STARC distance between a learned reward and a true reward is small, an agent trained on the learned reward is mathematically guaranteed to perform well on the true reward.

Why It Matters: Robust and Efficient Reward Learning

Shifting our perspective from raw reward parameters to canonical behavioral classes fundamentally changes how we diagnose and train AI systems.

Diagnosing Failure Modes

In preference-based reward learning (such as RLHF), reward models can easily suffer from "causal confusion"—latching onto spurious correlations in the data rather than the true causes of human preference. Jeremy Tien et al. demonstrated that the EPIC pseudometric serves as a powerful diagnostic tool in these scenarios. Because EPIC isolates the true task structure independently of the policy used during evaluation, it can detect when a learned reward model matches human preferences on paper but fails spectacularly when used to train a new policy.

Improving Sample Efficiency

When we try to learn a reward function from human feedback, optimizing for the exact numerical parameters of the true reward function is a waste of effort. As Evan Ellis et al. argue, we only need to learn the reward function up to its behavioral equivalence class. By designing acquisition functions that focus on distinguishing between behaviorally distinct classes rather than identical variations, reward learning systems can drastically reduce the amount of human feedback required.

Ultimately, reward canonicalization acknowledges a core truth of reinforcement learning: the numbers in a reward matrix matter only in service of the behavior they produce. By stripping away representational noise, canonicalization gives us the mathematical tools to evaluate, diagnose, and learn alignment targets with provable confidence.

Key Foundational Work

Quantifying Differences in Reward Functions (2021) — contributed 4 concept elements to this direction.
Repeated Inverse Reinforcement Learning (2017) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reward_canonicalization_ea48c59e,
  title = {reward canonicalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-canonicalization/rdr_ea48c59e0c52445aa0c02f660ca2745d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_reward_canonicalization_ea48c59e_v1,
  title = {reward canonicalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-canonicalization/rdr_ea48c59e0c52445aa0c02f660ca2745d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake