Many modern control systems do not maximize reward greedily. They sample actions in proportion to the exponentiated reward. Soft actor-critic and soft Q-learning do this, and so do the sampling-based planners used to steer imitation-learned robots back on track. For any such consumer, the size of the reward relative to a "temperature" knob decides how decisive the resulting behavior is. Yet the tools we use to compare reward functions deliberately throw scale away, because under ordinary reward maximization only the ordering of returns matters. We propose a reward distance with three properties. It removes the differences that really are irrelevant to a soft-optimal agent: additive "shaping" terms that telescope away along trajectories. It keeps scale, measured relative to temperature. It is computed on the states a drifting imitation policy actually visits. We then test whether this distance predicts, without any policy rollouts, which learned reward models will successfully pull a behavior-cloned robot back from its own mistakes.
The idea in two sentences: a reward only matters through the soft-optimal policy it induces, and that policy depends on reward differences divided by temperature, so reward comparisons for entropy-regularized agents should be scale-sensitive in exactly that unit. Measured this way, and on the off-expert states where recovery happens, the distance between successive learned reward models should tell us when a test-time recovery planner is about to change its behavior, and whether it will get better or worse.
Background and the gap
Reward canonicalization maps each reward function to a standard representative of the set of rewards that produce the same behavior. It then compares those representatives. The first method of this kind, EPIC, strips constant offsets and potential-based shaping by taking expectations over a coverage distribution, and uses a correlation distance that bounds regret (Quantifying Differences in Reward Functions). Potential-based shaping means adding $\gamma\Phi(s')-\Phi(s)$ for some state function $\Phi$. DARD fixes EPIC's habit of evaluating physically impossible transitions by drawing next states from true or learned dynamics (Dynamics-Aware Comparison of Learned Reward Functions). STARC recasts the whole approach as canonicalize, normalize, then measure. It proves upper and lower regret bounds and reports that its value-adjusted variant correlates 0.873 with worst-case regret on random MDPs, against 0.778 for EPIC and 0.782 for DARD (STARC). All three motivate this work and supply its machinery. All three also normalize scale away and define equivalence with respect to plain expected return.
The research file frames the gap as "potential shaping breaks invariance under maximum-entropy RL." We think that framing is wrong, and correcting it is the starting point of this proposal. Maximum-entropy RL adds a policy-entropy bonus weighted by a temperature $\alpha$ to the return. The soft-optimal policy is $\pi(a\mid s)\propto\exp(Q_{\text{soft}}(s,a)/\alpha)$ (Reinforcement Learning with Deep Energy-Based Policies). Adding exact potential shaping shifts $Q_{\text{soft}}(s,\cdot)$ by $-\Phi(s)$, which is the same constant for every action at that state, so the soft-optimal policy is unchanged. What does break is scale. Multiplying the reward by $c$ is the same as dividing the temperature by $c$. That turns a multi-modal policy, the kind that finds all four goals in the energy-based policies paper, into a collapsed one. The practitioner complaint that reward scale and $\alpha$ must be tuned jointly (State-Dependent Maximum Entropy RL; Conservative Q-Learning) is the everyday face of this fact. Offsets are also not free in tasks where episodes can terminate: there a constant acts as a survival bonus. This is the reward bias documented for adversarial imitation (Addressing reward bias in Adversarial Imitation Learning with neutral reward functions). The correct equivalence class for a soft agent is therefore "shaping yes, scale no, offsets only with fixed horizons." No existing distance respects all three.
The application regime is imitation learning under distribution shift. A behavior-cloned policy makes small errors, drifts into states the demonstrations never covered, and its errors compound. SAILOR tackles this without a human in the loop (A Smooth Sea Never Made a Skilled SAILOR). It learns a world model and a reward model from expert and learner data. At test time it runs a path-integral planner that weights imagined trajectories by the exponentiated predicted return divided by a planner temperature $\lambda$, and it distills the recoveries back into the policy over several rounds. SAILOR treats its reward model as an uncalibrated black box. Across rounds that model can grow or shrink in scale, which silently changes $\lambda$'s effective meaning. It can also acquire shaping-like terms, and inside a planner truncated at horizon $H$ these do not cancel: the telescoped sum leaves a terminal bonus $\gamma^H\Phi(s_H)$. Adversarial reward models are known to drift during training (Model-Free Imitation Learning with Policy Optimization frames the underlying min-max game). Guided cost learning (Guided Cost Learning) shows that learned maximum-entropy costs stay local to the demonstrations. Neither line asks how to measure such drift in the units that matter to the planner.
Core Claim
Let $R_1,R_2$ be rewards consumed by soft agents with temperatures $\alpha_1,\alpha_2$, and let $\mathcal{C}$ be a shaping-removing canonicalization with no scale normalization. We propose the distance
$$d_T(R_1,R_2)=\Big\| \tfrac{\mathcal{C}(R_1)}{\alpha_1}-\tfrac{\mathcal{C}(R_2)}{\alpha_2}\Big\|_{\mu},$$
where the norm is taken over a transition distribution $\mu$. Its justification is a short, standard argument. The soft Bellman operator is a $\gamma$-contraction and log-sum-exp is 1-Lipschitz in the sup norm. So when the rewards differ by at most $\delta$ after the best shaping correction, the log-probabilities of the two soft-optimal policies differ by at most about $2\delta/(\alpha(1-\gamma))$, and so does their KL divergence. Temperature-scaled, shaping-canonical differences therefore bound behavioral divergence of soft agents in a way that scale-normalized distances cannot: STARC assigns zero distance to $R$ and $10R$.
Our first empirical claim is that $d_T$ predicts soft-optimal policy divergence and soft regret better than EPIC, DARD and STARC whenever scale differs across the rewards being compared. Our second claim is about where to measure. When $\mu$ is the distribution of states the drifting learner visits, rolled out in the world model, $d_T$ between reward-model checkpoints predicts which checkpoint yields better closed-loop recovery. The expert-state distribution and uniform coverage both predict worse, because recovery lives off the expert manifold but on reachable states. This extends DARD's "only feasible transitions" principle to "only feasible and actually visited transitions." That is the tension between global coverage and narrow support the file identifies.
Research Program
The work has three parts.
The first part builds the estimator. It uses EPIC-style expectation canonicalization with DARD-style next-state sampling from the world model, over transitions drawn from $\mu$. It keeps the unnormalized canonical reward and divides by the consumer's temperature. For the path-integral planner, the consumer's temperature is $\lambda$ times the planning horizon's effective discount mass. For finite-horizon planners the estimator also reports the residual terminal-shaping term. This is estimated as the variance, across candidate end states, of the difference between the learned reward's short-horizon return and its canonical counterpart, so that truncation artifacts are measured rather than assumed away. We will state the continuous-space estimation assumptions plainly, since STARC's completeness results are finite-space only.
The second part validates the claim where ground truth is exact: small MDPs and gridworlds where soft value iteration is cheap. There we can compute soft-optimal policies at many temperatures and correlate every distance with KL divergence and with soft regret.
The third part is the application. We reproduce SAILOR, save reward-model checkpoints across its rounds and seeds, and ask whether $d_T$, measured on learner-visited states, ranks checkpoints by the closed-loop success they deliver when plugged into the planner.
A constructive variant is included but treated as contingent. It adds a trust-region penalty that keeps $d_T$ between successive reward models below a budget during SAILOR's rounds. This variant is kept only if it beats simpler scale fixes (see below).
Manual reward engineering serves as a controlled stress test rather than as the core. Hand-written dense rewards give pairs of rewards that engineers regard as "the same task" but that differ by scale, shaping, or milestone jumps. Examples are the PointMass sparse/dense pair on which EPIC reports near-zero distance, and Meta-world dense rewards with and without jump terms. These pairs let us check whether $d_T$ agrees with the behavioral differences soft agents actually show.
Baselines
The naive reference point is the raw $L^2$ distance between reward vectors on the same transitions, with no canonicalization. It is sensitive to everything, including irrelevant shaping, and every other metric should beat it.
The strongest existing metric is STARC with value-adjusted levelling (STARC). It has the best reported regret correlation in the file, 0.873 on random MDPs, and the tightest theory. We run it on exactly the transitions and dynamics samples given to our metric, with the same value-estimation budget. EPIC (Quantifying Differences in Reward Functions) and DARD (Dynamics-Aware Comparison) are also run on those transitions. DARD shares our world model, so any gap cannot come from better dynamics.
The gold-standard but expensive reference is direct rollouts: optimize a policy or run the planner on each reward and measure task success. It defines the targets we correlate against, and it sets a cost we report metrics relative to. EPIC's authors report 17 seconds for EPIC versus 31 hours for the rollout-based NPEC comparison.
Two ablations isolate our ingredients. The first is STARC with its scale normalization replaced by division by temperature. It separates "keep scale" from "canonicalize differently." The second is $d_T$ computed on expert-only states and on uniform coverage. It isolates the support choice.
On the imitation side, the references are Diffusion Policy behavior cloning and vanilla SAILOR (SAILOR). SAILOR reports that about 20 demonstrations beat Diffusion Policy trained on 200. It also reports beating DPPO-IRL at five times the interaction budget. The file gives no per-task success percentages, so we report our own reproduction's numbers as the reference. The strongest competitor for the constructive variant is SAILOR with the planner temperature re-tuned every round so that the effective sample size of the planner's trajectory weights stays fixed. All control variants share demonstrations, world-model training, planner samples per step, and an equal hyperparameter-search budget.
Beating the metric baselines means higher rank correlation with the rollout-derived targets, with bootstrap intervals over seeds that exclude STARC's value. Beating the control baselines means higher closed-loop success than ESS-adaptive SAILOR at matched compute.
Why not just…?
Why not just use STARC, which already has the strongest theory? STARC's guarantees concern regret under expected return, where scale is irrelevant by construction. For a soft agent or path-integral planner, scale is behavior. Our first experiment is built to expose exactly this: STARC must assign zero distance to scaled pairs whose soft-optimal policies differ sharply. If STARC nonetheless predicts SAILOR checkpoint quality as well as $d_T$, that would show scale drift is not what limits recovery, and we would report it as such.
Why not just evaluate each reward model by rolling out the planner? For final evaluation, we do. But rollouts cost a full closed-loop evaluation per checkpoint. They are exactly what one cannot afford when monitoring reward drift between rounds or selecting among many checkpoints. They also confound reward error with world-model error, which is known to compound over horizons (Investigating Compounding Prediction Errors in Learned Dynamics Models). The proposal's value claim is precisely that $d_T$ approximates the rollout ranking at a small fraction of the cost. The cost ratio is reported next to every correlation.
Why not just adapt the planner temperature each round to keep weight concentration fixed? This is the real threat. It is standard practice in path-integral control, and it may absorb most of the scale-drift problem with no reward analysis at all. It cannot, however, see shaping-induced terminal bonuses or changes in reward shape on the learner's off-expert states. Adaptive temperature is therefore the head-to-head baseline for the constructive variant. If it matches the trust-region version, we drop the constructive claim and keep only the diagnostic one: $d_T$ as a rollout-free predictor. That claim does not depend on how the planner's temperature is set.
Why not just collect more demonstrations or query an expert (DAgger)? SAILOR's own comparison shows ten times more demonstrations does not close the gap to learned recovery. DAgger's gains, from roughly 50 m to over 220 m mean flight distance in Learning Monocular Reactive UAV Control, require an expert on call, which ThriftyDAgger tries to ration. Our setting assumes no online expert. The question of whether the reward model guiding autonomous recovery is trustworthy only arises there.
Proposed Experiments
The first experiment is the exact check. On random MDPs and gridworlds with known dynamics, we generate reward pairs related by known shaping, scale factors from 0.1 to 10, offsets under termination, and random perturbations. For each temperature we compute soft-optimal policies by soft value iteration. We correlate each metric with policy KL divergence and with soft regret. The decisive prediction is that $d_T$ tracks divergence across scale changes while STARC, EPIC and DARD report zero. On shaping-only pairs, every canonical metric, $d_T$ included, should report near zero. This confirms the correction to the file's gap statement.
The second experiment uses hand-engineered rewards with soft agents. On the multi-goal point-mass task, where soft Q-learning captures four goals and DDPG one, and on Meta-world drawer-put-in, where SDME reports 86% versus 44% success for the best soft actor-critic baseline under randomized starts, we train soft actor-critic at fixed $\alpha$ on scaled, shaped, and milestone-jump variants of the dense reward. We ask whether $d_T$ from the reference reward predicts the number of goal modes retained and the final success ordering, and whether the scale-normalized metrics fail to.
The third experiment is the decisive application test. We reproduce SAILOR on RoboMimic Can and Square and ManiSkill PullCube, with five seeds, saving every round's reward model. For each checkpoint we compute all metrics against the final-round reward model and against a held-out "oracle" reward model trained on the true success signal. The metrics are computed on learner-visited, expert-only, and uniform transition distributions. We then run the planner with each checkpoint for closed-loop success. The primary outcome is Spearman correlation between metric and success. The claim fails if $d_T$ on learner-visited states does not beat STARC by a margin outside seed-bootstrap intervals, or if the expert-only version does as well. Either result would mean temperature units or support restriction add nothing in the regime that motivated them.
The fourth experiment is the contingent constructive test. It compares vanilla SAILOR, SAILOR with ESS-adaptive planner temperature, and SAILOR with the $d_T$ trust region on the same tasks, at matched compute. We report success rate and success versus planner samples. The trust region earns a place in the method only if it beats ESS-adaptive SAILOR, and specifically on seeds where the residual terminal-shaping diagnostic is large. That is the case adaptive temperature cannot fix.
References
- Quantifying Differences in Reward Functions
- Dynamics-Aware Comparison of Learned Reward Functions
- STARC: A General Framework For Quantifying Differences Between Reward Functions
- Reinforcement Learning with Deep Energy-Based Policies
- State-Dependent Maximum Entropy Reinforcement Learning for Robot Long-Horizon Task Learning
- Conservative Q-Learning for Offline Reinforcement Learning
- Addressing reward bias in Adversarial Imitation Learning with neutral reward functions
- A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search
- Model-Free Imitation Learning with Policy Optimization
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Investigating Compounding Prediction Errors in Learned Dynamics Models
- Learning monocular reactive UAV control in cluttered natural environments
- ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning