Predicting Real Performance from Simulated Scores with a Policy-Weighted Dynamics Gap

Methods that transfer control policies from a simulator to a different target system are, almost without exception, judged in simulation. The "target" is usually the same simulator with one knob turned: half gravity, doubled body density, tripled friction. Such a shift stays inside the simulator's own family of physics. Real mismatch is different in kind. It comes from contact, friction, and actuation effects the simulator cannot represent at any parameter setting. We propose to measure, for each specific policy, how different the target's dynamics are in the states that policy actually visits. The measurement uses a small, shared batch of target transitions. We then use it to predict how much of the policy's simulated score will survive in the target. The purpose is practical. From a pool of candidate policies trained in simulation, pick the one to deploy, and flag when the simulator's verdict cannot be trusted, without rolling out every candidate on the costly system.

The two directions in this note fit together as follows. The unknown-dynamics problem supplies the mechanism: we cannot write down the target transition law $P_{\text{tgt}}(s'\mid s,a)$, the probability of landing in state $s'$ after action $a$ in state $s$, but we can sample a little of it. The reality-gap problem supplies the regime and the test: whether a score earned in simulation predicts anything once the physics changes structurally. The first contribution is an evaluation result: parametric-shift benchmarks give misleading rankings. The second is a method: a cheap estimate that corrects for this.

Core Claim

We make two claims. The first is that the rankings of dynamics-mismatch methods under in-simulator parameter shifts do not carry over to structural shifts, such as moving the same robot model into a different physics engine. The second is that a policy-weighted, localized gap estimate predicts target return much better than any of three alternatives. Those alternatives are raw simulated return, return in a simulator calibrated to target trajectories, and a single global measure of dynamics difference. The estimate is computed from roughly a thousand target transitions gathered by a few exploratory policies and reused for every candidate.

The estimate rests on the standard simulation lemma, which bounds how far a policy's return can move when the dynamics change:

$$ \left| J_{\text{tgt}}(\pi) - J_{\text{sim}}(\pi) \right| \le \frac{\gamma R_{\max}}{(1-\gamma)^2}\, \mathbb{E}_{(s,a)\sim d^{\pi}_{\text{sim}}}\big[\, \mathrm{TV}\big(P_{\text{tgt}}(\cdot\mid s,a),\, P_{\text{sim}}(\cdot\mid s,a)\big) \big]. $$

Here $J$ is expected discounted return, $\gamma$ the discount factor, $R_{\max}$ the largest reward, and $d^{\pi}_{\text{sim}}$ the distribution of state-action pairs that policy $\pi$ visits in simulation. TV is the total-variation distance between the two next-state distributions. The constant is loose. What matters is the expectation on the right, which we call $\Delta(\pi)$. The gap is averaged only over where $\pi$ goes.

Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch proves and observes a loss that grows linearly in the global distance $\|P_E - P_L\|_1$. A global distance charges every policy for mismatch in regions it never enters. We expect that to be the main reason global measures rank policies poorly.

Relation to Prior Work

Three papers motivate the proposal by showing that mismatch is local and that restricting attention to relevant behavior pays off. Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (LoDADA) clusters next states. It then trains a source-versus-target classifier inside each cluster to estimate a local divergence, and uses that divergence to filter source data for offline training. We reuse its estimator, but for a different job: scoring policies rather than filtering data. We also weight clusters by each policy's own visitation.

A New Simulation Metric to Determine Safe Environments and Controllers for Systems with Unknown Dynamics (SPEC) shows that a worst-case gap taken over all controllers is far too conservative. Measuring the gap only for relevant controllers and falsifying trajectories certifies much larger safe sets. Our $\Delta(\pi)$ applies the same restriction idea to return prediction rather than reach-avoid certification. It uses learned local divergences in place of scenario optimization.

Multi-policy Grounding and Ensemble Policy Learning for Transfer Learning with Dynamics Mismatch shows that target data from a single policy grounds the simulator only along that policy's path. This constrains our design. Target transitions must come from several deliberately diverse policies. Any cluster a candidate visits but the target data does not cover must be reported as unknown, not as zero gap.

Two papers set the transfer methods whose rankings we test. State-only Imitation with Transition Dynamics Mismatch (I2L) keeps the learner's own feasible trajectories in a buffer and pulls that buffer toward the expert's states. MPGE trains across an ensemble of grounded simulators. Both, like LoDADA and robust inverse reinforcement learning, are evaluated only under synthetic in-family perturbations. None of them asks whether its score predicts behavior under structural mismatch.

Traversing the Reality Gap via Simulator Tuning supplies both the structural-shift testbed and our strongest calibration competitor. It uses Differential Evolution, a population-based black-box optimizer, to tune physics parameters against a real Kinova arm. It reports 14% to 91% lower trajectory error than untuned settings, with residual error remaining. The residual error is exactly the structural component we target.

Research Program

We will build a structural-shift testbed. The same MJCF/URDF robot models for HalfCheetah, Hopper, Walker2d, and Ant will be instantiated in MuJoCo (source) and in PyBullet's Bullet engine and ODE (targets). Collins et al. show these engines disagree materially even after tuning. The I2L parametric shifts will run alongside: gravity at 0.5x, density at 2x, friction at 3x.

Porting models across engines is not exact. Differences in joint limits or actuator conventions would add mismatch of their own. We will therefore report per-engine sanity checks, such as open-loop torque replay error. We will also treat engine-porting artifacts as part of the structural gap rather than hide them.

For each task and shift, we will train a pool of about fifty candidate policies. The pool will include LoDADA, MPGE, I2L, an Implicit Q-Learning (offline RL) baseline, and SAC with and without domain randomization, across seeds and checkpoints. We will then collect a fixed budget of 1,000 target transitions from five entropy-regularized exploratory policies. This budget matches MPGE's reported regime.

On this shared batch, we fit LoDADA-style next-state clusters and per-cluster classifiers. Each classifier yields a local divergence estimate. For each candidate policy, we compute its simulated visitation over clusters and form $\Delta(\pi)$. We also record the fraction of its visitation that falls in poorly covered clusters. The predicted target return is the simulated return minus a penalty proportional to $\Delta(\pi)$. The single scale factor is fit on held-out tasks, so no per-task tuning happens on target returns.

The deliverables are three. The first is a ranking-transfer analysis comparing parametric and structural shifts. The second is the predictor itself. The third is a coverage diagnostic that says when to abstain.

Baselines

The do-nothing reference is ranking candidates by simulated return alone. This is how every paper in the file implicitly argues for transfer. Beating it is the minimum bar.

The strongest calibration baseline is simulator tuning in the style of Collins et al. Differential Evolution adjusts the target-facing engine's parameters to fit the same 1,000 target transitions, and candidates are ranked by return in the tuned simulator. Tuning is the most direct way to "fix the simulator," so it is the natural rival. For fairness, it receives the same target data and a matched wall-clock compute budget. It will be restricted to the "Big Three" parameters Collins et al. identify (timestep, lateral friction, maximum joint velocity) plus masses, because their full 57-parameter search degraded convergence and cost hundreds of hours. On the Kinova tasks they report a 14% to 91% reduction in trajectory error. No reported result was found on Lacuna for tuned-simulator policy ranking or policy selection, so we have no published target number to beat.

The data-only baseline is fitted Q evaluation, which estimates each candidate's target return directly from the shared 1,000 transitions without using the simulator. The global-gap baseline penalizes every policy by one domain-classifier divergence estimated over all target data. This corresponds to the $\|P_E - P_L\|_1$ view of robust inverse reinforcement learning. No reported result was found on Lacuna for any of these as return predictors.

Two ablations isolate the key ingredients. The first removes policy weighting by averaging cluster divergences uniformly. The second removes localization by using one global classifier weighted by policy visitation. The oracle is full target evaluation of every candidate.

Winning means three things. On structural shifts, our predictor must achieve higher Spearman rank correlation with true target return than every baseline. It must also yield lower top-1 selection regret (the target-return gap between the chosen policy and the best in the pool), with the difference holding across seeds by a paired test. For context on method-level transfer, I2L reports 3975 return against GAIL-S's 923 on HalfCheetah at 2x density, and MPGE reports over 75% of optimal target performance with 1,000 target samples. These are training results, not prediction results. We use them only to check that our re-implementations reach the reported range.

Why not just…?

Why not just use domain randomization and skip prediction? Randomization changes how policies are trained. It does not tell you which trained policy to deploy. Collins et al. also note that widening parameter ranges cannot represent contact effects outside the engine's model family. We include randomized SAC in the candidate pool. If randomized policies top the target ranking under structural shift and simulated return already identifies them, our method adds little. The ranking analysis will show whether that happens.

Why not just tune the simulator and trust its scores? This is a real threat on in-family parametric shifts. There, tuning can in principle recover the true parameters exactly, and we expect it to match or beat us. The argument for our method is structural shift. Collins et al.'s residual error after tuning, and their convergence problems as parameters increase, suggest that some mismatch cannot be tuned away. That residual mismatch will be concentrated in particular regions of the state space, which is what a localized estimate can see. The parametric-versus-structural comparison settles this directly. If tuned-simulator ranking wins on structural shifts too, we will report that outcome.

Why not just evaluate candidates directly on the target, or with off-policy evaluation? Direct evaluation costs rollouts for every candidate, including unsafe ones. With fifty candidates and a 1,000-transition budget, each candidate gets about twenty transitions. We will run exactly this matched-budget version as a baseline. Off-policy evaluation from 1,000 transitions shares data across candidates, as we do, but ignores the simulator's dense information and is likely high-variance. Our predictor uses the target data only to estimate a correction. If fitted Q evaluation matches us, the simulator is contributing nothing, which would itself be a finding about sim-based evaluation.

Why not just use worst-case robust training? Robust inverse reinforcement learning and SPEC's analysis of the standard metric both document over-conservatism when the real gap is small. A worst-case penalty would push every policy toward the same pessimistic score and flatten the ranking. The global-gap baseline approximates this view. Its rank correlation will be reported alongside ours.

Decisive Checks

The first check is whether rankings flip. We compute Kendall's tau between method rankings under parametric and structural shifts. A tau near one would undercut the paper's motivation, and we would report it as a negative result.

The second check is prediction quality versus target-data budget. We sweep from 100 to 5,000 target transitions and plot rank correlation and selection regret. The relevant comparisons are the tuned-simulator baseline, fitted Q evaluation, and the ablations. The claim holds only if policy weighting and localization each contribute on structural shifts.

The third check is the coverage diagnostic. Among candidates whose visitation falls mostly in poorly covered clusters, prediction error should be visibly larger. Abstaining on those candidates should reduce the rate of large mispredictions without discarding most of the pool.

As a stretch goal, we will repeat the selection experiment on a physical manipulator doing Kinova-style pushing, if hardware access allows. The cross-engine testbed is our main proxy for real-world structure. We say this plainly: it is a proxy, not real-world validation.