Simulator-Anchored Bellman Validation for Choosing Sim-Trained Policies from Real Logs

Picking which of many simulator-trained policies to put on a real robot is usually done by trial and error on hardware. The obvious offline alternative is to score each candidate's value function by how well it satisfies the Bellman equation on logged real transitions. A value function is the model's prediction of future reward, and it should equal the immediate reward plus the discounted value of the next state. That score is corrupted by the double sampling problem. Squaring a one-sample Bellman error adds the variance of the next state's value, so a candidate is penalized for the environment's randomness as well as for its own mistakes. Physical time cannot be rewound to draw a second next state from the same state and action. A simulator that can be reset can draw many next states, but they come from the wrong dynamics.

We propose to combine the two. At each logged real state and action, we reset the simulator many times and average the candidate's value over the simulated next states. That average is a noise-free but biased "anchor". We then learn only the correction between real and simulated expected next-state value by supervised regression on the real log. The resulting estimate of the real expected backup needs far fewer real transitions than learning the whole backup from scratch, provided the simulator is roughly right. It also degrades gracefully to purely data-driven estimation when the simulator is wrong. We would test this as an offline model-selection tool for sim-to-real transfer on perturbed-physics MuJoCo benchmarks with ground-truth returns.

Core Claim

Define the following for a candidate value function $V$ with discount $\gamma$, at a logged real pair $(s,a)$ with reward $R$:

  • $m_{sim}(s,a)=\frac{1}{K}\sum_{k=1}^K V(s'_k)$ is the average over $K$ simulator resets.
  • $g(s,a)=\mathbb{E}_{P_{real}}[V(s')]-\mathbb{E}_{P_{sim}}[V(s')]$ is the true real-minus-sim gap.
  • $\hat g$ is a regressor fit on real transitions to the targets $V(s'_{real})-m_{sim}(s,a)$.

The score of the candidate is the average over held-out real data of

$$\hat\delta^2_{anchor}(s,a)=\Big(V(s)-R-\gamma\big(m_{sim}(s,a)+\hat g(s,a)\big)\Big)^2 .$$

Its error relative to the true squared Bellman residual is driven only by $\gamma(\hat g-g)$ plus the Monte Carlo error of $m_{sim}$, which shrinks as $1/K$. It is not driven by the transition variance $\gamma^2\mathrm{Var}_{s'}(V(s'))$ that contaminates the naive score. The regression error of $\hat g$ scales with the complexity of the gap $g$ rather than of the full backup. Where the simulator is accurate, $g$ is small and smooth.

Two special cases bracket the method. If the simulator is exact ($g\equiv 0$), the estimator reduces to reset-based double sampling as analyzed by The Power of Resets in Online Reinforcement Learning. If the simulator is useless ($m_{sim}\equiv 0$), it reduces to Supervised Bellman Validation.

The claim is specific and falsifiable. On small real logs with a moderately wrong simulator, anchored validation should rank sim-trained candidates by true real return more accurately than both the naive score and Supervised Bellman Validation. Its advantage should vanish as the real log grows or as the simulator's error grows.

Relation to Prior Work

Three papers supply the method.

  • Supervised Bellman Validation. Revisiting Bellman Errors for Offline Model Selection trains a "Bellman network" to regress next-state values and so removes transition noise from held-out Bellman errors. It picked top-5% candidates on Pong and Seaquest, but only in-domain and with no simulator. We keep its supervised-regression trick but regress a residual on top of a simulator anchor.
  • Unbiased double samples. Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning shows that independent double samples make batch RL as statistically efficient as supervised learning. This motivates our wish to approximate them.
  • Resets. The Power of Resets in Online Reinforcement Learning proves that resets remove the double-sampling bottleneck but assumes the simulator is the target environment. Our correction term $\hat g$ is exactly what is missing when that assumption fails.

Two papers already sit at the intersection, and we differ from both.

Our proposal instead uses resets at real states to evaluate, and explicitly corrects the simulator's bias with real data. Separately, Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics measures cross-domain Bellman consistency to weight a source critic, but computes target errors from single samples. Anchored validation could replace its consistency signal, though we leave that to future work.

One paper constrains the proposal. Deep Residual Reinforcement Learning argues that double sampling is irrelevant because popular simulators are deterministic. We accept this for the simulator side: with a deterministic simulator, $K=1$ suffices. However, the real side is stochastic, and that is where the naive score fails. Our benchmarks therefore inject real-side noise deliberately rather than relying on MuJoCo's determinism.

Research Program

Estimator. We will build the estimator for continuous-control value functions. The simulator must support setting state from a logged real state, which the perturbed-MuJoCo setup of SR² already provides. The regressor $\hat g$ is a small network trained per candidate on a training split of the real log with cross-fitting. Scores are computed on a disjoint validation split, so that $\hat g$ is never scored on its own training points.

Analysis. We will prove a bound of the form: score error ≤ (regression error of the gap) + (Monte Carlo error) + (distribution-shift term from resetting to logged states). We will then check empirically whether the regression error of the gap indeed tracks the size of the physics perturbation.

Diagnostic. We will study a failure mode. When the simulator's error is non-smooth, for example contact changes under reduced friction, $g$ becomes hard to regress. The anchor may then hurt relative to Supervised Bellman Validation. To guard against this, we will include a held-out check that falls back to Supervised Bellman Validation when the validation loss of $\hat g$ exceeds that of a plain Bellman network.

Training loss (secondary). Only if the model-selection results hold will we test the same anchored target as a residual-minimization training loss.

Baselines

Naive reference points. The first is the one-sample squared Bellman error on real validation data, equivalent to sample cloning. Borrowing From the Future shows sample cloning carries a bias that never vanishes, so we expect it to favor low-variance rather than accurate candidates. The second is to score each candidate by its return in the nominal simulator, since that is what practitioners implicitly do. Anchored validation must beat both on every perturbation setting where real noise is present.

Strongest existing competitor: Supervised Bellman Validation. Revisiting Bellman Errors for Offline Model Selection is the same regression idea without the anchor. We will give its Bellman network the same architecture, the same real-data splits, and the same hyperparameter search budget as $\hat g$. The only reported figure is in-domain: it "consistently selected policies in the top 5% of available candidates" on Pong and Seaquest. No reported result was found on Lacuna for Supervised Bellman Validation under sim-to-real shift. Beating it means lower top-1 regret and higher rank correlation at small real-log sizes, with overlapping confidence intervals at large sizes being the expected and acceptable outcome.

Other competitors. We will also run Fitted Q-Evaluation and weighted importance sampling, which Revisiting Bellman Errors for Offline Model Selection reports being outperformed on its tasks.

Ablations isolating the key ingredient. The first ablation is the anchor alone ($\hat g\equiv 0$). This is pure reset double sampling in the wrong simulator, as in Mhammedi et al. and RNAC's use of resets. It isolates how much the correction contributes. The second is the anchor replaced by a learned dynamics model trained on the real log. It isolates whether the value of the anchor comes from the simulator's physics prior or merely from having any second-sample generator.

Why not just…?

Why not just use Supervised Bellman Validation on the real log? This is the real threat. With enough real data, the plain Bellman network learns the real backup directly and the anchor adds nothing. Our claim is limited to the regime of small real logs, which is the realistic regime for hardware. The real-log-size sweep settles it: if Supervised Bellman Validation matches us at the smallest sizes, the idea is wrong.

Why not just double-sample in the simulator, as RNAC does? RNAC and reset theory estimate the simulated backup. Under doubled gravity or 30% friction, the kind of gap used in State Revisit and Re-explore, that backup is systematically wrong for the real system. The anchor-only ablation measures exactly this bias. We expect its ranking quality to fall as the perturbation grows while the corrected version holds.

Why not just fit a dynamics model to the real log and double-sample from it? A learned model gives a second sample without a simulator. However, on small logs it compounds errors, a failure mode both Deep Residual Reinforcement Learning and SR² flag for model-based rollouts. It also lacks the physics prior that the simulator supplies. The learned-model ablation tests this directly. If it performs as well, the contribution reduces to "use any generative second sample", and we would report that.

Why not just skip validation and rely on robust training? Robust training such as RNAC reached "100% success rate" versus PPO's "0% success rate" on the physical TurtleBot. That result reduces the need for selection but does not remove it: robust methods still produce many checkpoints and hyperparameter settings. Our candidate pool therefore includes RNAC-style robust policies, so the evaluation measures selection among robust candidates.

Proposed Experiments

Environments. Following State Revisit and Re-explore, "real" environments are HalfCheetah and Walker2d with these perturbations:

  • doubled gravity
  • friction reduced to 30%
  • Big Thigh and Long Torso morphologies

We add Gaussian action and observation noise so that real transitions are stochastic, which addresses the determinism argument of Zhang et al. The nominal MuJoCo model serves as the resettable simulator.

Real logs. Logs are drawn from D4RL-style behavior data (D4RL) rolled out in the perturbed environments. We subsample them to sizes from a few thousand to a million transitions.

Candidates. Several dozen candidates are trained purely in the nominal simulator: PPO and SAC across seeds, hyperparameters, and checkpoints, plus a robust variant. Ground truth is each candidate's mean return from fresh rollouts in the perturbed environment.

Metrics.

  • Spearman rank correlation between score and true return
  • Top-1 regret, normalized on the D4RL 0–100 scale
  • Top-5% hit rate, for comparability with Supervised Bellman Validation's reported figure

Decisive checks. Three checks decide the question.

  1. Plot the advantage over Supervised Bellman Validation against real-log size. The advantage must be positive at small sizes and shrink toward zero as the log grows.
  2. Plot the advantage against perturbation magnitude, which should reveal where the anchor stops helping.
  3. On the reduced-friction setting, check whether the fallback rule correctly detects non-smooth gaps.

Hardware extension (optional). If simulation results are positive, we will attempt a small hardware check on a balance task analogous to SR²'s wheel-legged benchmark. There, SR² baselines "lost balance within 5–15 seconds", so selection errors are visible.