Pessimism Only Where the Simulator Is Uncertain: A Saddle-Point Critic for Sim-to-Real Transfer

Robots and control policies trained in a simulator often fail on real hardware because the simulated physics (friction, mass, joint damping) differ from reality. A common recipe trains on plentiful simulated transitions plus a small log of real ones. It then corrects for the mismatch with a classifier that estimates how much more or less likely each simulated transition would be in the real world. The dominant way to use that classifier, DARC, turns it into a reward penalty. The policy therefore learns to avoid any region where simulation and reality disagree, including regions where the disagreement is large but has been measured precisely. We propose to use the same ratio for a different job: to correct the value estimate, and to charge a penalty only for what the real data cannot pin down.

The mechanism comes from primal-dual reinforcement learning, which recasts "make the Bellman equation hold" as a two-player game. One player, the value function, reduces the Bellman error. The other player, a dual weight function, looks for the states and actions where that error matters most. We give the dual player a third partner: an adversary that may perturb the sim-to-real transition ratio, but only inside a confidence band set by how much an ensemble of classifiers disagrees. Because of the game's structure, the adversary's influence is weighted by where the evaluated policy actually goes. The result is a lower bound on real-world return that fully corrects mismatches it can measure and is pessimistic only about uncertain transitions the policy relies on. We test this first as a policy-selection tool and then as the critic of an offline actor-critic learner on the ODRL off-dynamics benchmark.

Core Claim

Off-dynamics methods currently choose between two failure modes. Reward-penalty methods fall into a conservatism trap. DARC adds the log transition ratio $\log p_{\text{target}}(s'\mid s,a)/p_{\text{source}}(s'\mid s,a)$ to the reward. This works well when little target data is available, but, as MOBODY documents, it steers policies away from high-reward regions where the dynamics differ. Methods that trust the classifier more are exposed to its hallucinations when the target log is small, the classifier over-optimism problem identified by Dual Action Policy.

Our claim is that separating known mismatch from uncertain mismatch resolves most of this tension. The separation also has a natural home in the saddle-point view of the Bellman equation. There, the dual variable plays the role of a visitation weight, which Minimax Weight and Q-Function Learning (MWL/MQL) shows for single-dynamics off-policy evaluation. Concretely, the dual variable tells the ratio adversary where its perturbations actually change the target return. Pessimism then lands only on uncertain transitions that the policy visits and that lead to valuable states. It does not land uniformly on every simulator transition that looks unrealistic.

The Estimator

Let $\mathcal{D}$ pool the source (simulator) transitions and the target (real) transitions, each tagged with its domain. Let $\rho(s,a,s')$ denote the transition ratio $p_{\text{target}}/p_{\text{source}}$. On target-tagged tuples we fix $\rho = 1$, so real data tests the real Bellman equation directly. This also covers target transitions the simulator cannot produce, which is where DARC's support assumption breaks.

For a policy $\pi$, value function $Q$ and dual weight function $w$, we use the regularized Lagrangian

$$ L(Q,w;\rho) = (1-\gamma)\,\mathbb{E}_{s_0}[Q(s_0,\pi)] + \mathbb{E}_{\mathcal{D}}\big[w(s,a)\big(r + \gamma\,\rho(s,a,s')\,Q(s',\pi) - Q(s,a)\big)\big] - \tfrac{\alpha}{2}\,\mathbb{E}_{\mathcal{D}}[w(s,a)^2]. $$

Here $\gamma$ is the discount, $s_0$ is drawn from the start distribution, $Q(s',\pi)$ averages $Q$ over the policy's next action, and $\alpha$ sets a strongly convex penalty on the dual. We take that penalty from Beyond the Return and SBEED, which use it to stabilize this kind of game. Because the dynamics expectation enters linearly, every gradient is unbiased from a single logged tuple. This is the property Proximal Gradient TD and its finite-sample analysis exploit for linear gradient-TD methods, and SBEED extends to neural networks.

The ratio is not fixed. An ensemble of DARC-style classifiers gives a mean log-ratio $m(s,a,s')$ and a spread $\sigma(s,a,s')$. We let an adversary choose $\log\rho \in [m-\kappa\sigma,\ m+\kappa\sigma]$, subject to the requirement that $\rho$ remain a valid ratio, $\mathbb{E}_{s'\sim \text{source}}[\rho \mid s,a]=1$. We enforce that requirement with its own Lagrange multiplier function $\lambda(s,a)$. The requirement matters: without it, the adversary could shrink every ratio at once and the bound would become vacuous. The reported lower bound is

$$ \hat J_{\text{lo}}(\pi) = \min_{\rho \in \mathcal{C}_\kappa}\ \text{Saddle}_{Q,w}\ L(Q,w;\rho), $$

where $\mathcal{C}_\kappa$ is the confidence set just described.

The derivative of $L$ with respect to $\rho$ at a tuple is $\gamma\, w(s,a)\, Q(s',\pi)$. In words, the adversary spends its budget where the dual weight is large, meaning the policy visits there, and where the next state is valuable. This is the formal version of "pessimism only where it matters." Setting $\kappa = 0$ gives a plug-in estimator that corrects for dynamics using the mean ratio with no pessimism, and it is our key ablation.

For control, an actor ascends $\hat J_{\text{lo}}$ through $Q(s,\pi)$, as in an ordinary actor-critic. The saddle-point critic replaces SAC's semi-gradient critic, which DARC, DAP and the ODRL baselines use.

Research Program

The work has three parts. The first builds the estimator. The value function is a neural network. Because MWL/MQL and the file's open-problems summary both flag instability with over-parameterized adversaries, the dual $w$ and the multiplier $\lambda$ start as random-feature (kernel-like) classes, and we move to neural duals only if the random-feature classes prove too restrictive. Ratios are clipped at a fixed ceiling, and we report sensitivity to it.

The second part analyzes the estimator in the linear case. There, the inner game has a closed form. We want to show that the bias of $\hat J_{\text{lo}}$ is controlled by $\kappa$ times the dual-weighted classifier spread, rather than by the raw shift magnitude that controls DARC's penalty. We do not expect rates faster than those in Beyond the Return, which proves $O(n^{-1/4})$ for recovering the full value function. That result constrains our claims to bounds on the scalar return. The linear distributionally robust analysis of We-DRIVE-U is the natural point of comparison. Its uncertainty set is a total-variation ball fixed in advance. Ours is shaped by data and weighted by visitation.

The third part runs the empirical program below. It evaluates policies before optimizing them, because a pessimistic estimator that does not rank policies correctly will not train them correctly either.

Baselines

The naive reference points are two. The first is source-domain return, which ODRL reports correlates poorly with target return. The second is offline RL on the small target log alone, using IQL with the same network sizes.

The simple competitor we most need to beat is training on a mixture of source and target data. ODRL reports that "simple baselines, such as training on a mixture of source and target data, often remain competitive with specialized, dynamics-aware algorithms." We run it as IQL on the pooled data, with a tuned mixing ratio searched over the same budget we give our own hyperparameters.

The reward-penalty family is represented by DARC and by DAP+U. DAP+U is DARC with decoupled actions and ensemble-based resampling. Its authors report it "achieved double the return of the next best baseline" in several MuJoCo shift tasks and nearly matched an oracle on Hopper and Walker2d. We reimplement both with their published regularization (DAP's $\lambda \approx 0.10$) and give them the identical classifier ensemble we use. That way, any difference comes from how the ratio is used, not how it is estimated.

The strongest existing method in the offline-offline regime is MOBODY. With 1,000,000 source and 5,000 target transitions on ODRL, it reports a "44% average improvement over the best baseline (REAG) on gravity and friction tasks," a 25% improvement on kinematic and morphology shifts, and best-or-second-best results in 28 of 32 gravity/friction scenarios. No reported absolute normalized return for MOBODY was found on Lacuna, so we rerun it from its described configuration under the same data, seeds and compute. Beating it means higher mean normalized target return over ODRL's gravity/friction suite, with non-overlapping 95% bootstrap intervals over ten seeds, and no worse on kinematic/morphology shifts.

Two ablations isolate our ingredients. Setting $\kappa=0$ asks whether the adversarial ratio matters. Replacing the saddle-point critic with fitted-Q regression onto $\rho$-weighted targets, with $\rho$ taken as the classifier mean, asks whether the game matters at all.

Why not just…?

Why not just plug the classifier ratio into an ordinary critic as an importance weight on the next-state value? This is the most serious threat. The regression target $r + \gamma\rho V(s')$ is already correct in expectation, so double sampling is not the issue, and fitted-Q with $\rho$ weights has no need for a dual variable. What it lacks is any mechanism for classifier error. It inherits the over-optimism DAP documents whenever the ensemble mean is wrong on transitions the target log never covered. The $\kappa=0$ and fitted-Q ablations answer this directly. We expect them to match our method with the full 20,000-transition target log and fall behind as the log shrinks toward 1,000 transitions, the regime where DAP reports classifiers become unreliable. If they do not fall behind, the adversarial component is unnecessary and we will say so.

Why not just mix source and target data? ODRL's finding that mixing is competitive constrains us. MOBODY's 200:1 source-to-target ratio, however, means the pooled gradient is dominated by simulator physics. That hurts most on large gravity and friction shifts, exactly where a known ratio should help most. The mixing baseline, tuned over mixing ratios, is run on every task. We report per-shift-magnitude breakdowns so that a win confined to small shifts is not mistaken for a general one.

Why not just use DARC or DAP with a better-tuned penalty? Any additive log-ratio penalty treats a well-measured large shift the same as an uncertain one. Tuning its weight moves the policy along the trade-off between conservatism and optimism without leaving it. The specific test is the gap between our method and DARC as shift magnitude grows while target data is held fixed. This is the conservatism-trap regime MOBODY targets.

Why not just use MOBODY's learned latent dynamics model? MOBODY does escape the conservatism trap, but it does so by imagining rollouts through a learned model. Such rollouts accumulate error with horizon and require identical state spaces across domains. Our method never rolls out a model. If MOBODY wins on long-horizon Adroit tasks, that would suggest model imagination supplies coverage that no ratio correction can. We would report that as a limit of the approach.

Why not just take the minimum over an ensemble of critics? Ensemble-minimum pessimism is uniform: it penalizes wherever the critics disagree, whether or not the disagreement comes from dynamics mismatch. We include min-over-five-critics on the $\kappa=0$ estimator. Our method should beat it specifically on tasks where source data is abundant and critic disagreement stems from sparse policy coverage rather than from sim-to-real mismatch.

Proposed Experiments

All experiments use ODRL's MuJoCo locomotion tasks (Ant, Hopper, HalfCheetah, Walker2d) under gravity, friction, broken-joint and morphology shifts, plus Adroit Pen and Door. The data follow MOBODY's protocol of 1,000,000 source and 5,000 target transitions across ODRL's random, medium and expert tiers. Because the "real" domain here is a shifted simulator, ground-truth target returns come from rollouts, which is what makes the evaluation study possible.

The first experiment is off-dynamics policy evaluation. We train a pool of roughly fifty policies per task from checkpoints of every baseline, then estimate each policy's target return with $\hat J_{\text{lo}}$, the $\kappa=0$ plug-in, fitted-Q on mixed data, fitted-Q on target only, DARC-adjusted fitted-Q, and plain MQL that ignores the shift. We report the Spearman rank correlation with true target return and the regret of selecting the top-ranked policy. We also report lower-bound validity: the fraction of policies whose true return lies above $\hat J_{\text{lo}}$. Validity should approach one as $\kappa$ grows, while rank correlation should degrade gracefully. Source return is the reference that ODRL says fails at this.

The second experiment is control, in the same regime, against the baselines above. It adds DAP's setting of online source interaction with 20,000 offline target transitions, ablated down to 1,000. Here we compare to DAP+U under its own protocol. DAP notes that HalfCheetah "remains a challenge for all current methods," and we report it separately rather than averaging it away.

Two decisive checks decide the proposal. The first holds target data fixed and sweeps ODRL's gravity scale from mild to 5.0x. Our method should keep its lead over DARC and fitted-Q on mixed data as the shift grows, while DARC's return should fall. If our method falls just as fast, the conservatism-trap claim is wrong. The second shrinks the target log from 5,000 to 1,000 transitions. The gap between full $\kappa$ and $\kappa=0$ should widen as the log shrinks. If it does not, the uncertainty-weighted adversary is not earning its complexity, and the simpler plug-in estimator is the real contribution. All of this is sim-to-sim; a physical-hardware check such as DROPO's Franka pushing task is left for follow-up once the simulated evidence is in.