Pessimism Where the Real Data Is Silent: Posterior-Conservative Domain Randomization

Robots are usually trained in simulators and then deployed on real hardware. The standard trick, domain randomization, trains the policy across many randomly perturbed versions of the simulator's physics (masses, friction, latencies) so that the real world looks like one more variation. A more recent variant fits those random ranges to a few seconds of real robot data. Both versions average performance over the simulators they sample. That hides a basic problem. A short real log pins down some physical parameters well and leaves others unconstrained, and which parameters matter depends on where the robot is. A friction coefficient never exercised in the log can decide the outcome in exactly the states the log never visited.

We propose to treat those unconstrained parameters the way offline reinforcement learning treats unseen actions. Offline reinforcement learning is learning a policy from a fixed dataset without new interaction, and its standard move is to be pessimistic about actions the data never shows. We apply the same pessimism to uncertain physics. There are two steps. First, a network trained only on self-labeled simulator rollouts infers a full posterior over physics parameters from a small, reward-free real log; the labels are free because the simulator knows its own parameters. Second, the policy is trained in simulation against a critic that is pessimistic in proportion to how much its value estimate disagrees across that posterior. This makes the policy cautious only in states where the real data fails to identify the dynamics, and nowhere else.

Core Claim

The claim is that, for a fixed budget of real-world data, local pessimism beats the two existing options. Local pessimism means penalizing value by its spread across plausible physics, state by state. The first existing option is to average over a data-fitted randomization distribution, as DROPO: Sim-to-Real Transfer with Offline Domain Randomization does. The second is to be uniformly robust to the worst case. The advantage should be concentrated in states the real log does not cover, and it should grow as the log shrinks or narrows in behavior.

The theory in Statistical Guarantees for Offline Domain Randomization and Provable Sim-to-Real Transfer via Offline Domain Randomization motivates this directly. Those papers show that under partial coverage, likelihood-based fitting converges only to an "identified set" of indistinguishable parameters. They leave open what a policy should do with that set, and that is the question we address.

Research Program

Step 1: amortized posterior from self-labeled simulation. This is the label-efficient ingredient. The randomized simulator produces unlimited pairs of short trajectory windows and the parameters $\xi$ that generated them. We train an encoder, pretrained with a next-state prediction objective, on these windows. The pretraining follows Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning, which showed that dynamics pretraining lets offline RL match full-data performance with 10% of the data. On top of the encoder sits a conditional density head that outputs $q(\xi \mid \mathcal{D})$ from a set of real transitions $\mathcal{D}$.

This replaces DROPO's derivative-free CMA-ES search over a Gaussian with a single forward pass. It also allows posteriors that are non-Gaussian and multimodal, and it scales past DROPO's 18 parameters. It differs from the physics-signature network in Latent Adaptation of Foundation Policies for Sim-to-Real Transfer because we output a distribution rather than a deterministic point. The spread of that distribution is the signal Step 2 needs.

Step 2: a parameter-conditioned critic with posterior pessimism. In simulation, parameters are sampled from $q$. A critic $Q(s,a,\xi)$ sees the true $\xi$, which is privileged information the policy never receives, in the spirit of the privileged critic in Asymmetric Actor Critic for Image-Based Robot Learning. The policy $\pi(a \mid h)$ sees only its observation history $h$ and is trained to maximize

$$J(\pi) = \mathbb{E}_{s,\, a\sim\pi}\Big[\, \mathbb{E}_{\xi\sim q}\, Q(s,a,\xi) \;-\; \beta\, \mathrm{Std}_{\xi\sim q}\, Q(s,a,\xi) \Big],$$

where $\beta \ge 0$ sets the degree of caution.

The standard-deviation term is small wherever the posterior is tight or where the physics does not affect the outcome. It is large only where unidentified parameters change the return. This is the dynamics analogue of the out-of-distribution action penalty in Conservative Q-Learning for Offline Reinforcement Learning.

Because RORL: Robust Offline Reinforcement Learning via Conservative Smoothing showed that point-wise pessimism produces jagged value functions that collapse under small state perturbations, we add RORL's asymmetric state-smoothing loss to the critic. That paper therefore constrains our design as well as supplying one of its components.

Step 3: calibration under misspecification. When the true physics lies outside the simulator's parameter family, amortized posteriors can become confidently wrong. We therefore temper $q$, meaning we flatten it by a single temperature, chosen so that held-out real transitions have calibrated likelihood under simulator rollouts. We report how much tempering each target needs, and treat that as a diagnostic of misspecification.

Baselines

The naive reference is uniform domain randomization over hand-set wide ranges, as in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. It uses no real data and marks the floor. On the Franka pushing task, DROPO reports roughly 10 cm final placement error for uniform randomization.

The strongest existing method is DROPO. It fits the mean and variance of a parameter distribution to the real log and then trains a policy averaged over that distribution. On the Franka Panda push with an unmodeled asymmetric center of mass, it reports about 2 cm placement error from 13 seconds of real data, against about 7 cm for DROID.

For the dynamics-shift benchmarks we also run two further baselines. The first is SR² from State Revisit and Re-explore, which penalizes simulator transitions that deviate from the real log. It reports superior or competitive performance on 20 of 23 modified MuJoCo tasks. The second is Found-adapt, which adapts a latent prompt at deployment. For normalized scores of either method on specific gravity or friction levels: no reported result found on Lacuna, so we rerun both.

We also run conservative offline RL (CQL and IQL) on the real log alone, as the no-simulator reference.

Fairness is enforced in four ways. Every simulator-based method gets the same real log, the same number of simulator steps, and the same policy architecture. Hyperparameters are tuned on a separate set of source-target pairs with an equal search budget per method. DROPO is given the same parameter family as our posterior.

Two ablations isolate the key ingredient. The first uses our posterior with $\beta=0$, which is posterior-averaged randomization with a better fitter. It separates the value of amortized inference from the value of pessimism. The second replaces the per-state spread with a global worst case over the posterior's 90% region, which separates local pessimism from pessimism as such.

Our method beats DROPO if it lowers target failure rate or raises target return at matched real-data budgets, with non-overlapping seed confidence intervals, and does not lose on full-coverage logs. On hardware, the bar is matching or improving on DROPO's roughly 2 cm.

Why not just…?

Why not just use DROPO's fitted distribution and train normally? This is the most serious threat. If the real log covers the states the task visits, the posterior is tight everywhere and our penalty vanishes, so the two methods should tie. The distinction appears only under partial coverage. We therefore vary coverage explicitly, and we predict that the gap between our method and the $\beta=0$ ablation will be near zero at full coverage and grow as coverage narrows. If it does not grow, the idea is wrong.

Why not just widen the randomization variance, or be robust to the worst case? Inflating the variance is DROPO's own mechanism for absorbing unmodeled physics. DROPO and the offline-randomization theory both note that overly broad randomization wastes probability mass and produces timid policies. Global worst-case training is the extreme form of this. The global-pessimism ablation and a variance-inflated DROPO tuned for the same failure rate test directly whether per-state pessimism buys return at equal safety.

Why not just let the policy identify the physics online from its history? A history-conditioned policy can infer parameters during deployment, as AdaptManip does with recurrent state estimation. Our policy is also history-conditioned, so this is not an alternative so much as a component. Online inference cannot help, however, when the first visit to a poorly identified regime is itself the failure, such as a fall or a dropped object. We report failures that occur before the policy's own parameter estimate has converged, with and without the penalty.

Why not just run offline RL on the real data, or behavior cloning? With 13 seconds to a few minutes of data, both are limited to what the log shows. D4RL documents that behavior cloning cannot stitch together behavior beyond its demonstrations. The real-log-only CQL and IQL baseline measures this gap directly.

Proposed Experiments

Sim-to-sim with known ground truth. We use the modified D4RL locomotion suites (HalfCheetah, Hopper, Walker2d) with the gravity levels G1–G4 and friction levels F1–F6 used by Found-adapt. The "real" system is a target simulator whose parameters are held out. The real log is drawn from D4RL-style behavior of controlled quality (random, medium, expert) and truncated to budgets from 13 seconds to 10 minutes. Behavior quality varies coverage, and the budget varies quantity.

Misspecified targets use the morphology shifts ("Big Thigh", "Long Torso") from PUORL and SR², which lie outside the randomized parameter family. They test Step 3.

Metrics are normalized target return, failure rate, and return as a function of real-data seconds.

The decisive mechanistic check. For each state in the target rollouts, we compare the posterior value spread $\mathrm{Std}_{\xi}\,Q$ against the realized error between the averaged value and the true target value. If the penalty is doing what we claim, the two should correlate, and the correlation should be strongest in states far from the real log under a nearest-neighbor coverage measure. We also split return gains by coverage bin. The method is supported only if its improvement over the $\beta=0$ ablation concentrates in the low-coverage bins.

Hardware. We reproduce the DROPO Franka pushing task with the unmodeled asymmetric center of mass. The real log is restricted to pushes in one region of the table, and evaluation includes targets outside it. We report placement error against DROPO and uniform randomization. A negative result, meaning no hardware gain despite the sim-to-sim gains, would indicate that real misspecification dominates the parameter uncertainty our method models. The tempering diagnostic from Step 3 should reveal that case.