Factoring the Reality Gap Out of Learned Rewards

Robots trained by reinforcement learning usually need two expensive human inputs. The first is a hand-written reward function, a formula that scores every moment of behavior. The second is a way to cope with the fact that the simulator's physics is wrong. These two problems are almost always solved separately. Yet both are often answered by the same small resource: a few dozen real-robot demonstrations. Methods that close the physics gap with real data, such as H2O and H2O+, assume the reward is already engineered. Methods that replace the reward with a model learned from demonstrations, such as adversarial imitation, assume the demonstrations and the training environment share the same physics.

This note proposes using the real demonstrations for both jobs at once, and argues that the naive way of doing so fails for a specific reason. In adversarial imitation, a classifier learns to tell demonstrator behavior from policy behavior, and its output becomes the reward. When the demonstrations are real and the policy acts in a flawed simulator, the classifier's easiest cue is "this looks like simulator physics." The policy cannot change that cue by acting differently. The fix is to split the classifier's score into two parts. The part the policy controls, what it chooses to do, becomes the reward. The part it cannot control, how the simulator responds, is routed into the value update as a dynamics correction.

Core Claim

Write $d_E(s,a)$ for how often the real demonstrator visits state $s$ and takes action $a$, and $d_\pi(s,a)$ for the same frequency for the current policy in the simulator. Let $P_{\text{real}}(s'\mid s,a)$ and $P_{\text{sim}}(s'\mid s,a)$ be the real and simulated probabilities of landing in next state $s'$. A classifier that compares whole transitions $(s,a,s')$ from real demonstrations with transitions from the simulated policy estimates the log-ratio

$$\log\frac{d_E(s,a)\,P_{\text{real}}(s'\mid s,a)}{d_\pi(s,a)\,P_{\text{sim}}(s'\mid s,a)} \;=\; \underbrace{\log\frac{d_E(s,a)}{d_\pi(s,a)}}_{\text{behavior term}} \;+\; \underbrace{\log\frac{P_{\text{real}}(s'\mid s,a)}{P_{\text{sim}}(s'\mid s,a)}}_{\text{physics term}}.$$

The identity follows directly from factoring each joint distribution into a state-action frequency times a transition probability.

Naive transition-level imitation rewards the policy with the sum of the two terms. A classifier on $(s,a)$ alone drops the physics term from the reward. It still lets that term distort which states the policy reaches, because the simulated policy's states drift under wrong physics, and nothing corrects the Bellman backups. We propose three steps. First, estimate the physics term with a separate real-versus-simulated dynamics classifier, the construction H2O uses. Second, subtract it from the transition classifier to obtain a reward made of the behavior term only. Third, use the physics term as an importance weight on simulated Bellman backups, in the style of H2O.

The new claim is that this split matters, and that its benefit grows with the size of the reality gap. A pipeline that learns the reward first and corrects dynamics afterwards should not match it. At zero gap, the split should reduce exactly to ordinary off-policy imitation.

Position Relative to Prior Work

Several papers motivate the gap. H2O trains in an imperfect simulator alongside a fixed batch of real transitions. It uses discriminators to penalize and reweight simulated transitions on a CQL backbone (conservative Q-learning, which suppresses value estimates for unseen actions). H2O+ replaces CQL with in-sample offline learning and anchors simulated Q-values to a value function fitted on real data. Both assume the same engineered reward in simulator and reality. We keep their dynamics-correction machinery and remove the reward assumption. This changes what the real data are for: they define the task, not just the physics.

Other papers supply the method. Sample Efficient Imitation Learning for Continuous Control turns adversarial imitation into an off-policy actor-critic. It reports reaching expert level on Ant with about 100 times fewer interactions than GAIL, but assumes no dynamics gap. We reuse its off-policy replay structure and its bounded-action policy. Our backbone otherwise follows standard soft actor-critic, without new bias-control tricks, so that gains are attributable to the reward split.

Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions is the closest deployed system. It learns a style reward from 4.5 seconds of motion capture and transfers to a real quadruped through domain randomization. It reports a cost of transport of 0.93 to 1.12 against 1.37 to 1.65 for hand-designed style rewards. AMP still hand-writes the task term and weights it against style. It also never uses real-robot data to correct physics; it relies on randomizing physics broadly.

PRISM replaces reward formulas with vision-language-model judgments on rendered views. It reports 82% real success from five demonstrations versus 13% for behavior cloning. However, it assumes the reconstructed simulator's contact physics is correct, which is exactly the assumption we target.

Reinforced Grounded Action Transformation corrects the simulator itself by remapping actions. It needs a hand-written reward for the policy, and it cannot represent unmodeled internal state. We treat it as a complementary dynamics fix rather than a competitor on reward.

Research Program

The method has three learned components trained alongside a soft actor-critic agent in the simulator. The first is a dynamics classifier pair, real versus simulated on $(s,a,s')$ and on $(s,a)$, whose difference estimates the physics term. It is trained on the real demonstration transitions and on the policy's simulated replay. The second is a transition classifier, real demonstrations versus simulated policy transitions, whose output minus the physics estimate gives the behavior-term reward. The third is a critic whose simulated backups are reweighted by the clipped exponentiated physics term. Clipping addresses the known variance of these ratios when simulator and reality barely overlap.

The analysis has three parts. The first measures, in the naive transition-level variant, what fraction of the reward's variance the physics classifier explains. If our argument is right, that fraction should rise with the gap. The second tracks whether policies trained with the naive reward drift toward states where simulator and reality happen to agree rather than toward task states. The third characterizes failure when the real demonstrations cover too few states for the dynamics classifier to be trusted. The demonstrations are both the expert data and the only real dynamics data, so poor coverage is the main expected failure mode.

Baselines

The naive reference point is soft actor-critic trained in the nominal simulator on the true hand-written reward and deployed unchanged. H2O reports that this "failed immediately" on its real robot. A second floor is behavior cloning on the real demonstrations alone.

The strongest existing method on dynamics is H2O/H2O+ given the true engineered reward. This is a privileged oracle, since our method never sees that reward. The comparison number is H2O's reported average return of 7085 on HalfCheetah-Gravity, against 6066 for CQL, from H2O. That figure used a large real dataset. We will rerun H2O+ on our own real-data budgets and use those reruns, not the quoted number, for curves. Coming within a modest margin of the reward-oracle H2O+ without any reward would count as success.

The strongest reward-free competitor is a two-stage pipeline. It learns a reward by off-policy adversarial imitation on the real demonstrations, in the style of Sasaki et al., then trains H2O+ with that frozen or co-updated reward. A further reward-free competitor is the AMP recipe: transition-level imitation against the real demonstrations with broad dynamics randomization and no real-data correction.

Few-shot behavior-cloning fine-tuning of a simulator-pretrained policy follows Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation. That paper reports parity with meta-learning at 10 or more trajectories but gives no return numbers; no reported result found on Lacuna for iMuJoCo returns.

Ablations isolate the key ingredient. They are: the full sum as reward (naive transition imitation), the $(s,a)$-only classifier as reward, the behavior-term reward without backup reweighting, and backup reweighting without the reward split.

All methods share the same real demonstrations, the same simulator step budget, the same actor-critic backbone and network sizes, and the same number of tuning trials per method on held-out variants. Each method gets at least five seeds.

Why not just…?

Why not just learn a reward from the demonstrations first and then run H2O+? This is the real threat. In the two-stage pipeline, the reward classifier is trained against simulated transitions, so it absorbs the physics term we argue is harmful. The split exists precisely to remove that contamination. If the two-stage pipeline matches the full method across the gap sweep, our claim is wrong, and we will report it that way. The physics-variance diagnostic shows directly whether the two-stage reward is contaminated.

Why not just randomize the simulator physics, as AMP does? Randomization worked for AMP's quadruped. The file notes, however, that randomization tends to make policies conservative under narrow stability margins. It also needs hundreds of millions of simulated steps, and it can only cover gaps inside the randomized parameter family. Our iMuJoCo variants include limb-length and joint-range changes, which a friction and mass randomization does not span. The AMP-style baseline on those variants tests whether real-data correction adds anything beyond randomization.

Why not just fine-tune on the real demonstrations with behavior cloning? Patacchiola et al. show this is competitive once 10 or more target trajectories exist, so it is a strong baseline in the high-data end. PRISM's 13% versus 82% suggests cloning breaks down with very few demonstrations and randomized starts. Our budget sweep from 1 to 50 real trajectories will show where, if anywhere, simulator interaction beats cloning.

Proposed Experiments

The main testbed is simulated-to-simulated transfer, where "reality" is an iMuJoCo variant of Hopper, HalfCheetah or Walker2d. Each variant comes with 100 expert soft actor-critic rollouts. The simulator is the nominal model, and HalfCheetah-Gravity is added to match H2O.

Real-data budgets are 1, 5, 10 and 50 trajectories. The primary metric is return in the target variant under the hidden true reward, normalized between a random policy and the variant's expert. Secondary metrics are simulator steps to reach 80% of final performance and the physics-variance fraction.

The decisive check is a gap sweep: gravity scaled from 1.0 to 2.0, and friction and limb-length offsets of increasing size. The prediction is that all reward-free variants tie at zero gap. As the gap widens, the full method's advantage over the naive transition reward and the two-stage pipeline should grow. The physics-explained variance of the naive reward should rise in step with that advantage. A flat advantage curve, or a two-stage pipeline that tracks the full method, would falsify the core claim.

Only if the simulated results hold will a hardware stage follow on an available arm or legged platform. Real-world success rate will be measured against H2O+ given an engineered reward.