Robots trained in simulation now often carry a learned "world model": a small network that predicts what happens next and how good it is, used by a planner to choose actions on the real machine. When such a robot reaches reality, the usual practice is to fix what the simulator got wrong about physics and to leave the model's judgement of good and bad behaviour untouched. That judgement can also be wrong. A simulator cannot measure battery drain, motor heating, or whether a screw actually seated. For battery drain the error is large: the textbook energy proxies that simulators can compute correlate negatively with the power a real quadruped actually draws (Fine-Tuning Hard-to-Simulate Objectives for Quadruped Locomotion). A robot that plans against a frozen, wrong reward will optimize the wrong thing, however well its dynamics are corrected.
We propose to keep the simulation-trained reward and value models frozen and to learn, online on the robot, a small residual correction to them. The correction is trained from whatever sparse real signal the hardware provides. The planner may trust it only inside the states real data has actually covered. Because the robot plans at every step, a corrected reward changes its behaviour within minutes, with no retraining in simulation. Because the base models never move, the collapse seen when simulation value functions are unfrozen is avoided by construction. The claim to test is concrete: this bounded correction should match the offline "real-to-sim" relay on a hard-to-simulate objective in a fraction of the wall-clock time and robot wear, and it should not let the planner exploit the correction itself.
Core Claim
Two results in the file pull in opposite directions. Simulation Distillation (SimDist) pretrains a full world model in simulation: an image encoder, a latent dynamics model, a reward model and a value model. On the real robot it updates only the dynamics, and a sampling-based planner (Model Predictive Path Integral control, MPPI) chooses actions. Its ablations show that unfreezing the value function or encoder causes collapse. This paper constrains our design: we may not naively fine-tune these components. The quadruped power study motivates the proposal, because it shows the reward itself has a sim-to-real gap. That study closes the gap only through an offline relay. It collects real battery-current logs, trains a sequence-model "virtual sensor" offline, inserts it into Isaac Gym, and retrains the policy with PPO under a KL penalty.
Our hypothesis is that the reward gap can be closed online without touching the frozen models. The planner scores a candidate trajectory with
$$\hat r(s,a) = r_{\text{sim}}(s,a) + g(s,a)\,\delta_\phi(s,a) - \beta\,u(s,a),$$
where $r_{\text{sim}}$ is the frozen simulation reward and $\delta_\phi$ is a small residual network trained on real transitions. The gate $g\in[0,1]$ reports how well real data covers $(s,a)$. It is computed from ensemble agreement of $\delta_\phi$ in the frozen latent space. The penalty $u$ is ensemble disagreement, and $\beta$ sets how strongly the planner avoids states where the correction has not been checked. The terminal value receives the same kind of correction. A residual value head is trained by short-horizon temporal-difference learning on real data, with the frozen simulation value as its base, and is regularized toward zero. The idea borrows TRANSIC's structure. TRANSIC adds a gated residual to a frozen simulation policy using human corrections. We apply the gated residual to the objective instead, so the robot can keep judging its own behaviour after supervision stops.
Research Program
We first build the base system following SimDist's recipe: world-model pretraining in Isaac Gym (Isaac Gym), then real deployment with MPPI and dynamics-only system identification. On top of it we add the residual reward and value heads, the coverage gate, and the disagreement penalty. The robot learns them online from a real scalar signal logged alongside its states.
The main testbed is the Unitree Go1 energy task from the power study. Its real signal is 50 Hz battery current, and simulation is structurally blind to it. On this testbed the relay paper's reported numbers serve as a clear external bar.
The research questions follow from the design. First, does the gate stop the planner from rushing into regions where an untested $\delta_\phi$ spuriously predicts savings? This is the robotics form of the reward hacking that motivates online reward learning in Deep Reinforcement Learning from Human Preferences and Reward Learning on Policy. Those works refine rewards on current-policy samples for Atari-style control and language models respectively. Neither faces a frozen simulation prior or physical exploration cost. Second, does the residual stay small and stable as data accumulates, or does it drift?
Before touching hardware, we run a controlled sim-to-sim study. The "real" environment is a second Isaac Gym instance with a hidden cost term: a nonlinear function of torque, velocity and a slow thermal state, standing in for electrical losses. The pretrained model never sees this term. The study gives exact ground truth, so we can measure how well the correction tracks the hidden cost and how often the planner exploits it. The analysis output is a curve of true-objective improvement against real minutes, for each ablation.
A secondary, smaller test checks that the mechanism is not specific to an objective measured by a physical sensor. It uses the UR5e manipulation setting shared by SimDist and PRISM. There the real signal is a sparse binary success/failure label from a human pressing a button at episode end. This is the setting where the reward models of PRISM and RoboFuME stay frozen after deployment. PRISM's reward is trained on labels from vision-language models queried in simulation, and RoboFuME's is a vision-language classifier calibrated offline.
Baselines
The naive reference point is SimDist unchanged: frozen reward and value, dynamics-only adaptation, and MPPI. It is the strongest published recipe for fast world-model transfer. It reports success "often around 2x higher than baselines" with 15–30 minutes of real data, but on manipulation. There is no reported result found on Lacuna for SimDist on quadruped energy. We run it with the same pretrained model, the same real-data budget, and the same planner settings. On the energy task it effectively optimizes whatever proxy the simulation reward contains, so we train its simulation reward once with mechanical power and once with no energy term.
The analytical-proxy policy is the second reference. It is PPO trained in simulation on mechanical power with domain randomization. The power study reports that it achieved "only … a 5–12% net reduction" on the Go1.
The strongest existing method on this question is the offline relay of Fine-Tuning Hard-to-Simulate Objectives. It reports a "24-28% net reduction in total power consumption from the battery pack at various speeds" and "~20% reduction in total battery drain." We re-run it on our Go1 under a matched budget. It gets the same total minutes of real battery-logged walking that our method uses. It gets the same reward-model architecture budget. It gets as many relay cycles as fit in the same wall-clock time, with its simulation retraining time counted. Our method beats it if it reaches at least the relay's net power reduction, at matched speed-tracking error, using no more real minutes and substantially less total wall-clock time. Alternatively, it beats the relay if it gets strictly more reduction at equal real minutes. Matching the relay's final number with less time counts as a positive but modest result.
Three ablations isolate the key ingredient. The first unfreezes and fine-tunes $r_{\text{sim}}$ and the value directly on real data, with no residual, no gate and no penalty. It tests whether the residual structure is needed at all. The second keeps the residual but removes the gate and penalty. It tests whether coverage-bounding is what prevents exploitation. The third corrects the reward but not the value. It tests whether a stale terminal value undoes a short-horizon reward fix.
Why not just…?
Why not just use the offline relay that already works? It is a real threat: it produced a 24–28% net reduction, and our method may not beat its final number. Its weaknesses are cost and latency. Every correction needs a full simulation retraining and a new deployment, and the policy only ever sees the reward model's errors on the previous policy's states. The matched wall-clock comparison above settles the question. If the relay reaches the same saving in comparable total time, we will report that the online variant is not worth its complexity for sensor-measured objectives. The remaining case for it would then be objectives like sparse human success labels, where a regression "virtual sensor" is harder to fit offline.
Why not just unfreeze the reward and value models and fine-tune them? SimDist's ablations show that unfreezing the value function or encoder collapses performance in exactly this planning setup. Nothing in the file, however, shows the same for the reward head alone. The first ablation tests this directly. If plain reward fine-tuning with a small learning rate performs as well as the residual, the residual structure is unnecessary and we will say so.
Why not just freeze the reward and adapt dynamics, as SimDist does? Correcting dynamics cannot fix an objective that the simulation reward never contained. The power study shows the natural proxy is negatively correlated with real battery draw. The frozen-reward baseline measures how much of the saving is available from dynamics correction alone. We expect little, but the experiment decides.
Why not just randomize the simulator harder? Domain randomization spreads the policy over plausible physics. It cannot introduce a quantity the simulator does not compute, such as battery current. Isaac Gym delivers throughput, not missing physics. We include a heavily randomized proxy-reward policy in the analytical-proxy baseline, so the question is answered empirically rather than argued.
Proposed Experiments and Decisive Checks
The sim-to-sim hidden-cost study comes first and is decisive for the mechanism. We record the true hidden cost achieved as a function of real steps for the full method, each ablation, and frozen-reward SimDist. We also record the rate at which the planner visits states where $\delta_\phi$ predicts a large saving that the true cost contradicts. The idea is supported if the no-gate ablation shows such exploitation episodes and the full method does not, while both improve the true cost. If the gate only slows learning without preventing exploitation, the idea is wrong as stated.
On Go1 hardware we follow the power study's protocol: fixed forward speeds from 0.5 to 1.1 m/s, net and gross battery power measured from the pack, and 400–500 m outdoor runs compared by final state of charge. We report speed-tracking error and falls as safety guards, and real minutes and total wall-clock time per method. The main comparison is against the matched-budget relay described above.
On the UR5e we report task success after a fixed number of labelled real episodes. The comparison is SimDist frozen against the full method. It shows whether sparse binary labels are enough to learn a useful residual.
Across all settings, a final check tracks the norm of $\delta_\phi$ and of the residual value over time. A residual that grows without bound would mean the frozen prior is being overridden rather than corrected, and the proposal's stability argument would fail.
References
- Fine-Tuning Hard-to-Simulate Objectives for Quadruped Locomotion: A Case Study on Total Power Saving
- Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation
- TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction
- Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning
- Deep Reinforcement Learning from Human Preferences
- Fine-Tuning Language Models with Reward Learning on Policy
- PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrations
- Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning