Robots trained in a physics simulator often fail on real hardware because the simulator gets the dynamics subtly wrong: actuators respond late, joints stick, friction changes with speed. A common fix is to record a few minutes of real robot motion and train a small neural network, the "residual model," that adds a correction to each simulator step. The policy is then trained inside this corrected simulator. The flaw is that the residual model is usually trained one step at a time, with the true recorded history as input. During policy training, though, the corrected simulator runs for hundreds of steps on its own outputs. Small errors feed back as inputs the model never saw, and the rollout drifts. This is the same failure that sequence models show when trained on ground-truth text and then asked to generate from their own output. That failure is called exposure bias, and scheduled sampling is its standard remedy. Scheduled sampling is a training curriculum that increasingly replaces true past inputs with the model's own predictions.
This note proposes training residual sim-to-real models with scheduled sampling, where the probability of feeding back the model's own prediction depends on position in the rollout. It also proposes testing whether the resulting cut in long-horizon simulation error actually yields better policies on the real system. The claim is narrow on purpose. Scheduled sampling belongs on the simulator side of sim-to-real, not the policy side. The experiments are built to show when it beats simpler fixes: identifying physical parameters, randomizing them, or backpropagating through unrolled rollouts.
Why the simulator, not the policy
The literature suggests applying scheduled sampling directly to visuomotor policies trained on logged demonstrations. We think that formulation is ill-posed. In an offline log, if the policy's own action replaces the demonstrated one, the next recorded state no longer follows from the input, so the training target is wrong. Fixing that requires an interactive expert, which is DAgger (dataset aggregation, where an expert relabels the states the learner visits). A dynamics model has no such problem. Its training target is always the next recorded real state given the recorded real action. Replacing a past state in its input history with its own prediction leaves the target well defined. This is exactly the setting in which High-fidelity data-driven dynamics model for reinforcement learning-based control in HL-3 tokamak used scheduled sampling. That work trained a purely data-driven LSTM on real tokamak telemetry and reported that scheduled sampling, self-attention and ensembling together cut mean absolute error by at least 58.8% against a standard LSTM. A reinforcement-learning agent trained in that model then controlled 17 coils at 1 kHz for up to 400 ms on the live reactor.
That result motivates this proposal but leaves three questions open. The paper does not separate the contribution of scheduled sampling from the other two ingredients. It replaces the physics simulator entirely, which needs large logged datasets. It does not test whether lower open-loop error is what drove the transfer. We keep the physics simulator and learn only a correction from a small real log. We isolate scheduled sampling. We measure transfer directly.
Core Claim
Let $s_t$ be the robot state and $a_t$ the commanded action. The hybrid simulator predicts
$$\hat s_{t+1} = f_{\text{sim}}(s_t, a_t) + g_\theta(\tilde h_t, a_{t-L:t}),$$
where $f_{\text{sim}}$ is the unmodified physics engine and $g_\theta$ is the learned residual. The residual conditions on a length-$L$ history $\tilde h_t$ of states and on the last $L$ actions. History matters because the gaps we target, such as latency and backlash, depend on the past. During training, each state in $\tilde h_t$ is the recorded real state with probability $p(k, e)$, and otherwise it is the hybrid model's own earlier prediction. Here $k$ is the step index within a training window and $e$ is the training epoch. Following Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks, $p$ starts near 1 to avoid the cold-start collapse that paper documents and then decays over epochs. Following Scheduled Sampling Based on Decoding Steps for Neural Machine Translation, $p$ also decreases with $k$, so late steps in a window see more self-generated history. Predictions fed back are treated as constants, with no gradient through them.
We claim three things. Relative to one-step teacher forcing, this training cuts long-horizon rollout error of the hybrid simulator. That cut translates into higher real-system success for policies trained inside it. The advantage over parametric fixes is largest when the gap is non-parametric, meaning state-dependent friction or backlash rather than a single latency constant.
Research Program
The motivating failure comes from Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation. That paper trained tactile policies in PyBullet and translated real TacTip images into simulated depth maps with pix2pix, which transferred four of five tasks. Object balancing failed because "latency and joint limits prevented it from matching the high-agility movements required by the simulated policy." Image translation corrects what the robot sees, not how the arm moves. Our residual model targets exactly the part it leaves uncorrected.
The build has four parts, and every experimental arm shares all of them. The first is a residual network, a small GRU or temporal convolution network, sitting on top of the Tactile-Gym PyBullet environments and the Isaac Gym Unitree A1 environment used in World Model-based Perception for Visual Legged Locomotion. The second is a data-collection protocol that records a fixed budget of target-system trajectories under a behavior policy with exploration noise. The third is an ensemble of five residual models, as in the HL-3 work. The policy is penalized for entering states where the ensemble disagrees, because policies will otherwise exploit places where the residual is extrapolating. The fourth is PPO (proximal policy optimization) policy training inside each hybrid simulator.
There are two stages of evaluation. Stage one is sim-to-sim, which gives clean ground truth. The "real" system is the same simulator with hidden, unmodeled perturbations that the nominal simulator lacks. There are two families. The parametric family uses a fixed actuator latency of 10–40 ms and a changed friction constant. The non-parametric family uses gear backlash, speed-dependent friction and soft joint limits. Stage two runs on physical hardware: the UR5 with TacTip for edge following and object balancing, and the A1 for rough-terrain walking.
We also allow two rounds of real-log refresh. Each round collects new target data with the current policy, in the few-shot spirit of HL-3's finding that 2–4 extra shots cut prediction error by 90% on unseen discharges. The refresh applies equally to all arms.
Baselines
The naive reference point trains the policy in the unmodified simulator with no residual and deploys it zero-shot. This is the setting in which Church et al. report roughly 1.5 mm mean edge-following error and 0.57 mm surface-following depth error on the real TacTip. They give no reported success number for object balancing, which they describe as a failure. Beating this reference means succeeding on balancing and on the perturbed sim-to-sim targets, where the unmodified-simulator policy should degrade.
The key ablation is the identical residual model trained with one-step teacher forcing. It uses the same data, architecture, ensemble and tuning budget, and differs only in the input-feeding scheme. It isolates scheduled sampling. If the scheduled-sampling version does not beat it on 100-step and longer rollout error and on target-system return, the claim fails.
A second ablation injects Gaussian noise into the history inputs instead of the model's own predictions. It tests whether "realistic" self-generated error matters or any perturbation would do. This mirrors the argument in Multi-Step Denoising Scheduled Sampling: Towards Alleviating Exposure Bias for Diffusion Models that simulated accumulated drift beats plain Gaussian perturbation. On CIFAR-10 with a 10-step DPM-Solver, that method moved FID from 26.08 to 12.11.
The strongest competitors are three, and each gets the same real-data budget and hyperparameter search budget, measured in trials. The first is system identification: fitting the simulator's latency, friction and damping to the real logs, then training the policy with modest domain randomization around the fitted values. The second is a residual trained with a fully unrolled multi-step loss, with gradients backpropagated through the whole rollout window. The third is the HL-3-style purely learned simulator with no physics engine underneath. No reported result was found on Lacuna for system identification or unrolled-loss residuals on Tactile-Gym or the A1 benchmark, so we will produce these numbers ourselves. The closest reported figure for the approach we build on is the HL-3 claim of at least a 58.8% MAE reduction over a standard LSTM. That figure bundles three ingredients and is on a different system.
Winning means two things. In the non-parametric family, the scheduled-sampling residual must match or exceed system identification on target return with a 95% bootstrap interval excluding zero. It must also match the unrolled-loss residual at equal or lower wall-clock cost.
Why not just…?
Why not just identify the physical parameters, or randomize them? If the real gap is a single latency constant, system identification should win, because it has the right structure built in and needs almost no data. We expect this and predeclare it. The parametric family exists so that system identification can win there. The proposal's value rests on the non-parametric family and on hardware, where backlash and state-dependent friction do not fit a few parameters. If system identification wins there too, the idea is not worth its complexity, and we will report that.
Why not just backpropagate through unrolled rollouts? This is the most serious threat. States are continuous, so the non-differentiability that motivated scheduled sampling in text does not apply. An unrolled loss is also an unbiased multi-step objective, whereas Professor Forcing: A New Algorithm for Training Recurrent Networks points out that scheduled sampling is biased. Our bet is practical. Over windows of hundreds of steps through a stiff contact simulator, gradients through the rollout become unstable and expensive, and cutting the gradient at fed-back states avoids that. The head-to-head sweep over window lengths of 20, 100 and 400 steps settles it. If the unrolled loss wins at every length at matched compute, we adopt it and the contribution becomes the transfer study itself.
Why not just use DAgger in simulation, or fine-tune the policy on real data? DAgger corrects the policy's drift under the simulator's physics. Synthetic vs. Real Training Data for Visual Navigation shows this works well when those physics are benign: its Habitat-trained FAINT policy reached 90% real-world success. When the simulator's dynamics are wrong, as in the balancing task, DAgger teaches recoveries that do not exist on hardware. Real-world fine-tuning needs on-robot rollouts or teleoperation, which X-Sim (X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real) reports to be about ten times slower to collect than its alternative. We include a policy-fine-tuning arm that gets the same real-interaction budget as our log collection, so the comparison is on equal data.
Decisive Checks
The first check is mechanistic. We plot rollout error against horizon for teacher forcing, noise injection and scheduled sampling. We also compute a compounding ratio, the $k$-step error divided by $k$ times the one-step error. Scheduled sampling should flatten this ratio at a small cost in one-step error. That trade-off appeared in the original paper's TIMIT result, where frame error rate fell from 46.0% to 34.5% even though next-step training error rose.
The second check asks whether the rollout gain matters for control. Across all arms and both perturbation families, we correlate long-horizon hybrid-simulator error with target-system policy return. If the correlation is weak, lowering open-loop error is the wrong goal, whatever scheduled sampling achieves.
The third check is on hardware. Does the scheduled-sampling residual make TacTip object balancing succeed where the unmodified-simulator policy fails, and does it do better than system identification on the same logs? We run twenty trials per arm with the success criterion fixed in advance. A positive result on balancing, together with a sim-to-sim ranking that matches the non-parametric prediction, would support the claim. A system-identification win on hardware would refute it.
References
- High-fidelity data-driven dynamics model for reinforcement learning-based control in HL-3 tokamak
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Scheduled Sampling Based on Decoding Steps for Neural Machine Translation
- Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation
- World Model-Based Perception for Visual Legged Locomotion
- Multi-Step Denoising Scheduled Sampling: Towards Alleviating Exposure Bias for Diffusion Models
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Synthetic vs. Real Training Data for Visual Navigation
- X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real