Models that predict the future one step at a time, such as learned physics simulators, driving-scene generators and robot world models, become stable over long horizons when they are trained on deliberately corrupted histories. The model sees a slightly wrong past and learns to steer back toward realistic futures. The corruption used today is almost always isotropic Gaussian noise or a random walk, meaning equal-sized random perturbations in every direction. When such a model is trained in simulation and then fed real sensor histories, it meets off-distribution inputs of a very different kind. Real histories arrive late, lose points on glass and dark asphalt, carry calibration offsets and jitter, and deviate in structured, state-dependent directions that Gaussian noise almost never samples. We propose to learn the corruption itself. A small, constrained corruption model is trained, using only unpaired real sequences, to make simulated history windows statistically indistinguishable from real ones. That learned corruption then replaces Gaussian noise as the context noise during autoregressive training, so the model learns to recover from the specific errors it will actually see at deployment.
In two sentences: noise-augmented training already teaches autoregressive models to pull bad histories back toward good futures, but it teaches recovery from the wrong kind of badness. We fit the badness to real data, with no paired real/sim states and no simulator resets, and test whether a model trained this way beats both tuned Gaussian noise and hand-designed sensor randomization when conditioned on real histories.
Core Claim
A sizable share of the sim-to-real failure of autoregressive predictors comes from the input side: real conditioning histories sit off the simulated data in structured directions. This part of the gap can be closed by matching the distribution of context corruption, without modeling the real dynamics. The claim is deliberately scoped. We do not claim to fix unmodeled physics such as backlash or deformable contact, because a model trained to map corrupted histories to clean simulated futures will still predict simulated physics. The dynamics gap is left as a separate, measured residual.
Three lines of prior work supply the mechanism, and all three use stationary Gaussian noise. Learning to Simulate Complex Physics with Graph Networks adds random-walk noise to particle histories and obtains rollouts stable over thousands of steps. DPOT trains a PDE transformer to map noisy input frames to clean next frames. Diffusion Forcing gives each token its own diffusion noise level, which permits conditioning on partially noised histories at test time. None of these works leaves the domain it was trained in. Diffusion Forcing's real Franka result is trained on real teleoperation data, not transferred from simulation.
The motivation for learning structured corruption comes from sim-to-real perception. Adversarially Masking Synthetic to Mimic Real (ASM) learns, with a differentiable Gumbel-Softmax mask, which points to drop from synthetic LiDAR scans so that they resemble real ones. It raises SynLiDAR→SemKITTI segmentation from 15.2% to 22.8% mIoU. ASM is per-frame and feed-forward: it never touches temporal context or compounding error.
Two further papers constrain the design. Sim-to-Real Transfer with Neural-Augmented Robot Simulation (NAS) learns a sim-to-real residual, but only by resetting the simulator to the real state at every step. We rule out any design that needs paired states. World Model-based Perception for Visual Legged Locomotion (WMP) shows that a recurrent latent world model trained in simulation can run on noisy real depth. Its transition model, however, is trained without any context corruption. WMP is exactly the kind of system our method should plug into.
Research Program
The method has three parts.
The first part is a corruption model $c_\phi$. It takes a simulated history window $h = (x_{t-k}, \dots, x_t)$ and outputs a corrupted window $\tilde h = c_\phi(h)$. The corruption model is not a free generator. It is limited to a small set of physically interpretable operations:
- per-element dropout masks, conditioned on local state as in ASM;
- per-channel time shifts that model latency and packet delay;
- smooth low-rank additive drift that models calibration and bias;
- gain changes.
A budget term caps how much of the signal it may destroy. The restriction matters because it keeps the corruption model from erasing the signal or inventing new dynamics.
The second part is a discriminator $D$. It sees history windows, not single frames. Its job is to tell corrupted simulated windows from real windows, and it is trained only on unlabeled real sequences. ASM's lesson is that a discriminator will latch onto the "clean versus noisy" contrast. We turn that tendency into the training signal: the corruption model must remove exactly that contrast.
The third part is the autoregressive predictor $f_\theta$, which maps corrupted histories to clean simulated futures:
$$\min_\theta \; \mathbb{E}_{h,\,x_{t+1}\sim \text{sim}} \big\| f_\theta(c_\phi(h) + \epsilon_\sigma) - x_{t+1} \big\|^2, \qquad \min_\phi \max_D \; \mathcal{L}_{\text{adv}}(D; c_\phi(h), h^{\text{real}}) + \lambda\, B(\phi).$$
In these objectives:
- $x_{t+1}$ is the clean simulated next state.
- $\epsilon_\sigma$ is Diffusion-Forcing-style noise at a randomly sampled level $\sigma$.
- $\mathcal{L}_{\text{adv}}$ is a standard adversarial loss.
- $B(\phi)$ is the corruption budget, weighted by $\lambda$.
Two noise sources are kept on purpose. At deployment the history mixes real observations with the model's own earlier predictions. The learned corruption covers the observed frames, and the generic noise covers self-generated errors.
The analysis program asks where the learned corruption puts its mass. This is inspected directly: learned latency distributions, where the dropout masks fall, and the drift spectrum. We also measure the "gap budget", the part of real-log prediction error that remains after context matching, as a direct estimate of the dynamics gap that this method cannot fix.
Baselines
The naive reference is teacher forcing on clean simulated data with no context noise, which shows the full size of the problem.
The first serious baseline is Gaussian or random-walk context noise as in GNS and DPOT. Its noise scale gets the same hyperparameter-search budget as our $\lambda$ and budget terms, and it is also tuned directly on held-out real validation logs, which gives it an advantage. Beating it means lower multi-step error on real logs and higher closed-loop success at matched training compute.
The strongest existing method for the noise side is Diffusion Forcing with per-token noise levels. It is run with the same backbone, data and number of training steps, and its test-time history noise level is swept. Beating it means improving real-history rollouts without losing the in-simulation stability it provides.
On the driving benchmark, the reference is SMART, which already perturbs discrete motion tokens. It reports a Waymo Sim Agents realism meta-metric of 0.7614 when trained in-domain and 0.72 zero-shot when trained only on NuPlan. The target is to close a meaningful fraction of that 0.04 gap using unlabeled Waymo histories only, and to do so over our own re-run of SMART, not over the quoted numbers.
The competitor that most threatens the claim is hand-designed structured randomization. Latency, dropout and bias are randomized over ranges chosen by an expert, in the style of standard domain randomization. It is given the same corruption operations our model uses, but with uniform sampling in place of the learned distribution. This is the key ablation: it isolates whether the learning from real data matters or only the choice of corruption types.
A second ablation keeps the learned corruption but removes the adversary and fits the corruption parameters by simple moment matching to real statistics.
The oracle trains or fine-tunes on real sequences with future labels. It bounds what is achievable.
Why not just…?
Why not just tune the Gaussian noise scale? Gaussian noise spreads its mass evenly in every direction. A latency shift or a structured dropout pattern is a large, low-dimensional move that isotropic noise at any tolerable scale rarely produces. The research file also notes that heavy noise over-smooths and makes models ignore their history. So raising the scale should buy robustness to real inputs at the cost of in-simulation accuracy. We test this directly by tracing the curve of real-log error against simulation error across noise scales. The claim fails if some Gaussian setting lies on or beyond our curve.
Why not just randomize sensor latency, dropout and bias by hand? This is the real threat, and we treat it as the main ablation, not a strawman. DrEureka reports that automated, physics-grounded randomization ranges beat expert human-tuned ones on hardware: 34% higher velocity and nearly 300% more cube rotations. That result suggests hand ranges are often misplaced, but it concerns policies, not autoregressive predictors. If expert-ranged randomization matches learned corruption on our testbeds, the honest conclusion is that corruption type matters and learning adds nothing. We would report that conclusion. The controlled sim-to-sim setting below, where the true corruption is known, tells us whether the learned model recovers ranges and state dependence that a hand designer would plausibly miss.
Why not just learn a sim-to-real residual as in NAS, or fine-tune on real data? NAS needs the simulator reset to the real state at every step, which is impossible for depth images or uninstrumented driving logs. Its residual network is itself an autoregressive predictor trained without corruption. Our method uses only unpaired real histories. Fine-tuning needs labeled real futures, so it serves as our oracle, not our competitor. Where real futures are cheap, fine-tuning may simply win, and we say so. The method targets the regime where they are not.
Proposed Experiments
The first testbed is a controlled sim-to-sim setting in which ground truth is known. Following the validation practice of Humanoid-Gym, which trains in Isaac Gym and checks in MuJoCo, we train dynamics models in one simulator. The "real" domain is a second simulator with injected state-dependent latency, surface-dependent dropout and slow bias drift. Here we can check whether the learned corruption recovers the injected distribution. We can also vary how structured the gap is to find where Gaussian noise stops being enough. For isolated context corruption there is no reported result on Lacuna for any baseline.
The second testbed is cross-dataset driving generation: SMART's NuPlan→Waymo zero-shot setup, scored with the Waymo Sim Agents realism meta-metric against the 0.72 and 0.7614 anchors. Waymo is used only as unlabeled history windows during training. This is a cross-dataset shift, not sim-to-real, and we present it as such.
The third testbed is a quadruped depth-and-proprioception world model in the style of WMP, trained in Isaac Gym. We measure open-loop k-step prediction error on recorded real Unitree logs and closed-loop success on WMP's gap, crawl and tilting-platform obstacles. For world-model prediction error on real logs there is no reported result found on Lacuna, so all numbers here come from our own re-runs.
The decisive check is the comparison with hand-designed structured randomization on all three testbeds at matched corruption operations and tuning budget. The second decisive check is the gap-budget measurement. If the error remaining after context matching dominates real-log error, the input-side hypothesis is wrong for that domain, and the work reduces to a diagnostic that tells practitioners when a dynamics correction is unavoidable.
References
- Learning to Simulate Complex Physics with Graph Networks
- DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
- Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Sim-to-Real Transfer with Neural-Augmented Robot Simulation
- World Model-Based Perception for Visual Legged Locomotion
- SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction
- DrEureka: Language Model Guided Sim-To-Real Transfer
- Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer