Robots and video models learn a lot from simulators because simulated data is cheap and comes with exact labels. Simulators get one thing systematically wrong that current transfer methods do not address. When a pushed block hits another object, a simulator usually produces one outcome, or a narrow band of outcomes if friction and mass are randomized. A real block, with surface micro-texture and imperfect contact, can slide, rotate or tip. A stochastic video predictor is a model that, given past frames, samples many different plausible future frames instead of one. If it is trained in simulation, it learns the simulator's range of outcomes, and that range is too narrow at the moments that matter. Our proposal is to keep a simulator-trained stochastic video predictor almost entirely frozen and use a small number of unlabeled real clips to correct only its sense of how spread out the future is, with the correction depending on the current state. The expected effect is that the predictor becomes wider at contacts and stays tight during free motion.
The case for this rests on a contrast already visible in the literature. STReSSD shows that real bouncing-ball collisions are intrinsically random. It fits a hand-designed von Mises–Fisher noise model on exit direction from microphone data, and the calibrated stochastic simulator improves real tracking success by 24% absolute over a deterministic one. That approach needs a hand-chosen noise family and low-dimensional state. Pixel-based sim-to-real world models such as TWIST and WMP work from images, but they treat simulator transitions as deterministic given the state and use randomization only for appearance and static parameters. No existing method learns, from pixels, how the real distribution of outcomes differs from the simulated one.
Core Claim
We claim that the sim-to-real error of a stochastic video predictor separates into three parts: an appearance error, a bias in the average dynamics, and an error in how spread out the outcomes are. We further claim that the spread error is concentrated in specific states, mainly contacts and branch points, rather than uniform over time.
If this is right, a small correction network attached to the predictor's learned prior can be fit on a few dozen to a few hundred real clips without labels. Here the learned prior means the per-step distribution from which the random variables driving each future frame are drawn. The correction should give better-calibrated real futures than three alternatives: a single global temperature on the prior, broad physics randomization in the simulator, and full fine-tuning on the same small real set.
If the spread error turns out to be uniform in state, a global temperature should match our method. That outcome would falsify the state-dependence part of the claim, and we treat it as a genuine possible result.
Research Program
The backbone is a learned-prior stochastic video model in the style of SVG-LP, which predicts each frame from past frames plus a random variable $z_t$. During training, an inference network sees the true future and proposes $z_t$. At test time, a recurrent prior $p_\psi(z_t \mid x_{<t})$ supplies $z_t$, and its variance already grows at branch points. We use this model rather than a diffusion model such as MCVD because the explicit prior gives a single place where outcome spread lives. A diffusion version is a later extension, not the first step.
The model is trained in a simulated tabletop pushing environment built in Isaac Sim, which ReBot shows can be composited against real backgrounds. The environment matches the BAIR robot-pushing camera setup approximately, and training uses appearance randomization in the manner of domain randomization so that the appearance gap is reduced before any real data is seen.
Adaptation freezes the frame decoder, the inference network and the prior. It then trains a small head $g_\phi$ that reads the predictor's hidden state $h_t$ and outputs a per-dimension shift $a_t$ and log-scale $s_t$. The corrected prior is
$$p_\phi(z_t \mid x_{<t}) = \mathcal{N}\big(\mu_\psi + a_t,\ \mathrm{diag}(\sigma_\psi^2 \, e^{2 s_t})\big),$$
where $\mu_\psi$ and $\sigma_\psi$ are the frozen simulator prior's mean and standard deviation. The head is fit on real clips by minimizing
$$\sum_t \mathrm{KL}\big(q(z_t \mid x_{\le t}) \,\|\, p_\phi(z_t \mid x_{<t})\big),$$
where $q$ is the frozen inference network applied to real frames. In words, the corrected prior learns to propose the random variables that real outcomes actually require.
The shift term plays the role that Found-adapt gives its adapted latent: Found-adapt freezes a simulator-pretrained policy and adapts only a low-dimensional conditioning vector to capture average dynamics differences. Our scale term is the addition that Found-adapt lacks, because Found-adapt assumes stationary target dynamics rather than random ones.
The main risk is that the frozen inference network, applied to real frames, encodes appearance errors into $z$, so that the head "widens" the prior to explain rendering mismatch rather than physical randomness. This is the confusion between physical randomness and sim-to-real mismatch that HAUWM's ensemble uncertainty also leaves untested (HAUWM). We address it two ways. First, we include an optional unpaired translation of real frames toward the randomized simulator style, following PixelDA. Second, and more importantly, we use a controlled sim-to-sim stage in which appearance gap and stochasticity gap can be switched on independently.
Baselines
The naive reference point is the simulator-trained predictor applied to real clips unchanged. It tells us how large the spread error is to begin with.
The simplest real competitor is a global temperature: one scalar multiplying $\sigma_\psi$, tuned on the same real clips. It isolates whether the state dependence of our head matters. Beating it means a lower calibration error specifically on contact segments, with no loss on free-motion segments, across at least three seeds.
The physics-side competitor retrains the predictor in a simulator with broad randomization of friction, restitution and contact noise. As a stronger version of the same idea, we inject a STReSSD-style von Mises–Fisher perturbation on post-contact object velocity, with its concentration tuned by grid search against the same real clips. This gets the identical real-data budget and the same training compute as the base model plus adaptation.
Full fine-tuning updates all weights on the same real clips with early stopping on a held-out real split and the same tuning budget as our head.
Two ablations isolate the key ingredient: a shift-only head that corrects average dynamics but not spread, and a scale-only head.
As an upper reference we report a predictor trained on the full real BAIR set. The strongest reported long-horizon number in the file is from Causal State Compression, which reports a 64-frame FVD of 96.4 on BAIR versus 218.7 for an autoregressive baseline. FVD, or Fréchet Video Distance, measures how far the feature distribution of generated videos is from that of real videos (FVD). We do not expect a few-shot-adapted simulator model to beat 96.4. Instead, the question is what fraction of the gap between the naive model and full real training it closes with one to two orders of magnitude less real data. Image-quality reference points are SAVP's roughly 35% fooling rate in human real-versus-fake tests (SAVP). No reported result was found on Lacuna for any sim-to-real stochastic video predictor on BAIR or a comparable pushing set.
Why not just…?
Why not just apply a global temperature to the prior? This is the most serious threat, because it costs one parameter. Our argument is that simulators are roughly right during free motion and wrong mainly at contacts, so any single scale must either over-widen free motion or under-widen contacts. That argument is a hypothesis, not a known fact. The segment-wise calibration comparison below settles it directly. If temperature ties our head on contact segments, we drop the state-dependence claim and report the simpler fix.
Why not just randomize physics more broadly in simulation? Randomization widens outcomes everywhere a parameter acts, and it is a guess about which parameters matter. When Simulation Lies reports that randomizing static inputs leaves roughly a 30% gap from runtime transition failures; that evidence comes from tool-use agents, not robots, so it is suggestive only. STReSSD shows that a well-chosen injected noise does work, which is why its tuned von Mises–Fisher variant is a baseline rather than a straw man. If the tuned injected noise matches our method, the honest conclusion is that hand-designed noise suffices when its form is known. Our remaining advantage would then be that the learned head works without knowing that form, which we test by giving the real environment a contact behavior, such as tipping, that the injected noise family does not include.
Why not just fine-tune the whole predictor on real video? With plentiful real video this is the right answer; RoboNet shows pretrained visual dynamics need far less real data than training from scratch (RoboNet). The question is the low-data regime. A few dozen clips contain few contact events, and full fine-tuning should overfit the observed outcomes and shrink spread. We sweep the real-data budget from 25 to 2,000 clips and report where full fine-tuning overtakes the head. That crossover is the result that defines when the method is worth using.
Proposed Experiments
Stage one: sim-to-sim. The target domain is a second simulator configuration with a different renderer and STReSSD-style injected contact noise, so the true outcome distribution is known exactly from repeated rollouts from identical starts. We cross two factors, appearance gap on or off and stochasticity gap on or off. The decisive checks are two:
- With the appearance gap on but the stochasticity gap off, the learned scale should stay near one. If it inflates, appearance errors are leaking into $z$, and translation must be added before going further.
- With the stochasticity gap on, the head's widening should concentrate in frames near contact.
Calibration is measured on tracked object displacement in predicted versus true outcomes, using coverage of central prediction intervals and an energy score. The energy score is a single number that rewards both getting the average right and getting the spread right. Results are reported separately for contact and free-motion segments.
Stage two: real BAIR. The target is real BAIR robot-pushing video with small real budgets. BAIR has no repeated starts, so calibration is measured at population level: the distribution of tracked displacement across held-out clips, compared within contact and non-contact segments. We also report FVD with at least 256 samples, following the FVD paper's sample-size caution, and best-of-100 accuracy.
Stage three: downstream planning. This stage tests whether calibration matters for a robot. A visual-foresight planner of the kind RoboNet uses on a physical Franka pushing task scores candidate action sequences with each predictor. We measure success and, following STReSSD, end-effector kinetic energy as a sign of over-reaction.
The work counts as a success if, at budgets of 200 real clips or fewer, the state-dependent head beats the global temperature on contact-segment calibration and beats full fine-tuning on held-out calibration, while matching the tuned injected-noise simulator without being told the noise form.
References
- STReSSD: Sim-To-Real from Sound for Stochastic Dynamics
- TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer
- World Model-Based Perception for Visual Legged Locomotion
- Stochastic Video Generation with a Learned Prior
- MCVD - Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
- ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesis
- Training Deep Networks With Synthetic Data: Bridging the Reality Gap by Domain Randomization
- Latent Adaptation of Foundation Policies for Sim-to-Real Transfer
- Learning to Be Uncertain: Pre-training World Models with Horizon-Calibrated Uncertainty
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Causal State Compression for Long-Horizon Video World Models: A Bounded-Drift Theory and Efficient Architecture
- FVD: A new Metric for Video Generation
- Stochastic Adversarial Video Prediction
- When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents
- RoboNet: Large-Scale Multi-Robot Learning