Video generators produce clips that look real but move wrong. Objects vanish mid-flight, bounces gain energy, and falling objects accelerate at a fraction of real gravity. Current fixes mostly retrain the generator on simulator output where the true trajectory is known. That dependence on paired ground truth is the same problem computational imaging faced when fully-sampled MRI scans were unavailable. Imaging solved it with measurement splitting: hide part of the measurement, reconstruct from the rest, and score only on the hidden part. This proposal carries that trick into video physics. We track objects in a generated clip, fit a small physical model (ballistic flight, a pendulum, bounces with a fixed restitution coefficient) to some frames, and ask how well that fitted model predicts frames it never saw. A clip that obeys some consistent Newtonian dynamics predicts its own held-out frames. A clip that merely looks smooth does not.
The result is a physics score that needs no simulator, no ground-truth trajectory, and no vision-language judge. It can therefore serve as a training reward on generated rollouts from text prompts or single frames, where no real continuation exists. We use it both to post-train an open video model and as an evaluation metric. The claim to test is that this reference-free score can match simulator-trained post-training on real-world physics benchmarks without paying the visual-quality cost that simulator training incurs.
Core Claim
The naive transplant of measurement splitting to video is "hold out frames and predict them." That is simply masked video modeling or frame interpolation. Ordinary video diffusion training already does it and still learns bad physics, because a pixel loss is dominated by appearance (PISA Experiments). What is new here is where the split is applied. We apply it not to pixels but to a low-dimensional physical explanation of the clip, and we contrast that with a generic smooth explanation.
Let $p_t$ be the tracked 2D positions of an object's points at frame $t$. Let $M_\phi(t)$ be the position predicted by a physical model family with a few parameters $\phi$: initial 3D state, depth, gravity or restitution, and contact times. Let $S_\psi(t)$ be a generic smooth predictor, such as a cubic spline, with parameters $\psi$. Both are fit only on frames $\Theta$ and scored on disjoint held-out frames $\Lambda$:
$$R_{M} = \frac{1}{|\Lambda|}\sum_{t\in\Lambda}\big\|p_t - M_{\hat\phi(\Theta)}(t)\big\|^2, \qquad \text{score} = \frac{R_{M}}{R_{S} + \sigma^2}.$$
Here $R_S$ is the analogous held-out error of the smooth predictor, and $\sigma^2$ is the tracker noise floor, estimated from real videos.
The ratio does three things. A physically consistent clip has $R_M \approx R_S$. A clip with smooth but non-Newtonian motion, such as mid-air deceleration or energy-gaining bounces, has $R_M \gg R_S$. Tracker jitter inflates both terms and largely cancels out.
The split is essential for the same reason it is essential in MRI (Self-Supervised Physics-Based Deep Learning MRI Reconstruction Without Fully-Sampled Data). An in-sample fit lets a flexible model family absorb violations. For example, extra contact events can be placed to explain a fake bounce. Scoring held-out frames removes that escape.
We follow Zero-Shot Self-Supervised Learning for MRI Reconstruction, which uses a third partition $\Gamma$ to decide when to stop fitting. We likewise hold out a third frame set that selects model complexity, meaning the number of contact events and which model family to use. This keeps the score honest without human tuning.
Research Program
Model families. The first component is a library of projected physical model families, each with a handful of parameters. The first is 3D ballistic flight under a pinhole camera with unknown depth, since constant 3D acceleration is not constant acceleration in pixels once depth changes. The second is sliding or rolling with constant friction deceleration. The third is a planar pendulum. The fourth is piecewise ballistic flight with bounces sharing one restitution coefficient, so fitting two bounces predicts the third, which makes energy consistency testable. Object permanence enters as track survival: a track that dies without leaving the frame or being occluded by another tracked object counts as a held-out failure.
Split design. Split design matters because frames within a clip are not independent. Following the finding that split ratios matter (Self-Supervised Physics-Based Deep Learning MRI Reconstruction Without Fully-Sampled Data found roughly 40% held out optimal), we compare interleaved splits, which test local dynamics, against extrapolation splits, which hold out the final segment and test whether early motion determines later motion.
The stillness problem. The score has an obvious exploit: a frozen video is perfectly predictable. The literature documents that generators already retreat to stillness on hard contact events (Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems). We therefore condition generation on two or three real initial frames, which fix a nonzero initial velocity. The reward is gated on displacement being consistent with that velocity, so stillness scores as a violation rather than a success.
Calibrating the score. Before any training, we validate the score as a metric. We run it on the real videos in PisaBench, Morpheus, and GAUGE-style recordings, and on generated counterparts. It must rank real above generated in the cases where vision-language judges are documented to invert that ranking (Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems, $PhyWorldBench$).
Post-training. We then post-train an open generator. We use Open-Sora v1.2 to match PISA, and CogVideoX-5B to match VideoREPA. Prompts and conditioning frames come from unannotated OpenVID clips filtered by the tracker for moving objects. Optimization uses reward-weighted fine-tuning on the model's own rollouts, so neither the tracker nor the fitter needs to be differentiable. No simulator data and no ground-truth trajectories enter at any point.
The scale question. GAUGE reports that models get the form of free fall right but the scale badly wrong, estimating gravity at 1.84 m/s² (GAUGE). A reward with gravity as a free parameter cannot fix this. We therefore also test a "pinned" variant that fixes gravity at 9.81 m/s² and converts it to pixels using an object-size or monocular-depth anchor.
We pair this with a diagnostic on the training corpus itself. We fit free gravity to falling objects in OpenVID clips to test whether slow-motion and frame-rate-resampled footage produce a broad, low-biased distribution of apparent gravity. If they do, the scale failure is a temporal ambiguity in the data, and pinning gravity should repair it. If apparent gravity in the data is correct, the failure lies in the model, and pinning should still help but for a different reason.
Baselines
Do nothing. The naive reference is the unmodified base generator, run on identical prompts and conditioning frames with the same sampling settings. All gains are reported against it.
Simulator-trained post-training. The strongest existing method on real-world physics is PISA's two-stage pipeline (PISA Experiments). It fine-tunes on 5,000 Kubric simulator drops, then applies object reward optimization with segmentation, optical-flow, and depth rewards computed against simulator ground truth. On PisaBench real drops it reports trajectory L2 error of 0.067 against 0.174 for Sora, and object-permanence IoU of 0.14 against roughly 0.06 for baselines. It is the strongest competitor because it directly answers the question we pose, but uses paired simulator data.
We run PISA from the same Open-Sora v1.2 checkpoint, with the same number of gradient steps and the same hyperparameter-search budget as our method. Beating it means matching its trajectory L2 error within seed noise on PisaBench real drops while scoring better on visual quality. It also means outperforming it on the out-of-domain categories it never trained on: bounces, pendulums, and sliding in Morpheus and GAUGE.
Simulator-free methods. Two baselines represent the simulator-free alternatives. The first is VideoREPA (VideoREPA). It distills token-similarity structure from a frozen VideoMAEv2 model into CogVideoX and reports a physical-commonsense score of 40.1 on VideoPhy for the 5B model. We run it from the same CogVideoX-5B checkpoint on the same OpenVID subset, and beating it means a higher VideoPhy score at equal training compute.
The second is the spectral SIM(2) motion loss (Physics-Guided Motion Loss for Video Generation Model). SIM(2) is the group of planar translation, rotation and uniform scaling, and the method reports about 11% better motion accuracy and 22–37% lower warping error. We apply it to the same backbone and data. We expect it to match us on planar rigid motion and lose on gravity and bounces, and we treat a loss on those categories as the relevant test.
Ablations of the key ingredient. Three ablations isolate what matters. The first scores in-sample residuals with no split, which is essentially an ordinary-differential-equation-fit score used as a reward, in the spirit of Morpheus's physics-informed dynamical score. The second drops the physics family and rewards only held-out predictability under the smooth predictor $S$, which tests whether we are just rewarding smoothness. The third ablation removes gravity pinning, to separate the form and scale effects. The proposal's central mechanism is supported only if the full score beats all three on PisaBench and Morpheus.
Why not just…?
Why not just post-train on simulator data, as PISA does? Simulator post-training works on the domain it covers, but it has two documented costs. PISA reports blurrier textures and ghosting after Kubric fine-tuning. GAUGE finds that even the best physics engine's bouncing-ball error is 15.63 times the variance of real experiments, so training on engines bakes in their contact errors. The head-to-head above settles whether a reference-free reward can match PISA in its home domain while avoiding both costs. If PISA wins decisively on real drops, simulator data carries information our score cannot recover, and we will report that.
Why not just supervise on real videos, using tracked real continuations as pseudo-ground-truth trajectories? This is the real threat. Real footage is its own ground truth for the future, and a PISA-style flow reward against real tracks needs no simulator. Our method differs in one respect: it is reference-free. It can score rollouts from text prompts, single frames, or counterfactual setups where no real continuation exists. We include this "real-track reward" as an additional baseline on identical clips. If it matches us on PisaBench and Morpheus, the training contribution reduces to cheaper data coverage, and the lasting contribution is the metric. The deciding experiment is performance on prompts with no matching real footage, measured by PhyWorldBench success rate, where the best proprietary model reaches about 26% ($PhyWorldBench$).
Why not just use a vision-language model as the physics reward? Two benchmarks show these judges favor cinematic but physically wrong clips and flag real footage as violating physics (Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems, $PhyWorldBench$). Optimizing against such a judge would reward aesthetics. Our metric-validation step measures this directly by comparing each score's agreement with real-versus-generated labels.
Why not just score how well an ODE fits the whole clip, with no split? An in-sample fit lets a flexible model family explain violations away, and a generator optimized against it would learn to produce exactly those explainable-but-wrong motions. The no-split ablation tests this. If it performs as well as the split version, the imaging analogy is unnecessary and we will say so.
Decisive Checks
The first check is metric validity, with no training. On Morpheus, real videos score 0.96–0.99 on the benchmark's own aggregate and the best generator scores below 0.66. Our held-out score must separate these two groups at least as cleanly as Morpheus's physics-informed dynamical score does. Ranking agreement with Morpheus scores is reported per phenomenon.
The second check is PisaBench real drops, measured by trajectory L2 error and IoU against PISA's 0.067 and 0.14. Visual quality is measured alongside, using Fréchet video distance and a blinded human texture-sharpness rating, to test whether the quality cost disappears.
The third check is GAUGE-style physical constants. We measure free-fall gravity against the reported 1.84 m/s² for Seedance 2.0, and pendulum period against the real 1.06 s, where models were roughly double. We compare pinned and free-gravity rewards, and relate the outcome to the corpus diagnostic on apparent gravity in training clips.
The fourth check covers generalization and side effects. We measure VideoPhy physical commonsense against VideoREPA's 40.1, and PhyWorldBench success rate on categories the model families do not cover, such as fluids and deformables. That category must at least not regress. A regression there would show the reward narrows the model rather than teaching it physics.
References
- PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop
- Self-Supervised Physics-Based Deep Learning MRI Reconstruction Without Fully-Sampled Data
- Zero-Shot Self-Supervised Learning for MRI Reconstruction
- Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems
- $PhyWorldBench$: A Comprehensive Evaluation of Physical Realism in Text-to-Video Models
- GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models
- VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models
- Physics-Guided Motion Loss for Video Generation Model