Fit-the-Law Guidance: Steering Video Generators with Optical-Flow Physics Residuals

Video generators now render convincing textures, but the things in their videos often fail to fall, bounce or hold their shape the way real objects do. Two families of fixes exist, and each has a cost. The first retrains the generator on physically correct clips. That lowers trajectory error, but it blurs the images and breaks down on scenarios outside the training range. The second checks motion against a single "correct" reference trajectory. That punishes videos that are physically valid but differ from the reference, because the true mass, friction or drop height is unknown.

We propose a third route that needs neither retraining nor a reference. While the generator is sampling, we measure the motion in its draft video with optical flow, which is a per-pixel estimate of how image content moves between frames. We then ask whether any setting of a known physical law, such as projected constant-acceleration flight or rigid-body motion, can explain that motion. The leftover error after the best fit is the physics violation. Its gradient adjusts a few learnable prompt tokens, not the generator's weights. Because the law's parameters are fitted rather than fixed, the method accepts every physically valid future. Because the backbone is untouched, it should avoid the image-quality loss that retraining causes.

Core Claim

The claim is that a parameter-free law residual, computed on flow-tracked points of the model's own draft video, is a better steering signal than either a learned "natural motion" discriminator or a fixed-reference flow reward. When used only at inference time, it should close most of the gap between a base open generator and a physics-fine-tuned one on falling-object benchmarks. It should do so without the visual degradation of fine-tuning, and without collapsing when the scenario leaves the fine-tuning distribution.

Concretely, let $p_t^i$ be the image position at frame $t$ of point $i$, tracked by optical flow inside an object region. Let $\mathcal{F}$ be a law family with parameters $\theta$ constrained to an admissible set $\Theta$. For projected freefall, $f_\theta(t)$ is a ratio of quadratics in $t$, which is what a 3D parabola looks like after perspective projection. The admissible set requires downward acceleration in the scene's vertical direction and nonzero magnitude. The residual is

$$ r(\text{video}) = \frac{\min_{\theta \in \Theta} \sum_{i,t} \lVert p_t^i - f_\theta(t) \rVert^2}{\sum_{i,t} \lVert p_{t+1}^i - p_t^i \rVert^2 + \epsilon}, $$

where the denominator is the total tracked motion and $\epsilon$ is a small constant. It normalizes the residual so that a frozen video cannot score well, which guards against the "stillness trap." For the rigid-body family, $f_\theta$ is instead a per-frame planar homography shared by all points on the object, so an object that morphs or melts leaves residual that no single rigid motion can absorb. Object disappearance is caught by transporting the first-frame object mask forward with the flow and penalizing mass that leaks away without an occluder.

Relation to Prior Work

Six papers already join flow and physics, and the proposal sits in the gap they leave.

Two of them are evaluators. PhyCoBench predicts a reference future flow with a diffusion model and scores a video by its flow error against that reference, reaching a Kendall's Tau of 0.3367 with human rankings. MotionPhys computes curvature, angular velocity and jerk statistics on Lucas-Kanade tracks and feeds them to a classifier to spot generated videos. Neither steers generation. PhyCoBench's reference is only as physical as its own generator. MotionPhys's descriptors are law-agnostic statistics, so a smooth impossible motion passes. STREAM shows that such smooth impossible motions occur, which motivates our choice of explicit laws over smoothness statistics.

Three lines of work steer generators with flow, and they supply our machinery while defining the contrast.

  • MotionPrompt optimizes soft prompt tokens during sampling, using a discriminator trained on real-versus-generated flow. We reuse its route of sending a flow-based loss back into prompt embeddings. We replace its discriminator, which rewards generic smoothness, with a law residual that rewards specific dynamics.
  • PISA Experiments fine-tunes Open-Sora v1.2 with simulated drops and an optical flow reward against analytic ground-truth trajectories. It reports three failures our design targets: a visual quality tax, collapse on drop heights outside training, and the "likely sample trap" of fixed references.
  • MotionCraft warps diffusion latents along flow from an external simulator. We need no simulator, only a law family whose parameters are fitted from the model's own motion.

FOFPred predicts future flow at 7B scale but enforces no physical invariants.

Other work constrains the design. Morpheus already fits physical ODEs to SAM-2 tracks for evaluation and shows that vision-language judges are unreliable. This is the closest evaluative precedent. Our difference is that the residual lives on dense flow, which gives smooth gradients with respect to pixels, and that it is used to guide sampling rather than only to score.

How Far Is Video Generation from World Model shows that scaling from 22M to 310M parameters and from 30K to 3M videos does not reduce out-of-distribution physical error, because models retrieve nearby training cases. This is why we impose the law at inference rather than hoping it is learned.

Research Program

The program has three parts: a guidance method, a law library and a residual-as-metric study.

The guidance method. At selected reverse diffusion steps, we decode the current clean-video estimate and run a differentiable learned flow estimator. We track points seeded in the conditioning frame's object mask, obtained once with a segmenter on the input image, and solve the inner least-squares fit for $\theta$ in closed form or with a few Gauss-Newton steps. The fitted $\theta$ is treated as fixed, and the residual gradient flows to the prompt tokens, as in MotionPrompt. An auxiliary term keeps total flow energy near that of the unguided draft, so guidance changes the shape of motion, not its amount.

The law library. We deliberately keep it small, to three families that 2D flow can honestly test: projected ballistic flight, rigid motion and mask conservation. Impenetrability is out of scope. As WorldModelBench implicitly highlights, telling "through the wall" from "behind the wall" needs depth. Law selection comes from the benchmark category or the prompt. We do not claim automatic law discovery.

The residual as a metric. We score existing generated videos with $r$ and correlate the scores with human judgments. This tells us whether the steering signal is itself trustworthy before we optimize against it.

Baselines

The naive reference point is unguided sampling from the same backbone. We use Open-Sora v1.2 so that PISA's fine-tuned model is directly comparable, with identical conditioning frames, seeds and sampler steps. Any method must beat this to matter. It also gives the image-quality ceiling that guidance must not fall below.

The strongest existing method on falling objects is PISA's physics-fine-tuned Open-Sora (PISA Experiments). On PisaBench's real-world drops, it reports a trajectory L2 error of 0.067, against 0.174 for Sora and 0.187 for Runway Gen3. It also reports object-permanence IoU of 0.14, against roughly 0.06 for those baselines. We will run the released PISA checkpoint if available, and otherwise retrain it with the paper's 5,000-sample recipe on the same data. Beating PISA means one of two outcomes:

  • matching its in-distribution L2 within noise while scoring higher on image-quality metrics and human preference, or
  • beating it on drop heights outside its training range.

Matching 0.067 outright with no training would be a strong result. Landing well below Sora's 0.174 while preserving quality would still be a meaningful partial win.

The strongest steering baseline is MotionPrompt, run on the same backbone with the same number of prompt tokens and optimization steps per sampling phase. MotionPrompt's reported overhead is roughly 10 to 15 steps. We also report MotionPrompt on Lavie, where it improved VBench temporal flickering from 0.9487 to 0.9625, to confirm our reimplementation reproduces it.

Three ablations isolate the key ingredient, fitting the law's parameters rather than fixing them:

  • Fixed-reference residual. Replace the minimization over $\Theta$ with PISA-style ground-truth analytic trajectories.
  • Learned discriminator. Replace the analytic residual with a MotionPrompt discriminator retrained on Kubric physics flows.
  • Segmentation tracks. Replace dense flow tracks with SAM-2 centroid tracks. SAM-2 masks give no smooth pixel gradient, so this variant can only be used as a best-of-N selector.

Why not just…?

Why not just pick the best of N samples by the same residual? This is the most serious threat, and it is cheap. If the base model sometimes produces physical drops, filtering may match guidance. We will compare at matched wall-clock compute, choosing N to equal the guidance cost. We expect guidance to win where violations are near-universal. Evidence for that regime comes from Physics-IQ, where the best model reaches only 29.5%, and from Morpheus, where discard rates reach 50% under single-frame conditioning. We also expect filtering to be competitive on easy prompts. If best-of-N matches guidance everywhere, we will report the residual as a selector and metric, and drop the guidance claim.

Why not just fine-tune with a flow reward, as PISA does? PISA already shows that this lowers trajectory error. It also documents the visual quality tax, failure outside the training heights and the likely-sample trap. Our out-of-range drop experiment is designed to show whether an inference-time, fitted-parameter law avoids the latter two. Side-by-side quality metrics will show whether it avoids the first.

Why not just use MotionPrompt's natural-motion discriminator, or a VLM judge? STREAM shows that smooth flow can depict impossible events, so a smoothness discriminator is the wrong target. The retrained-discriminator ablation tests whether a physics-trained discriminator closes the gap. VLM judges are ruled out as steering signals by the evidence in Morpheus, Physics-IQ and Impossible Videos. In Physics-IQ, Gemini 1.5 Pro was fooled by Sora's videos 44.4% of the time despite Sora's 10.0% physics score. In Impossible Videos, Intern-VL-2.5 answered "AI-generated" 93.5% of the time regardless of content.

Why not just track objects with SAM-2 and fit ODEs, as Morpheus does? For rigid single objects, SAM-2 tracks may measure violations as well as flow does, and we will report that openly. The case for flow rests on two things. It gives a differentiable signal for guidance, and it covers deforming objects where masks lose identity. The SAM-2 ablation tests both.

Proposed Experiments

Freefall on PisaBench. The primary test uses the 361 real 120-fps drop videos and the Kubric split. We report trajectory L2 and permanence IoU against the numbers above. Image quality is measured with VBench subject consistency, flicker and imaging quality, plus a blinded human A/B test against both the unguided model and PISA.

The decisive out-of-range check. We render new Kubric drops at heights and depths outside PISA's training range and compare the error growth of PISA against ours. Following Kang et al.'s case-retrieval account, we predict that PISA's error rises while ours stays roughly flat, because nothing in our method was fit to the training range.

Broader coverage. On PhyCoBench, we apply the ballistic and rigid families to its Gravity, Projectile Motion, Collision and Rotation prompts. We also test whether $r$ alone correlates with human rankings above PhyCoPredictor's Kendall's Tau of 0.3367. On Morpheus, we report aggregate score, for which the file cites the best generator as unable to exceed 0.66, only on its falling, projectile and sliding subsets. We note explicitly that collisions and fluids lie outside what our 2D law library can check.

Stillness-trap audit. We verify that guided videos keep total flow energy within a set band of unguided ones. A method that lowers $r$ by freezing motion would be counted as a failure.