Annealed Trajectory Projection for Physically Lawful Video Generation

Video generators often produce clips that look real but break simple physics: a dropped ball hangs in the air, speeds up sideways, or passes through the floor. Kang et al. traced this to how the models generalize (How Far Is Video Generation from World Model). They behave like fuzzy lookup tables, pulling each new scene toward the nearest training example rather than applying a law such as constant downward acceleration. Signal processing has a long history with a similar problem. To find a known kind of signal in noisy data, you compare the data against a bank of pre-simulated "templates," and the bank grows exponentially with the number of physical parameters. A recent method, TpopT (TpopT), avoids that growth. It treats the set of lawful signals as a smooth low-dimensional surface and walks along it with a few unrolled gradient steps. It starts from a blurred version of the surface and sharpens it gradually, which keeps noise from trapping the search in false optima.

We propose to use this idea inside a video diffusion model, a generator that turns random noise into a clip over many denoising steps. At each denoising step, a light probe reads the moving object's trajectory from the model's current guess of the final clip. A TpopT-style optimizer then projects that trajectory onto the nearest physically lawful trajectory. A guidance gradient nudges the sample toward the projection. The central design choice is to tie the blur of the lawful-trajectory surface to the diffusion noise level. Early, noisy steps see a smooth, forgiving surface. Late steps see the sharp physical one. The target is the lawful trajectory closest to the model's own sample rather than a single reference future, so the method should correct physics without collapsing the variety of plausible outcomes. The whole correction runs in a space of a few dozen numbers rather than pixels, so it adds a small cost per step.

Core Claim

For rigid-object motion governed by a known family of laws (free fall, projectile, bounce with restitution, sliding with friction), we claim that annealed projection onto the lawful-trajectory surface during denoising has three properties. First, it cuts trajectory error on real footage below what physics post-training achieves. Second, it holds up on out-of-distribution initial conditions where scaling does nothing. Third, it does both at a small fraction of the compute of generating many samples and keeping the most physical one.

The novelty is not "add physics to video diffusion"; several papers do that. What is new is treating physical correction as template matching and borrowing the fix for its cost wall, namely continuous search on a manifold with coarse-to-fine smoothing. The smoothing schedule is aligned with the diffusion noise schedule, and the method corrects toward the sample's own nearest lawful future instead of a fixed target.

Relation to Prior Work

Two papers motivate the proposal by showing the gap. Kang et al. find that growing data from 30K to 3M videos and parameters from 22M to 310M gives no improvement in out-of-distribution extrapolation error, which stays an order of magnitude above in-distribution error (How Far Is Video Generation from World Model). Motamed et al. show that visual realism and physical competence are unrelated: Sora reaches only 10.0% on Physics-IQ even though it is the hardest model to tell apart from real video (Do Generative Video Models Understand Physical Principles?).

Existing fixes each give something up.

  • PISA fine-tunes on synthetic free-fall video and optimizes rewards against ground-truth trajectories (PISA). Its authors report a loss of visual quality and a "likely sample" trap, in which the model collapses to one trajectory mode.
  • PhysGen hands motion to an external 2D rigid-body engine, with parameters guessed by a vision-language model, and then renders the pixels (PhysGen). It cannot correct a wrong parameter guess by gradient, and it takes about 3 minutes per clip.
  • The frequency-domain motion loss is cheap but enforces only translation, rotation and scaling, not gravity or collisions (Physics-Guided Motion Loss).

Unlike PISA, our method needs no retraining and has no single target. Unlike PhysGen, it keeps the generator in charge of motion and fixes parameters by optimization. Unlike the motion loss, it enforces nonlinear dynamics.

The method comes from two sources. TpopT supplies unrolled manifold search with smoothing (TpopT). TpopT was tested only on 1D gravitational-wave signals and transformed MNIST, and never inside a generator.

The pulsar work supplies the "coarse retrieval, then local simulator refinement" pattern (Accelerating Posterior Inference from Pulsar Light Curves). It reports a 120x speedup over MCMC while keeping multiple modes. We reuse its lesson that a coarse step should keep several candidate basins alive, and apply it to choosing among law families.

Morpheus constrains our evaluation (Evaluating Newtonian Mechanics in Video Generative Models). It fits tracked object paths to physics equations instead of comparing against one reference video, because futures given a single frame are not unique. We adopt the same philosophy for both the correction and the scoring.

Research Program

Trajectory probe. The first component reads trajectories from a noisy generation. We train a small probe that maps the generator's predicted clean latent at any noise level to object centroid and extent per frame. It is trained on Kubric-style synthetic renders where the true tracks are known, following PISA's data recipe. Running SAM-2 on decoded frames at every step would be far too slow; the probe keeps the loop in latent space.

Lawful-trajectory surface. For a law family $f$, let $s_f(\theta)$ be the simulated track for physical parameters $\theta$: initial position, velocity, gravity scale, restitution and friction, roughly 4 to 8 numbers. We use a differentiable rigid-body simulator, and $s_f(\theta)$ is the set of tracks it can produce.

Projection. At denoising step $t$, with predicted track $\hat{p}_t$, we solve

$$\theta_t^* = \arg\min_{\theta}\; \big\| \hat{p}_t - (G_{\sigma_t} * s_f)(\theta) \big\|^2,$$

where $G_{\sigma_t} * s_f$ is the lawful surface smoothed with a Gaussian of width $\sigma_t$. The width shrinks along with the diffusion noise level at step $t$. The minimization is done by $K$ unrolled steps with learned step sizes, as in TpopT, warm-started from the previous step's $\theta^*$.

Law-family selection. The law family itself is chosen by evaluating a handful of families (fall, bounce, slide, rest-on-support) and keeping the smallest residual. This is a tiny discrete template bank, and it follows the pulsar paper's retrieve-then-refine pattern.

Guidance. We add the gradient of

$$\lambda_t \big\| \mathrm{probe}(\hat{x}_0) - s_f(\theta_t^*) \big\|^2$$

with respect to the latent, where $\hat{x}_0$ is the predicted clean latent and $\lambda_t$ is a guidance weight. Because $\theta_t^*$ is fitted to the sample itself, two samples that start falling differently are pulled toward two different lawful futures. This is the mechanism meant to avoid the single-mode and freezing traps.

Cost accounting. We report cost in generator function evaluations and wall-clock time. The projection runs in trajectory space and costs $O(K)$ simulator calls per step. A grid bank at matching precision would cost exponentially in the number of parameters. That contrast is the template-overhead argument applied to physics correction.

The base generator is an open image-to-video model; we use Wan2.1 because Morpheus reports it as the strongest tested system.

Baselines

The naive reference is the unmodified base generator with its default sampler. It sets how much physics is already present and how much diversity and visual quality we must not lose.

PISA post-training. The strongest existing method for the free-fall question is PISA, which reports real-world PisaBench trajectory L2 error of 0.067, against 0.174 for Sora and 0.187 for Runway Gen3 (PISA). To make the comparison fair, we apply the PISA recipe to the same base generator with the same 5,000-video synthetic set our probe uses. Our method needs no generator fine-tuning, so we report PISA's training compute separately, not as a free resource. Beating PISA means lower trajectory L2 on the real split without a worse object-permanence IoU, which PISA reports at 0.14. It also means higher sample diversity, measured as the spread of fitted physical parameters across seeds from the same first frame.

Best-of-N with a physics verifier. This is the most dangerous competitor at test time. We generate N samples, fit each with the same Morpheus-style equation fit, and keep the best. We sweep N and match wall-clock time against our method. Beating it means sitting below its error-versus-compute curve.

Dense-bank projection ablation. This isolates the template-overhead ingredient. The projection target comes from exhaustive search over a grid of simulated tracks, and everything else is identical. We expect equal accuracy at much higher cost at fine precision, or worse accuracy at equal cost.

Unannealed optimizer ablation. This isolates the smoothing schedule. It uses multi-start Gauss-Newton through the same differentiable simulator on the sharp surface, with no smoothing. We expect it to lock onto wrong bounce timings in collision scenes early in denoising.

Single-target guidance ablation. Guidance steers toward one trajectory simulated from VLM-estimated parameters, as in PhysGen. This isolates the "sample's own nearest lawful future" choice.

PhysGen. We run PhysGen itself on the 2D rigid subsets it supports and compare Motion-FID, where it reports 30.20 (PhysGen), along with runtime.

Why not just…?

Why not just scale the model and data? Kang et al. report zero improvement in out-of-distribution extrapolation error from a 100x data increase and a roughly 14x parameter increase (How Far Is Video Generation from World Model). Morpheus finds the best commercial and open models top out at 0.66 compliance, against 0.96 to 0.99 for real footage (Morpheus). Scaling does not target the failure we care about. Our out-of-distribution velocity split tests exactly that failure.

Why not just post-train for physics, as PISA does? This is a real competitor on in-distribution free fall. Its own authors report blurring and mode collapse, and fine-tuning on a fixed set of drops has no obvious reason to fix extrapolation beyond the training range. The comparison on PisaBench, plus the out-of-distribution Box2D split and the diversity measure, settles whether test-time projection beats it, matches it more cheaply, or loses. If PISA matches us on all three, the method's value reduces to avoiding retraining, and we will say so.

Why not just sample many videos and keep the most physical one? For easy scenes with a small N this may work as well as our method. The expected advantage appears when lawful samples are rare, as in collisions or out-of-distribution velocities, because selection can only choose among what the generator produces. The matched-compute sweep shows where, if anywhere, the curves cross.

Why not just run plain gradient descent through a differentiable simulator, without the TpopT machinery? With so few parameters this is a serious threat. The unannealed ablation exists to test it. If it matches the annealed version, we drop the TpopT framing and report the simpler optimizer, with the finding reduced to "projection-based guidance works."

Why not just use an external engine, as PhysGen does? PhysGen is limited to planar rigid motion. It cannot fix a wrong VLM parameter guess by gradient, and it takes about 3 minutes per clip (PhysGen). We compare directly on its supported subset.

Proposed Experiments

PisaBench. The first experiment uses the 361 real-world 120-fps drop videos and the synthetic split from PisaBench. Metrics are trajectory L2, object-permanence IoU, a frame-quality score to measure any visual loss, and the fitted-parameter diversity described above. The target to beat is PISA's 0.067 L2.

Out-of-distribution extrapolation. We rebuild the Box2D uniform-motion, parabola and collision tasks from Kang et al. with initial velocities outside the training range. We report velocity error in and out of distribution. Kang et al. give only relative figures; no reported result found on Lacuna for an absolute out-of-distribution velocity error. The decisive check is whether our out-of-distribution error drops markedly toward in-distribution levels while the base model's does not. Because the lawful surface is defined for every $\theta$, a gap that stays an order of magnitude would falsify the central claim.

Morpheus. We score rigid-body phenomena from the Morpheus real-video suite, with Wan2.1's 0.66 as the reference and the freezing and discard rate as a second metric. Our method should reduce discards rather than trade violation for stillness.

Compute curves. We plot error against generator evaluations and seconds for our method, best-of-N and dense-bank projection, at several target precisions. The template-overhead claim predicts that dense-bank cost grows sharply as precision tightens while ours grows slowly.

Negative control. We evaluate on Physics-IQ's fluid and deformable scenarios, where no law family applies. The method should detect a high projection residual everywhere and switch itself off, leaving scores unchanged rather than degraded.