Most methods that rebuild a moving 3D scene from video can replay what they saw but cannot say what happens next. They learn a deformation field, a function that moves each scene element over time, and this field drifts or warps badly once it is asked about any moment after the last observed frame. There are two existing fixes, and neither is good enough. The first runs a physics simulator, which is accurate but slow and needs material properties that a short clip rarely reveals. The second uses physics-guided diffusion sampling, meaning a generator that turns random noise into a sample through many small denoising steps and is pushed toward physically valid outputs at every step. That approach can represent uncertainty, but it pays a physics gradient at each of dozens of sequential steps. We propose a generator that outputs the entire future motion of a scene's moving parts in a single network pass. The physical laws are imposed during training, not during sampling, so inference costs one forward pass whatever the prediction horizon.
The approach rests on two observations. First, a short observation often cannot pin down the future. A soft ball and a stiff ball can look the same for ten frames and then bounce very differently. A deterministic predictor in that situation averages the possible futures, and the average of two valid trajectories is generally not itself a valid trajectory. Second, the physical check on a whole trajectory can be run in parallel across time. If the generator outputs all future frames at once, the per-step physics residual (how badly frame $k{+}1$ violates the dynamics given frames $k$ and $k{-}1$) can be computed for every $k$ simultaneously. No sequential rollout is needed, either when training or when sampling.
Core Claim
A one-step generative model of future per-Gaussian trajectories, trained with an average-velocity objective plus a time-parallel physics residual, should produce multiple physically consistent futures for non-rigid 4D Gaussian scenes. It should do so at lower latency than an ODE-integrated neural force field and much lower latency than multi-step physics-guided diffusion. When the future is ambiguous, it should also beat deterministic predictors on both coverage of the true outcome and physical validity. If the future is not ambiguous, a deterministic predictor should do as well, and our experiments are built to show which case holds.
Relation to Prior Work
The method comes mostly from two papers. Mean Flows for One-step Generative Modeling trains a network to predict the average velocity over a whole generative interval, so one integration step is exact, without a teacher. 3D MeanFlow shows that this objective works on point sets. We reuse both, but the object we generate is a trajectory: future positions and rotations of dynamic Gaussians over a horizon. A static shape is not enough for our purpose.
The closest prior idea is Phys-Instruct. It distills a multi-step diffusion PDE solver into a 1-to-4-step student, using a distribution-matching loss together with a PDE residual computed on each generated sample. It reports that its 1-step model often has lower PDE error than 200-step baselines. That result is our main evidence that putting the physics into training can survive compression to one step. Phys-Instruct, however, works on fixed 2D grids such as Navier–Stokes 64×64 and has no notion of scene, observation, or occlusion. We move the residual to Lagrangian Gaussian particles conditioned on a reconstructed 4D scene, and we train from data without a teacher.
Physics Informed Distillation also uses the word physics, but its "physics" is the generative probability-flow ODE itself, applied to CIFAR-10 and ImageNet. It shows that PINN-style training can produce one-step generators. It constrains nothing about real-world dynamics.
Three lines of scene-dynamics work motivate the gap. OmniPhysGS couples Gaussians to a Material Point Method (MPM) continuum simulator and fits materials with score distillation from a video model, which requires slow per-scene optimization. NGFF replaces MPM with a learned force field integrated by an ODE solver, reaching 0.36 s versus 39.29 s per sequence on GSCollision. It is deterministic, though, and its cost still grows with the horizon. R5DGS extrapolates cheaply by moving rigid object centroids, and it concedes that photometric fidelity drops under non-rigid motion.
On the static-scene side, 4D Gaussian Splatting and D-NeRF supply the canonical-plus-deformation representation we start from. Split4D and the "as static as possible" prior of Flux4D supply the static/dynamic split. We keep the static assumption wherever it holds and generate motion only for the Gaussians it fails on. That choice reflects a constraint: RobustSplat and MS-GS document how motion misattributed to static geometry spawns persistent floaters.
One caution shapes the evaluation. Monocular Dynamic View Synthesis: A Reality Check shows that good rendered images can hide broken motion. For that reason we score positions and physical residuals directly, not only image quality.
Research Program
Step one builds the observed scene. We reconstruct the observed window with a 4D Gaussian Splatting model, split static from dynamic Gaussians with a Split4D-style decomposition, and subsample the dynamic Gaussians into a few hundred control nodes. Dense Gaussians follow the nodes through linear blend skinning, a standard way to move many points by interpolating the motion of a few.
Step two is the generator. A point transformer takes as input the node positions and velocities in the last observed frames, the static geometry treated as collision boundaries, and noise. It outputs the node trajectories for $H$ future frames together with a latent material code $\hat\theta$. It is trained with the Mean Flows average-velocity loss $\mathcal{L}_{\text{MF}}$ on simulated futures, plus a residual term:
$$ \mathcal{L} = \mathcal{L}_{\text{MF}} + \lambda \sum_{k=1}^{H-1} \big\| \hat{x}_{k+1} - 2\hat{x}_k + \hat{x}_{k-1} - \Delta t^2\, a(\hat{x}_k;\hat\theta) \big\|^2 + \mu\, P_{\text{contact}}(\hat{x}). $$
Here $\hat{x}_k$ are the generated node positions at future frame $k$ and $\Delta t$ is the frame interval. The acceleration $a(\cdot;\hat\theta)$ comes from a differentiable as-rigid-as-possible elastic energy on the node graph with stiffness and damping set by $\hat\theta$, plus gravity. $P_{\text{contact}}$ penalizes penetration of the static geometry. The residual is a discrete momentum balance that is evaluated at every $k$ in parallel, which is why it adds no sequential cost.
Because $\hat\theta$ is sampled along with the trajectory, different noise draws can correspond to different materials. Each sample is physically consistent on its own terms, rather than all samples being averaged into one compromise. The deliberate scope is rigid, soft, and contact-dominated motion of the kind GSCollision covers. Fluids would need a different residual, and we leave them out.
Step three is data. We train on GSCollision-style MPM simulations, randomizing materials and initial impulses so that short observation windows are deliberately ambiguous. We then measure transfer to D-NeRF synthetic sequences and to the R5DGS indoor scenes.
Baselines
The naive reference is the 4D Gaussian Splatting deformation field queried past its last training timestamp, together with constant-velocity extrapolation of the dynamic Gaussians. This shows how bad the drift problem is when nothing is done about it. Both use the same reconstruction our method starts from.
The rigid baseline is R5DGS. It is the strongest cheap physics-informed extrapolator in the file, and its reported number is an 11 FPS extrapolation speedup over per-Gaussian prediction. For the strongest deterministic competitor, NGFF, the file reports 0.36 s per sequence against 39.29 s for MPM on GSCollision, and the lowest RMSE and Final Position Error (FPE) among its comparisons. No reported numeric RMSE/FPE value for NGFF was found on Lacuna, so we rerun it ourselves on identical splits. NGFF is the competitor to beat because it is already fast and physically grounded; our method has to justify generativity against it.
We also train a deterministic regressor with our exact architecture, data, and physics residual, differing only in having no noise input. It gets the same GPU-hour budget and the same hyperparameter search size as our method. This isolates the value of being generative.
The quality-ceiling baseline is a 50-step diffusion model over the same trajectories with residual-gradient guidance at each step, in the style of Saydemir et al.. We also run it at 4 and 8 steps. As an accuracy oracle that ignores latency, we include MPM with materials fitted per scene, which approximates the OmniPhysGS regime.
Two ablations isolate the key ingredient: a Mean Flows model trained without the residual, and the same model with the residual applied only as guidance at sampling time.
Beating the field means three things together. First, best-of-K FPE should match or beat NGFF and the deterministic twin on ambiguous scenes. Second, the mean physics residual should be within 10% of the 50-step guided sampler. Third, wall-clock latency should be under NGFF's and roughly flat as the horizon $H$ grows.
Why not just…?
Why not just use a deterministic force-field predictor like NGFF? If the observed window identifies the materials, a deterministic model should be as good, and the generative machinery would be wasted. That is a real threat. We settle it by sweeping the length of the observation window. As the window shrinks and the material becomes ambiguous, we expect the deterministic twin's error to remain competitive on average, while its residual rises because averaged futures are not physical, and its coverage of the true outcome to fall. If no such gap appears at any window length, the generative claim fails.
Why not just run physics-guided diffusion with fewer steps? Four-step guided DDIM may already be close to good enough. We report the full curve of quality and residual against step count. Our argument holds only if the one-step model matches the 4-step guided sampler at lower cost. Phys-Instruct's finding that residuals built into training beat 200-step guided baselines on grids makes this plausible but does not prove it for Lagrangian scenes.
Why not just sample physics-free and project onto the constraints afterward? Post-hoc projection is cheap, and for contact penetration it may be enough. Projection cannot fix a trajectory that has the wrong material behavior, though. A ball that is too bouncy remains too bouncy after its penetrations are removed. The no-residual-plus-projection ablation tests this directly, and we report it separately for the contact term and the momentum term.
Why not just use the "as static as possible" or rigid priors? These protect the background but contain no dynamics. R5DGS itself reports fidelity losses under non-rigid motion. Our soft-body splits test this baseline exactly where the claim lives.
Proposed Experiments
The primary benchmark is GSCollision, scored with RMSE, FPE, best-of-K FPE, an energy score over K = 16 samples, and the mean momentum residual. We also report latency as a function of horizon $H \in \{10, 30, 60\}$, alongside NGFF and the MPM oracle. The secondary benchmarks are held-out final frames of D-NeRF synthetic sequences, where 4D Gaussian Splatting reports 34.05 dB PSNR for interpolation and no extrapolation number is reported on Lacuna, and the R5DGS Dynamic Indoor Scene set. On these we score PSNR and LPIPS on dynamic-masked regions, in the spirit of ViDAR's Dynamic-D, along with background PSNR to check that the static scene is not corrupted.
Two checks are decisive. The first is the ambiguity sweep described above, which tells us whether generativity pays. The second is the horizon-latency curve, which tells us whether one-pass, time-parallel physics actually removes the cost that grows with horizon in ODE- and step-based methods. A negative result on either check would be informative: either short-clip futures in these benchmarks are effectively deterministic, or the residual cannot be enforced accurately in one pass for non-rigid bodies.
References
- Mean Flows for One-step Generative Modeling
- 3D MeanFlow: One-Step Point Cloud Completion and Generation via Average-Velocity Transport
- Ultra Fast PDE Solving via Physics Guided Few-step Diffusion
- Physics Informed Distillation for Diffusion Models
- OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation
- Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fields
- R5DGS: Semantic-Aware 4D Gaussian Splatting with Rigid Body Constraints for Efficient Dynamic Scene Reconstruction
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
- D-NeRF: Neural Radiance Fields for Dynamic Scenes
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Flux4D: Flow-based Unsupervised 4D Reconstruction
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- MS-GS: Multi-Appearance Sparse-View 3D Gaussian Splatting in the Wild
- Monocular Dynamic View Synthesis: A Reality Check
- Unfolding Time: Generative Modeling for Turbulent Flows in 4D