Fast dynamic scene renderers built on Gaussian splatting represent a scene as hundreds of thousands of small colored 3D blobs (Gaussians) that are projected onto the screen. They move those blobs over time with a learned deformation field: a small neural network that takes a blob's position and a time stamp and returns how the blob should shift, rotate and stretch. This network is an implicit neural representation, meaning a network that maps continuous coordinates to values instead of storing them in a grid. At render time, current methods query it for every blob in every frame, including blobs sitting on a wall that never moves. As a result, rendering cost scales with the size of the scene rather than with how much of it is moving. This is why the most accurate deformation models are also the slowest. Neural Hamiltonian Deformation Fields reaches 40.91 dB on the D-NeRF synthetic benchmark but renders at only 20–62 FPS, while purely explicit renderers exceed 100 FPS.
We propose to compile a trained deformation field, after training, into a mixture of cheaper runtime forms. Each blob, over each stretch of time, is assigned one of three forms: a frozen static pose, a short stored motion curve whose number of control points varies from blob to blob, or a live query to the original network. The assignment solves a budget problem. It minimizes measured rendering error subject to a memory budget and a per-frame compute budget, so the frame time grows with the amount of hard-to-compress motion instead of the total number of blobs. The central claim is that the neural field should be kept as one priced runtime option among several, not either always evaluated or baked away entirely. Treated this way, it recovers the quality of slow, accurate deformation models at the frame rates of explicit ones.
What exists, and what is missing
Several papers already trade the implicit field for speed. Each makes one uniform choice for the whole scene.
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (4D-GS) stores motion in six factorized feature planes and queries a tiny decoder for every Gaussian at every time step. It reports 34.05 dB and 82 FPS at 800×800 on an RTX 3090 on D-NeRF. It is our starting point and our main teacher model, and its per-Gaussian query is exactly the cost we remove.
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (4DGS-native) removes the network entirely by making each primitive a 4D spacetime Gaussian that is sliced at time $t$. It reports 114 FPS, 32.01 PSNR and 0.055 LPIPS on Plenoptic Video. The authors note that the number of primitives grows with sequence length.
Superpoint Gaussian Splatting predicts rigid transforms for clusters of Gaussians and can precompute and interpolate those transforms at inference, reaching 227 FPS on synthetic data. It assumes local rigidity and needs an extra network for non-rigid motion.
Fourier PlenOctrees bakes a fixed set of Fourier coefficients into every leaf of a tree that covers every place the object ever visits. This costs several gigabytes, because static and dynamic regions pay the same rate.
BakedAvatar fully bakes implicit head fields into meshes and textures and then fine-tunes the baked assets photometrically. We reuse that fine-tuning step. BakedAvatar, however, is tied to a parametric face model and keeps no runtime fallback.
These papers motivate and supply methods for our proposal. Two papers constrain it. RobustSplat shows that treating transient error as missing geometry creates permanent floating artifacts, which warns against assigning tiers from raw photometric error early in training. We assign tiers only after the teacher has converged. Several papers, including 4D-GS and the Hamiltonian model, also report that continuous fields degrade under fast motion and under objects entering or leaving the scene. Any single stored curve per primitive will inherit that weakness, which is the case for letting assignments change across time segments.
What no cited work does is treat static, curve and live-network as priced alternatives chosen per primitive and per time segment under explicit memory and latency budgets.
Core Claim
Given a converged dynamic Gaussian model with a deformation field $D_\theta(\mathbf{x}, t)$, a budgeted mixture of runtime forms achieves a better three-way trade-off between quality, frame time and memory than any single uniform form. The uniform forms are querying the field everywhere (4D-GS), baking everything at a fixed rate (Fourier PlenOctrees-style), rigid clusters (Superpoint Gaussian Splatting) and native 4D primitives (4DGS-native).
The mixture also has an operational benefit. Frame time becomes predictable from the fraction of primitive-time that is genuinely hard to compress. That makes latency a controllable knob rather than a property of the scene.
Research Program
We first train a teacher model: 4D-GS as the primary teacher, and the Hamiltonian model as a second teacher if it can be faithfully reproduced.
For every Gaussian $i$ we cut the sequence into temporal segments. For each segment we fit candidate runtime forms $o$ to the teacher's trajectory. The candidates are:
- a constant pose;
- cubic B-spline curves over position, rotation and scale with $K \in \{2, 4, 8, 16\}$ control points;
- a live network query.
The live query reproduces the teacher exactly by construction.
For each candidate we measure a distortion $e_i(o)$: the deviation of the baked trajectory from the teacher, weighted by the Gaussian's opacity and its projected screen footprint over the training cameras. We also record its storage $b(o)$ and its per-frame compute $c(o)$, where $c$ is measured, not estimated, on the target GPU. The assignment then solves
$$ \min_{\{o_i\}} \sum_i e_i(o_i) \quad \text{s.t.} \quad \sum_i b(o_i) \le B, \qquad \sum_i c(o_i) \le C, $$
where $B$ is the memory budget and $C$ is the per-frame compute budget. We solve this with a two-multiplier Lagrangian greedy search, the standard approach for multiple-choice knapsack problems.
Segment boundaries are placed where the teacher trajectory's residual under a small $K$ jumps. This lets a blob be frozen before an object enters the scene and live during the entry.
The runtime evaluates the live network only on the gathered subset of live primitives, which requires a batched query kernel over an index list. After assignment, all baked curves are fine-tuned against the training images while the assignment is held fixed, following BakedAvatar. This lets the compiled model correct teacher errors and not only imitate them.
The analysis half of the program tests the proportionality claim directly. For each scene we report the fraction of primitive-time assigned to the live network. We predict that speedup over the teacher is roughly inversely proportional to one plus the live fraction times the teacher's per-query cost. The form of this prediction follows from frame time being splatting cost plus live-query cost.
Baselines
Uncompiled teacher (naive reference). The teacher itself is 4D-GS querying its deformation decoder for all Gaussians every frame. It is retrained with the authors' settings on our hardware, and its reported D-NeRF figures are 34.05 dB and 82 FPS on an RTX 3090 at 800×800. It is the "do nothing" point. Beating it means at least 2× its measured FPS with a PSNR drop of no more than 0.2 dB and no increase in storage beyond its roughly 18 MB.
Native 4D primitives (strongest speed competitor). 4DGS-native reports 114 FPS, 32.01 PSNR and 0.055 LPIPS on Plenoptic Video, and 34.09 dB on D-NeRF. It is the strongest competitor because it already has no network at runtime. We rerun it with its public settings, on the same GPU and resolution, and report storage alongside quality and speed. Beating it means dominating it on the quality–FPS–memory front at matched memory. Matching its FPS with higher PSNR would count, as would matching its PSNR at lower memory, particularly on the longest sequences.
Precomputed rigid clusters. Superpoint Gaussian Splatting has no reported D-NeRF PSNR in our file, only the statement that it matches or outperforms TiNeuVox and HexPlane. It is run with interpolated precomputed transforms. It tests whether plain rigid grouping already captures most of the gain.
Quality ceiling. The Hamiltonian deformation model, at 40.91 dB on D-NeRF, sets the upper bound. If it can be reproduced, compiling it is the most interesting test. Recovering most of its quality at 4DGS-native frame rates would be the headline result.
Ablations. Four ablations isolate the key ingredient:
- uniform baking, where every primitive gets the same $K$ at matched memory;
- adaptive curves with no live option;
- tier assignment by motion magnitude instead of measured distortion;
- no photometric fine-tuning.
All methods get the same training views, the same initialization and the same GPU. The compiled model's additional fitting budget is reported as wall-clock time, not hidden.
Why not just…?
Why not just use native 4D Gaussians, which already need no network? This is the most serious threat. 4DGS-native reports 114 FPS on Plenoptic Video, and on short synthetic clips it may simply win. Our case rests on two points. Its authors report that primitive count grows with duration, and it trains motion and appearance jointly with no way to spend a fixed latency budget. The matched-memory comparison, and a stress test that concatenates Neu3D sequences into longer clips, will settle this. If 4DGS-native dominates there too, the proposal's value reduces to compiling slow high-quality teachers, and we will say so.
Why not just bake every primitive into adaptive-length curves and drop the network? If the ablation without a live option matches the full method at equal memory, the live tier is unjustified. The expected failure cases are the discontinuous motions and entering or leaving objects on which both 4D-GS and the Hamiltonian model report blurring, where no short curve fits well. We will report per-scene results on HyperNeRF and on D-NeRF's Bouncing Balls, a scene with topology-like events. If the gap is under 0.1 dB everywhere, we drop the live tier and publish the simpler method.
Why not just precompute a per-frame pose table? A lookup table is the fastest runtime form. At 100k Gaussians, 300 frames and about 10 floats per pose, it costs on the order of a gigabyte, similar to the memory problem of Fourier PlenOctrees, and it cannot answer arbitrary time stamps. It appears as the maximum-memory endpoint of our budget sweep rather than as a separate method.
Why not just shrink or distill the deformation network? A smaller network still pays per primitive for static content. We include a teacher whose plane resolution and decoder width are cut until its FPS matches ours, and compare PSNR at equal frame time.
Proposed Experiments
We run on three benchmarks:
- D-NeRF synthetic, as a controlled test of monocular dynamics;
- HyperNeRF vrig, for real monocular capture with topological change (the file reports no PSNR numbers here for our baselines, so all are rerun);
- Plenoptic Video / Neu3D, for multi-view real scenes with flames and reflections.
We report PSNR, SSIM, LPIPS, FPS at 800×800 on an RTX 3090 and on a GTX 1060-class GPU (as Superpoint Gaussian Splatting did), storage in MB, the per-frame share of time spent in deformation, and the live fraction.
Three checks are decisive:
- Sweeping the compute budget $C$ must trace a quality–FPS curve that lies strictly above 4DGS-native's single operating point on Plenoptic Video at matched memory.
- The measured speedup must track the live-fraction prediction across scenes; a poor fit would mean the cost model, and therefore the budgeting, is wrong.
- The full method must beat the no-live-option ablation on at least the discontinuous-motion scenes, or the three-tier design is abandoned in favor of adaptive curves alone.
References
- Neural Hamiltonian Deformation Fields for Dynamic Scene Rendering
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting
- Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction
- Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time
- BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS