Rendering a dynamic 3D scene, such as a person cooking in a kitchen captured by a camera rig, still means redrawing every part of the scene at every frame. Most of each frame is static background that barely changes from one frame to the next. Graphics systems have long saved work by reusing the previous frame, warping its pixels to the new camera position and redrawing only what is newly revealed. That trick breaks as soon as something in the scene moves. A walking person drags a "tear" across the warped image, and the background they uncover has no pixels to warp from. This proposal makes frame reuse work in dynamic scenes. It splits an explicit dynamic scene model into a static layer and a moving layer. The static layer is rendered without the moving objects, cached, and treated as a memory of the scene. Each frame warps that cached layer to the new view, redraws only the small regions the camera has newly revealed, and fully redraws only the moving layer before combining the two by depth.
The representation is 3D Gaussian splatting: the scene is a cloud of millions of small, colored, semi-transparent 3D blobs ("Gaussians") projected onto the screen and blended front to back, with no neural network evaluated per pixel. Dynamic variants give each blob a trajectory over time. Two observations drive the idea. First, background revealed by a moving object never needs to be filled in: the cached static layer was rendered without that object, so the uncovered background is already in the cache. Second, an explicit dynamic model knows exactly which blobs move and by how much, so it can tell which screen regions must be redrawn. The claim is that a dynamic Gaussian renderer can redraw only its moving content plus the regions the camera newly reveals, with a measured and bounded image error relative to the full render. The value lies in the high-resolution, stereo, and mobile settings where today's dynamic splatting frame rates are not enough.
Where this sits in prior work
Frame reuse for neural rendering is the motivating precedent, and it also defines the limit we must cross. Potamoi warps pixels from a previous reference frame and calls the expensive radiance model only for disoccluded holes. It reduces matrix-multiply work by up to 88% with under 1.0 dB peak signal-to-noise ratio (PSNR) loss, and reaches a 53.1x hardware speedup. The method explicitly assumes a static scene, and independent object motion breaks it. We keep the warp-and-fill structure but apply it only to a layer that really is static, and we treat moving content separately.
Explicit dynamic Gaussian models supply the representation we build on. Fully Explicit Dynamic Gaussian Splatting (Ex4DGS) stores Gaussians at keyframes and interpolates them with splines, removing deformation networks. It reaches 62.1 frames per second (FPS) at 1352×1014 on an RTX 2080Ti and 32.11 dB PSNR on Neural 3D Video. SWinGS fits local canonical models over sliding windows, at about 71 FPS on Neural 3D Video. STDR uses learnable masks to separate static structure from motion during training, reaching 41.00 dB average on the D-NeRF benchmark. All three still redraw the entire scene every frame. Our proposal does not change how they are trained; it changes what is redrawn at display time. Ex4DGS is the primary host because its explicit trajectories make the static/moving split and the "which tiles changed" test cheap and exact.
Visibility culling for splatting constrains the design. NVGS learns a small network that predicts which Gaussians are hidden, cutting memory from 17.7 GB to 4.0 GB on a static forest scene. OccluGaussian precomputes region visibility masks. Both assume nothing moves. Rather than learn a time-varying visibility function, we use the previous frame's render state as temporal context, which needs no training.
The temporal-memory framing comes from perception. OMR keeps a recurrent memory in 2D video lane detection and routes it only into regions its obstacle mask flags. It reaches F1 0.936 on VIL-100 at about 105 FPS. Our cache plays the same role in 3D: persistent state, updated selectively where change is detected.
Finally, CoDa-4DGS is the closest system touching all three directions. It distills 2D foundation-model features into 4D Gaussians and adds a refinement network for dynamic contours, at a reported 2.1x compute increase. We pursue the opposite trade: no added networks, and a lower per-frame cost than the base model.
Core mechanism
After training, each Gaussian is labeled static or moving from its trajectory: its maximum displacement and opacity change over the sequence are compared against a threshold tied to screen-space pixel size. At render time the static layer is stored as a "deep" cached image. Each pixel holds $K$ depth bins, and each bin stores premultiplied color, accumulated transmittance, and a representative depth. A single color and depth is not enough, because alpha blending in splatting does not separate cleanly when static and moving Gaussians interleave in depth, for example a hand moving behind a semi-transparent curtain. With $K$ bins, moving Gaussians are inserted into the correct bin, so the composite is exact whenever no bin contains both kinds of Gaussian.
Each new frame then proceeds in four steps. The cached static layer is reprojected to the new camera using its stored depths. Tiles whose reprojection stretches, reveals new surface, or exceeds a view-angle budget for view-dependent color are re-rasterized from static Gaussians only. Moving Gaussians are rasterized in full, and the result is composited into the warped deep layer. A full static refresh runs on a fixed schedule or whenever accumulated warp error is predicted to exceed a set value. The schedule is the knob that trades fidelity for speed, and we report the whole curve rather than one operating point.
Research Program
The work has three parts. The first builds the renderer: a deep static cache, a warp step, tile-level change detection driven by Ex4DGS trajectories, and a compositor, all implemented in the same tile rasterizer as the base model so the comparisons isolate the reuse logic. The second is an error analysis. We measure, per scene, the fraction of pixels where static and moving Gaussians interleave inside a bin, and how image error relative to the full render scales with $K$, warp distance, and refresh interval. The goal is a usable rule for setting $K$ and the refresh schedule. The third tests portability. We apply the same renderer to SWinGS and STDR models, whose motion comes from small deformation networks. There, labeling Gaussians static or moving requires sampling the deformation network, which is costlier and may misclassify slow motion, and we quantify how much of the gain survives.
Baselines
The naive reference is the unmodified base renderer drawing every Gaussian at every frame. For Ex4DGS on Neural 3D Video, the reported reference is 62.1 FPS at 1352×1014 on an RTX 2080Ti, with 32.11 dB PSNR, both from Ex4DGS. We re-run it on the same GPU, resolutions, and camera paths so speed comparisons are paired. Beating it means at least a 2x median frame-time reduction at 2x and 4x resolution and in stereo, with PSNR against the full render of the same model above 40 dB and ground-truth PSNR within 0.3 dB of the base model.
The strongest existing competitor is Potamoi-style whole-frame warping, our reimplementation of Potamoi's sparse radiance warping applied to the full dynamic render. Pixels are reprojected from the previous frame with rendered depth, and tiles with holes are redrawn. It runs on the same rasterizer and is given the same refresh-schedule tuning budget, a sweep over the same grid. Its known result is under 1.0 dB PSNR drop with large compute savings on static scenes; no reported result found on Lacuna for it on dynamic scenes. We beat it if, at matched frame time, our method has lower error around moving objects, measured on a motion-region mask, and less flicker.
The key ablation is our method with $K=1$, a single static depth layer, together with a variant that skips the static/moving split and labels everything moving. The first isolates the deep cache. The second shows that the split, not warping itself, carries the gain.
We also run temporal visibility reuse, which renders only Gaussians that contributed to the previous frame plus all moving ones. This represents the cheap culling answer to the problem NVGS addresses with a learned network. Finally, we run a low-resolution render followed by bilinear and learned upsampling, at matched frame time.
Why not just…?
Why not just render dynamic splatting in full, since it already runs at 60–100 FPS? This is the most serious objection, and on a desktop GPU at Neural 3D Video resolution it may be right. The reported 62.1 FPS for Ex4DGS and about 71 FPS for SWinGS are below a 90 FPS stereo headset budget, and cost grows with pixel count and with a second eye. If the measured speedup at 1x resolution on a single view is under 1.3x, we say so and confine the claim to the high-resolution, stereo, and mobile regimes, where the experiments below test it directly.
Why not just warp the whole frame as Potamoi does? Potamoi itself states the method relies on a static scene. Moving content, which is exactly what dynamic benchmarks emphasize, has no valid previous pixel, and newly revealed background has no source at all. The whole-frame warping baseline measures this failure on motion-region masks. If its error turns out to be small because motion is slow, that is a real finding: the static/moving split would then be unnecessary on those scenes, and we would report it.
Why not just reuse last frame's visible set of Gaussians? This cheap temporal culling needs no cache or compositor, and it could capture much of the gain when the scene is dominated by Gaussians hidden behind other Gaussians. It does not avoid re-blending the visible static Gaussians every frame, and that work is what our cache removes. We run it as a baseline and also stack it with our method. If it matches us alone, the deep cache is not worth its memory.
Why not just render at lower resolution and upsample? Upsampling saves the same work uniformly. It blurs fine static texture, the region where our method is nearly lossless, and leaves moving edges no better. The matched-frame-time comparison on LPIPS settles this, with LPIPS reported separately for static and motion regions.
Proposed Experiments
The main benchmarks are Neural 3D Video and Technicolor (multi-view, with Ex4DGS and SWinGS as host models) and the D-NeRF synthetic set (STDR as host). For each scene we render held-out test views and also smooth novel camera paths that sweep head-like rotations and translations at 90 Hz, since reuse depends on frame-to-frame camera motion. We report two error measures for every frame. Error against ground truth (PSNR, SSIM, LPIPS) shows that quality is preserved; error against the full render of the same model isolates what reuse itself costs. We also report a temporal flicker score, computed as frame-difference error after compensating for camera motion, together with median and 99th-percentile frame time, memory for the deep cache, and the fraction of tiles redrawn per frame. Hardware covers an RTX 2080Ti, to anchor against the Ex4DGS figures, a current desktop GPU, and a mobile-class GPU. Resolution runs at 1x, 2x, and 4x, in mono and stereo.
Three checks decide the claim. The first is that frame time falls roughly in proportion to the moving-content fraction plus the newly revealed fraction; frame time should track the redrawn-tile count, because unchanged tiles cost only a warp. If speedup does not track that count, warp and composite overhead dominate and the design fails. The second is that error against the full render stays above 40 dB in motion regions with $K$ of 4 or less. If interleaving forces $K$ much higher, the cache memory erases the benefit. The third is that the method beats whole-frame warping on motion-region error at matched frame time, and beats temporal visibility reuse on frame time at matched quality, in the high-resolution stereo setting. Failing either comparison means a simpler method suffices, and we would report that as the result.
References
- Potamoi: Accelerating Neural Rendering via a Unified Streaming Architecture
- Fully Explicit Dynamic Gaussian Splatting
- SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting
- STDR: Spatio-Temporal Decoupling for Real-Time Dynamic Scene Rendering
- NVGS: Neural Visibility for Occlusion Culling in 3D Gaussian Splatting
- OccluGaussian: Occlusion-Aware Gaussian Splatting for Large Scene Reconstruction and Rendering
- OMR: Occlusion-Aware Memory-Based Refinement for Video Lane Detection
- CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Driving