Today's methods for turning multi-camera video into a scene you can view from anywhere force a trade-off between file size and rendering speed. The smallest models store a compact code and decode it with a small neural network every frame. MEGA and 4D Scaffold Gaussian Splatting cut storage by 40× to 254×, but they render at about 70–130 frames per second (FPS). The fastest model, 4DGS-1K, stores explicit primitives, skips inactive ones, and exceeds 1000 FPS, but it still needs about 50MB per scene. We propose to bridge the two by distillation. A compact, slow "teacher" model renders many extra training images at camera positions and moments the real cameras never captured. A fast, explicit "student" is trained on those images under a hard budget on how many primitives it may use. The primitives here are 3D Gaussian splats: small colored, semi-transparent ellipsoids that are projected onto the screen and blended.
The catch, and the heart of the proposal, is that these teacher-made images are a form of data augmentation, and they can be physically inconsistent. Away from the real camera paths, dynamic teachers produce semi-transparent "ghosts" and floating blobs. These appear because reconstruction assumes a static world, an assumption that moving objects break. A student trained on such images faithfully copies the errors. We therefore accept a teacher pixel as supervision only if it passes geometric and motion checks. The checks are borrowed from augmentation work that recomputes geometry in step with the image instead of warping pixels blindly. In two sentences: teacher-rendered novel views and in-between timestamps give a tiny, fast dynamic model the dense supervision it needs to survive aggressive primitive budgets, but only if each pseudo-pixel is verified against real cameras and motion. The claim is that this checking, not the extra views alone, is what lets one model reach both MEGA-scale storage and 4DGS-1K-scale speed.
Core Claim
The research file shows no paper achieving both extreme compression and kilo-FPS rendering for dynamic scenes. This is its "Gap 3", and the reason is structural. Compression through neural decoders (MEGA's MLP-decoded color, Scaffold's anchor MLP, the six-plane deformation queries of 4D-GS) moves cost from storage into per-frame computation. We claim that an explicit, decoder-free student with far fewer primitives than 4DGS-1K can match a compact teacher's quality, provided three conditions hold. The student must be trained on teacher renders at unseen poses and in-between times. Those renders must be filtered by physical consistency. And the student's size must be fixed from the start, not grown by unconstrained cloning. The testable prediction is specific: without filtering, extra pseudo-views help little or even hurt in moving regions; with filtering, they recover most of the dynamic-region quality that aggressive pruning destroys.
Why this is a real gap and what prior work contributes
Four lines of work motivate the proposal.
The first is the fast explicit renderers. 4DGS-1K prunes Gaussians whose opacity varies erratically over time and uses per-keyframe masks of which primitives are visible. It reports that preprocessing, not rasterization, is now the bottleneck. This rules out any design that adds per-frame neural decoding, which is exactly what our student avoids. Native 4D primitives show that quantizing attributes at load time (residual vector quantization) cuts storage from 1183MB to 56MB for a 0.1 dB loss without touching frame rate. This supplies the post-hoc compression step we reuse, and it is also the strongest competing recipe.
The second is the compact decoder-based models. MEGA and 4D Scaffold GS supply our teachers. Scaffold also supplies a constraint: naive pruning preferentially deletes short-lived, fast-moving components, because their gradient signal is swamped by the persistent background. This predicts that a budgeted student trained only on real frames will fail in exactly the dynamic regions. Dense teacher supervision at intermediate times is meant to counter that failure.
The third is the failure of the static-world assumption, which motivates the checks. When a moving object passes through a scene, the step that adds new primitives wherever error is high ("densification") spawns frozen clusters of Gaussians in mid-air (RobustSplat). Separating moving objects leaves ghosting and blurred contact boundaries (Split4D). Teacher renders off the camera path inherit these errors, so they are unsafe augmentation. RobustSplat discards dynamic content. We instead need a signal for which dynamic pixels to trust.
The fourth is geometry-consistent augmentation, which supplies the method. Camera Pose Matters replaces random cropping with homography warps that simulate true camera rotation and recomputes depth in step, halving depth error (0.183 to 0.091 relative error). DepthMix enforces depth order when pasting. HOGSA uses Gaussian renders as augmentation but corrects poses to avoid interpenetration. All three operate on single frames, static meshes, or parametric hands, and all target downstream perception. None augments a 4D scene model's own training signal. We carry their principle into 4D: an augmented sample should be admitted only if its geometry and motion agree with what the real cameras saw. For motion, GaussianFlow supplies the tool. It projects each Gaussian's 3D velocity into 2D image motion, which lets us compare the teacher's implied motion against optical flow measured on its own consecutive renders and on the real frames.
Research Program
Teachers are MEGA and 4D Scaffold GS trained on the real views as published. The student is a 4DGS-1K-style set of explicit time-varying Gaussians with no MLP at render time. Its primitive count is capped at a target (for example 25%, 10%, and 5% of the 4DGS-1K count). Cloning is allowed only by swapping out low-contribution primitives, so peak training memory is bounded rather than growing unpredictably.
Pseudo-supervision comes in three kinds, ordered by how much we trust them. The first kind is exact rotations. Rotating a real camera about its optical center yields a new image that is an exact homography of the real one, as in Camera Pose Matters, so it needs no teacher and carries no teacher error. It adds little parallax and serves as a clean control. The second kind is in-between timestamps at real camera poses. These are rendered by the teacher and supervise the student's temporal opacity and visibility masks between keyframes, where 4DGS-1K currently interpolates. The third kind is translated novel viewpoints at arbitrary times, rendered by the teacher.
Every teacher pixel of the second and third kinds receives a trust weight in [0,1] from three checks. The first is a depth reprojection test. The teacher's rendered depth is used to warp the pixel into the nearest real camera at the nearest real frame, and the pixel is trusted if its color agrees there and it is not occluded. This is DepthMix's depth-order logic made multi-view. The second is a motion test. The teacher's projected Gaussian velocities, computed with GaussianFlow's differentiable splatting, must agree with optical flow computed between consecutive teacher renders. Disagreement flags drifting texture and ghosts. The third is a static-region shortcut. Where the teacher labels content as static, the pixel can be checked against real frames from any time, because the static assumption genuinely holds there. The student loss is the photometric loss on real views plus the trust-weighted loss on pseudo-views. We will measure how trust weights correlate with actual teacher error on held-out real cameras, since the checks are only useful if that correlation is high.
Baselines
The naive reference point is 4DGS-1K trained on the real views only, with its pruning pushed to the same primitive budgets as our student. This isolates what happens when you shrink a fast model with no extra supervision. The file reports that 4DGS-1K renders at "over 1000 FPS" on an RTX 3090 with storage "as low as 50MB" on N3DV. We found no reported PSNR for 4DGS-1K on N3DV on Lacuna, so we will reproduce it ourselves.
The strongest existing competitor on the storage-and-speed question is 4DGS-1K followed by load-time residual vector quantization from native 4DGS. That paper reports a 1183MB to 56MB reduction with a 0.1 dB PSNR drop on Plenoptic Video. This stack keeps kilo-FPS speed and shrinks storage with no new ideas, so it is the baseline we must beat. We will apply the same quantization to our student, so any gain comes from training, not the codec.
The quality references are the teachers themselves. 4D Scaffold GS reports 28.86 dB PSNR in dynamic regions on N3DV, versus 27.65 for 4DGS, at 149MB and about 130 FPS. MEGA reports 125× compression over 4DGS on N3DV at 70–80 FPS, and 32.02 dB on Technicolor Birthday at 31.42MB.
The ablations that isolate the key ingredient are three. First, the same student trained on ungated teacher views, which tests whether trust weighting matters. Second, exact-rotation views only, which gives clean but low-information augmentation. Third, time-only versus view-only pseudo-samples.
Fairness rules apply to all methods. Every method sees the same real cameras and frames. Tuning budgets are matched in GPU-hours, and the teacher's training cost is counted against our method. FPS is measured on one RTX 3090 at the published resolutions, including all preprocessing. Beating the strongest baseline means matching or exceeding its full-frame and dynamic-region PSNR at no more than half its quantized storage and at 1000+ FPS, or, at equal storage, gaining at least 0.5 dB in dynamic regions.
Why not just…?
Why not just stack 4DGS-1K pruning with load-time quantization and stop there? This is a real threat. The native-4DGS result suggests quantization costs almost nothing, so the stack may already reach 10–20MB at kilo-FPS. The open question is whether its quality holds when primitive count, not just bits per primitive, is cut further. Scaffold's observation that pruning kills short-lived content predicts it does not. Our budget sweep settles this directly. If the stacked baseline matches our student at every budget in dynamic-region PSNR, the distillation idea is refuted and we will report that.
Why not just render the compact teacher at test time, since 70–130 FPS is already real time? For a single monitor it often is. The target settings are different: stereo virtual reality needing two high-rate views with a tight latency budget (ESCA), closed-loop robot navigation that renders many hypothetical views per decision (vision-only navigation in a NeRF world), and weaker GPUs. In all of these, per-frame MLP decoding dominates cost. We will report latency at multiple views per timestep to show where the gap matters and where it does not.
Why not just add all teacher renders without checks, like ordinary pseudo-labelling? The augmentation literature says inconsistent samples actively mislead. Standard crops doubled depth error relative to perspective-correct ones (Camera Pose Matters), and depth-unaware pasting created impossible layouts that DepthMix removed. The dynamic-scene literature shows teachers do produce floaters and ghosts off-path (RobustSplat, Split4D). The ungated ablation measures this. If ungated views work as well, the checks are unnecessary and the contribution shrinks to plain distillation.
Why not just use an off-the-shelf 2D segmenter to mask moving regions in teacher renders? Split4D reports that tracking-based decomposition drifts under fast motion and occlusion (0.893 vs 0.592 mIoU against the tracker-reliant SA4D on Multi-Human). Masking also removes exactly the dynamic content we most need to supervise. We will include a segmenter-mask variant as a secondary comparison.
Proposed Experiments
The main benchmarks are N3DV and Technicolor, since they carry the published storage and FPS numbers. The D-NeRF synthetic scenes provide ground-truth views at arbitrary poses, which lets us measure teacher pseudo-view error exactly and validate the trust weights. 4D-GS reports 34.05 dB at 18MB there. We add a sparse-camera variant of N3DV, holding out half the rig, because pseudo-supervision should matter most when real views are scarce.
Metrics are PSNR, SSIM, and LPIPS on full frames and on dynamic-region masks, plus serialized size, peak training memory, and end-to-end FPS on an RTX 3090. A temporal-flicker score computed on in-between timestamps checks whether time pseudo-samples actually smooth the visibility masks.
The decisive checks are four. First, on D-NeRF synthetic, the trust weight must rank teacher pixels by true error well, with a clearly positive rank correlation, or the gating premise fails. Second, across primitive budgets on N3DV, gated distillation must beat both the stacked prune-plus-quantize baseline and ungated distillation in dynamic-region PSNR, with the margin widening as the budget shrinks. Third, the student's end-to-end FPS must stay within 10% of 4DGS-1K's at the same resolution. Fourth, in the sparse-camera setting, gains should grow relative to the full rig. If gains instead vanish there, that would indicate the teacher's own sparse-view errors are passing the checks.
References
- MEGA: Memory-Efficient 4D Gaussian Splatting for Dynamic Scenes
- 4D Scaffold Gaussian Splatting for Memory Efficient Dynamic Scene Reconstruction
- 1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
- 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Camera Pose Matters: Improving Depth Prediction by Mitigating Pose Distribution Bias
- Three Ways To Improve Semantic Segmentation With Self-Supervised Depth Estimation
- HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation
- GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation
- ESCA: Enabling Seamless Codec Avatar Execution through Algorithm and Hardware Co-Optimization for Virtual Reality
- Vision-Only Robot Navigation in a Neural Radiance World