Stopping Depth as Motion: Variable-Depth Residual Codes for Dynamic Gaussian Scenes

Most of a video of a dynamic scene does not move. Methods that reconstruct such scenes in 3D still pay for motion everywhere. They either run a neural network that predicts how every point moves at every moment, or they store a motion vector for every primitive, which is one of the small colored 3D blobs ("Gaussians") the scene is drawn from. A separate step then has to decide which blobs are background and which are moving, using learned masks or hand-set thresholds. We propose to merge these two jobs. Each Gaussian's motion over a short time window is written as a sum of entries from a shared codebook, refined in stages: the first stage gives a coarse motion and each later stage corrects what is left over. This is residual vector quantization. Each Gaussian is also allowed to stop after any number of stages, and a bit-cost penalty makes every stage justify itself. A Gaussian that stops at zero stages is static. The static-scene assumption then stops being a global premise that fails. It becomes the cheapest code in the dictionary, and the static/dynamic split falls out of the compression objective.

The payoff is practical. Uncompressed dynamic Gaussian scenes can exceed 2 GB (1000+ FPS 4D Gaussian Splatting). The best compressed pipelines keep a continuous motion network that becomes their storage bottleneck at low bitrates (P-4DGS). A variable-depth discrete motion code removes that network. It spends bits roughly in proportion to how much a region actually moves, and it hands back a static/dynamic decomposition for free. That last property is the part we most need to test.

Core Claim

The claim is that learned per-primitive stopping depth in a residual motion code does two jobs at once. It is a better static/dynamic decomposition signal than a learned per-Gaussian dynamic weight or a derivative-based pruning threshold. It is also a more compact motion representation than a deformation network or scalar-quantized motion residuals.

Concretely, time is split into windows of $W$ frames, each starting at a keyframe. For Gaussian $i$ in window $k$, define its motion vector $\tau_{i,k} \in \mathbb{R}^{d}$ as its position and rotation offsets at $S$ sample times, measured relative to its state at the window's keyframe. The dimension is $d = 7S$: 3 position and 4 rotation values per sample time. Offsets between sample times are interpolated. Measuring every offset from the keyframe, rather than from the previous frame, follows the finding in ResQ that frame-to-frame residuals pile up error while keyframe-to-frame residuals do not.

The motion is coded as

$$\hat{\tau}_{i,k} = \sum_{m=1}^{D_{i,k}} c_m(i,k),$$

where $c_m(i,k)$ is the codebook entry chosen at stage $m$ from codebook $\mathcal{C}_m$ and $D_{i,k} \in \{0,\dots,M\}$ is that Gaussian's stopping depth. Each codebook is shared across all Gaussians and all windows. Training minimizes

$$\mathcal{L} = \mathcal{L}_{\text{photo}} + \lambda \sum_{i,k} D_{i,k}\log_2 K + \beta\,\mathcal{L}_{\text{commit}},$$

where $\mathcal{L}_{\text{photo}}$ is the usual image rendering loss, $K$ is the codebook size (so each stage costs $\log_2 K$ bits), $\lambda$ trades bits against quality, and $\mathcal{L}_{\text{commit}}$ is the standard term pulling the continuous motion toward its code. With $D_{i,k}=0$ the Gaussian is fixed in space for that window.

Relation to Prior Work

Three compression methods motivate the gap and also constrain the design. CompGS applies single-codebook vector quantization to the colors, scales and rotations of static scenes. It finds positions too sensitive to quantize this way, which argues for residual refinement over a single lookup when motion is geometric. P-4DGS predicts attributes from anchors and motion from a deformation network, with adaptive scalar quantization and entropy coding. It reaches about 1 MB per scene, but its motion network is uncompressed and all points share one canonical space. CompGS++ uses video-codec-style I-frames and P-frames with static and dynamic sets and continuous motion residues. Its split comes from tracking motion residues rather than from the rate objective. We differ from all three in two ways. Our motion is a discrete code with variable depth rather than a network or scalar residues. And the static/dynamic split is the depth itself rather than a separate stage.

Two decomposition methods set the bar for the split. SDD-4DGS learns a per-Gaussian dynamic weight, guided by uncertainty masks from DINOv2 (a pretrained image-feature network). Split4D clusters Gaussians by velocity and identity features. Neither compresses anything. We replace SDD-4DGS's continuous weight with an integer depth that has a price in bits, and we use no DINOv2 prior.

4DGS-1K prunes short-lived "flickering" Gaussians using a temporal-derivative score with manual thresholds. Our rate term plays the same role without a threshold, because a Gaussian that needs deep codes but contributes little to the rendered images is simply expensive.

Three works supply the method. The codebook-stage machinery comes from RQ-VAE. Reusing codebook entries across time windows follows the cached-centroid idea in Quant VideoGen. Keyframe anchoring comes from ResQ. None of these three touches 3D geometry.

Research Program

We build on the explicit Gaussian representation used by 4D-GS, but replace its HexPlane-plus-network deformation with the per-window quantized motion vectors defined above. Static attributes use CompGS-style vector quantization so that the comparison isolates the motion code.

Training has three phases. First, we fit continuous motion vectors with densification, delaying densification as in RobustSplat so that moving objects do not spawn permanent floaters. Second, we initialize the stage codebooks by running k-means on the residuals left after each stage. Third, we train jointly using a straight-through estimator, which lets gradients pass through the non-differentiable code lookup. Stopping depth is learned by randomly truncating codes during training, in the style of nested dropout, which forces early stages to carry the most important motion. On top of that sits a relaxed per-Gaussian stopping probability, whose expected depth enters the rate term. At test time the depth is hard: a Gaussian stops when the next stage's error reduction falls below $\lambda \log_2 K$.

We will analyze three things. The first is whether depth maps onto motion magnitude and object identity. The second is whether codebook entries are reused across windows and across objects. Repeated gestures and rigid parts should share codes, and that sharing is where the temporal-redundancy gain should come from. The third is how the choice of window length $W$ trades keyframe overhead against drift.

Baselines

The naive reference point is uncompressed 4D-GS. On the D-NeRF synthetic benchmark it reports a mean PSNR of 34.05 dB at 18 MB and 82 FPS (4D Gaussian Splatting). A second naive point is our own pipeline with every Gaussian forced to depth zero. This is pure static reconstruction, and it measures how much the static assumption costs on each scene.

The strongest compression competitor is P-4DGS, because it attacks the same storage problem with predictive coding and entropy models and reaches about 1 MB. We found no reported average PSNR for P-4DGS on D-NeRF on Lacuna, so we will rerun its released configuration on the same scenes. We will then compare rate-distortion curves, meaning PSNR plotted against storage size. Beating it means a higher PSNR at matched size across at least the low-bitrate half of the curve, where its motion network dominates storage.

CompGS++ is the strongest real-data competitor. On Neu3D Coffee Martini it reports 28.37 dB PSNR at 0.05 MB per frame (CompGS++). Matching that PSNR at a lower per-frame size would count as beating it.

For quality at unconstrained size, SDD-4DGS reports 32.39 dB PSNR on Neu3D and 34.82 dB on D-NeRF (SDD-4DGS). We expect to lose a little PSNR to it. Success means staying within 0.5 dB at more than 10 times smaller storage.

All methods get identical training views, the same GPU, and the same iteration budget. Each gets a fixed sweep over its rate knob: $\lambda$ for us, quantization step sizes for P-4DGS, and the rate weight for CompGS++. The key ablation keeps everything in our pipeline but replaces learned stopping depth with a fixed depth plus SDD-4DGS's continuous dynamic weight thresholded at 0.5. This isolates whether depth-as-decomposition matters.

Why not just…?

Why not just use SDD-4DGS's dynamic weight to split the scene, then compress each part with P-4DGS-style scalar quantization? This is the most serious threat, because it is a straightforward assembly of existing parts. Our argument is that a continuous weight carries no notion of cost. A Gaussian with a small motion gets a middle weight and full-precision motion anyway, while depth-with-rate forces a discrete decision priced in bits. The fixed-depth-plus-weight ablation and a direct "SDD split + scalar quantization" pipeline settle this. If either matches our rate-distortion curve, the claim fails and we will report that.

Why not just use one larger vector-quantization codebook for motion, as CompGS does for appearance? Fine positional detail needs a codebook that grows exponentially with precision. CompGS itself leaves positions unquantized because single-stage codes create geometric artifacts (CompGS). Quant VideoGen reports about 28.7 dB PSNR with multi-stage residual coding against roughly 20–21 dB for plain low-bit baselines, albeit on video activations rather than geometry. We include a single-stage codebook with the same total bits as a baseline.

Why not just express motion in a low-rank linear basis (principal components of the motion vectors) with scalar-quantized coefficients? This is also a real threat, because much motion in these scenes is smooth and nearly rigid. The file contains no evidence either way. We will run it with the same bit budget, and its natural "number of kept coefficients" plays the role of our depth. If it matches us, the right conclusion is that variable-length coding matters and the discreteness of the codes does not.

Why not just prune flickering Gaussians as 4DGS-1K does? Pruning cuts about 2 GB to 50 MB but leaves the survivors in continuous form, and it needs thresholds tuned per scene (1000+ FPS 4DGS). Pruning is complementary to our method. We will report ours with and without its pruning, to test whether the rate term already removes the same primitives.

Proposed Experiments

We evaluate on D-NeRF synthetic, Neu3D (Coffee Martini, Cook Spinach, Sear Steak), and NeRF-DS. Metrics are PSNR, SSIM, LPIPS, storage in MB, and rendering FPS. The headline figure is the rate-distortion curve against P-4DGS, CompGS++, and the two "why not" pipelines.

The first decisive check is whether the decomposition is real. On Neu3D we render only the depth-zero Gaussians and inspect them for foreground leakage. On D-NeRF, where the background is empty and the object moves, we measure the fraction of depth-zero Gaussians inside the moving object. We compare both against SDD-4DGS's thresholded weight. On KITTI-MOT we check whether depth-zero Gaussians cover the road and buildings while moving cars get deep codes. We report PSNR on past-frame reconstruction next to 4DGS-WAM's 27.63 dB (4DGS-WAM) as context only, since that method solves a different, prediction-oriented problem.

The second decisive check is drift. On the longest Neu3D sequences we plot PSNR against time within windows and across window boundaries. We compare our keyframe-anchored codes with a frame-to-frame variant of the same code, which tests whether ResQ's anchoring result carries over to 3D motion.

The third decisive check is codebook reuse. We measure how many distinct stage-one codes appear per window and how often codes recur across windows. If reuse is low, the temporal-redundancy argument is weak and most of the savings come from static Gaussians alone, a result the depth-zero-only baseline would expose directly.