Most 3D reconstruction methods assume the world holds still while the camera moves. When people walk through the shot or a hand moves an object, that assumption breaks, and current fixes guess which pixels to ignore. They throw out pixels with large errors, pixels a learned model flags as uncertain, or pixels that look like "people" to a segmentation network. All of these guesses share one blind spot. A genuinely static object glimpsed in only two or three frames looks just as inconsistent as a moving one, so it gets deleted.
We propose to explain a casual capture with two models competing for the pixels. The first is a static 3D scene built from 3D Gaussian splatting, which represents the scene as many small colored blobs rendered from any camera. The second is a flat, time-indexed neural video: a small network that takes a timestamp and outputs an image, which carries anything the 3D scene cannot. Both models pay for their parameters in compressed bits. Each pixel is then assigned to whichever model describes it more cheaply, instead of being judged by its error size or its semantic class. A static object costs little in 3D because camera motion, through parallax, explains how it shifts across frames. A moving person would need a new copy of 3D geometry at every position, but a neural video encodes it cheaply as a temporally smooth pattern. The output is a clean static 3D model plus a compact record of everything transient.
Core Claim
Distractor removal is better posed as a joint compression problem than as outlier detection. If the static 3D branch and the 2D video branch are charged the same units, entropy-coded bits, then fitting the capture at minimum total rate separates static from transient content. That split would be more faithful than residual-based or uncertainty-based rejection, and it would hold specifically on rarely observed static objects, which is where those methods fail.
The objective is
$$ \min_{\theta,\phi}\; \sum_t \big\| I_t - \big(\alpha_t\, V_\phi(t) + (1-\alpha_t)\, R_\theta(\pi_t)\big)\big\|^2 \;+\; \lambda\,\big(\mathcal{B}(\theta) + \mathcal{B}(\phi)\big). $$
The symbols are defined as follows:
- $I_t$ is the captured frame at time $t$ and $\pi_t$ is its known camera pose.
- $R_\theta$ renders the static Gaussian scene with parameters $\theta$.
- $V_\phi(t)$ is the neural video's color output and $\alpha_t$ is its per-pixel opacity, both produced by one network with parameters $\phi$.
- $\mathcal{B}(\cdot)$ is the estimated bit cost of the quantized parameters under a learned entropy model.
- $\lambda$ sets the rate–fidelity trade-off.
The key design choice is that $\mathcal{B}(\theta)$ is not zero. Standard 3D Gaussian splatting spawns new Gaussians to chase any large error, which RobustSplat identifies as the source of floating artifacts. Here those Gaussians cost bits, so fitting a moving object in 3D becomes expensive. Nothing forbids it.
The break-even logic is intuitive rather than derived. A static object whose image-plane motion matches camera parallax costs roughly a fixed number of bits in 3D, however many frames see it. The video branch must pay for every parallax-induced change in its appearance. We therefore expect rare static objects to land in 3D whenever the camera baseline across their few observations is non-trivial. We also expect the method to be genuinely ambiguous when the camera barely moves, and we will report that regime rather than hide it.
Relation to Prior Work
The distractor-free reconstruction literature motivates the problem and supplies the baselines. RobustNeRF treats distractors as outliers and trims the highest-error half of pixels, aggregated over 16×16 patches. NeRF On-the-go predicts per-pixel uncertainty from DINOv2 features with a structural-similarity loss. RobustSplat delays Gaussian densification and bootstraps transient masks coarse-to-fine. 3DGS-HPC classifies patches by combining a mixture model over color errors with DINOv2 features, and reports that segmentation masks from SAM are fragile for this task. All four decide per pixel or per patch using error magnitude or appearance. That is exactly why the research file lists the "rare object" failure and the specular-highlight failure as open. Our arbiter is description length, which depends on geometric explainability rather than error size.
Explicit static/dynamic decompositions in 3D are the closest existing work. DeGauss pairs static background Gaussians with deformed foreground Gaussians blended by a soft mask. SUDS splits a city into static, dynamic, and far-field hash grids, using DINO features and optical flow. 3D-Aware Video Generation couples a static background field with a dynamic foreground field in a generator. These methods constrain our design in two ways. First, all of them place the transient branch in 3D, and DeGauss's reported weakness is that its deformation field fails to initialize under sparse or monocular capture. Second, none of them charges either branch a rate, so the split is governed by architecture and heuristics. We move the transient branch into the image plane, where it needs no geometry, and let a shared bit budget make the split.
The neural video representation line supplies the transient branch and the rate machinery:
- NeRV maps a timestamp to a whole frame through convolutional upsampling blocks.
- DS-NeRV splits a video into sparse static codes and dense dynamic codes fused by attention, without supervision, reaching 31.2 PSNR at 0.35M parameters on Big Buck Bunny.
- GSVC entropy-codes space-time Gaussians for video compression.
- GaRV represents video as slices of (x, y, t) Gaussians.
GaRV itself is candid about what these methods lack: they treat time as a third flat axis and cannot tell camera motion from object motion. Under a moving camera, DS-NeRV's "static codes" do not correspond to physically static things at all. We turn that weakness into a division of labor. The 3D branch handles everything projective geometry explains, and the video branch is left with only what geometry cannot explain.
Research Program
We will build the static branch on standard 3D Gaussian splatting with poses taken from the benchmarks. Each Gaussian's parameters are quantized, and their cost is estimated with a learned factorized entropy model of the kind GSVC uses, so that $\mathcal{B}(\theta)$ is differentiable.
The video branch is a DS-NeRV-style network with dense, interpolated dynamic codes and a small convolutional decoder. We modify it to output an opacity channel. Its codes and weights are entropy-coded in the same way. The shared, bit-denominated penalty is what makes the two costs comparable.
We keep RobustSplat's delayed densification as a warm start. The 3D branch is then initialized from content that persists early in training, and the rate term governs every later growth decision.
The analysis program has three parts. First, we sweep $\lambda$ and trace, for each ground-truth object class, when it flips from the video branch to the 3D branch. The classes are moving distractors, rare static objects, and specular surfaces. Second, we measure how that flip point depends on camera baseline, which tests the parallax argument directly. Third, we check whether the flip point is stable enough that a single $\lambda$ transfers across scenes. If it does not, the method needs per-scene tuning, and that would weaken the practical claim.
Baselines
The naive reference is vanilla 3D Gaussian splatting trained on all pixels. It shows how much damage transients do and anchors the "improvement over 3DGS" range that 3DGS-HPC reports, 1.39 to 5.54 dB PSNR on the RobustNeRF benchmark.
We also include RobustNeRF itself. Its reported result on the Robotics Lab scene is 30.87 dB PSNR, against 25.22 dB for Mip-NeRF 360 (RobustNeRF). NeRF On-the-go reports 21.41 dB on the Patio scene, against 20.54 dB for RobustNeRF (NeRF On-the-go).
The strongest competitors are RobustSplat and 3DGS-HPC. They are the most recent Gaussian-based distractor-free methods, and both are evaluated on the RobustNeRF and On-the-go benchmarks. The file reports no isolated per-scene numbers for either: no reported result found on Lacuna for RobustSplat per-scene PSNR, or for 3DGS-HPC per-scene PSNR. We will therefore rerun both from their released settings on identical splits, poses, and resolutions. We will use the same Gaussian renderer and training schedule, so any gap comes from the decomposition rule. On the egocentric benchmarks, DeGauss is the strongest 3D-decomposition competitor. The file reports only its 157 FPS rendering speed and a qualitative LPIPS advantage, so it too will be rerun.
Beating these methods means higher held-out-view PSNR and lower LPIPS on average across the RobustNeRF and On-the-go scenes. On the rare-object subset described below, it means a clearly higher recall of static geometry at equal transient-removal precision.
Two ablations isolate the key ingredient. The first removes the 3D rate term $\mathcal{B}(\theta)$ and keeps the video branch with its own rate. This tests whether charging the static branch is what prevents floaters. The second replaces the video branch's bit cost with a plain L1 sparsity penalty on $\alpha_t$. This tests whether temporally coherent coding, rather than mere area minimization, is what matters. Every configuration gets the same tuning budget: a fixed grid of five penalty weights selected on one held-out validation scene.
Why not just…?
Why not just use a robust loss or uncertainty mask, as RobustSplat and 3DGS-HPC do? These methods work well when the "high error means transient" premise holds, and on scenes without rarely seen static objects they may match us. The research file flags their failure on objects seen in one or two views, and on specular surfaces misread as dynamic. Our controlled rare-object subset tests exactly that. If the robust methods recover rare static objects equally well, the core claim fails.
Why not just put the transient branch in 3D, like DeGauss or SUDS? With dense multi-view coverage, a 3D deformation branch may be as good or better, because it also yields novel views of the moving content. We do not claim otherwise. DeGauss's stated weakness is that its deformation field fails to initialize under sparse or monocular capture, and a 2D video branch has no geometry to initialize. We will compare on the egocentric sequences and on subsampled-view versions of On-the-go. We expect our advantage to grow as views thin out and to vanish or reverse with dense views.
Why not just add an unpenalized or sparsity-penalized per-frame residual image? This is the most serious threat, because it is our method minus the idea. An unpenalized residual absorbs everything, including the static scene. An area penalty charges a rare static object and a moving person the same, per pixel. The rate formulation charges the video branch for temporal change and charges the 3D branch for geometry. We think that difference is what separates parallax-consistent content from the rest. The L1 ablation settles this question directly. If it matches the full method on the rare-object subset, we would report the simpler method as the contribution.
Why not just segment people with a pretrained segmenter? 3DGS-HPC documents that SAM-style masks mislabel moving shadows and shifted furniture and are unreliable on noisy early renders. Class-based masks also cannot tell a parked object from a moved one. We include a segmenter-masked 3DGS baseline for completeness.
Proposed Experiments
We evaluate novel-view synthesis on the RobustNeRF benchmark, including its synthetic Kubric scenes and Robotics Lab. We also evaluate on the twelve On-the-go sequences and on egocentric sequences from EPIC-Fields and Hot3D as used by DeGauss. Metrics are PSNR, SSIM, and LPIPS on held-out clean views.
The decisive test is a rare-object subset. We will render Kubric-style synthetic captures in which some static objects are visible in only two to four frames, across a controlled range of camera baselines. Other objects move, and some surfaces are specular. Ground-truth masks let us measure static-geometry recall and transient-removal precision for every method.
A secondary check tests whether the decomposition is useful beyond cleaning. We ask whether the combined static-plus-video representation reconstructs the original capture at fewer bits than coding the video alone with DS-NeRV or GaRV, which we will run ourselves on the same sequences since the file reports no numbers for moving-camera captures. That result would indicate the split reflects real structure in the data rather than an arbitrary partition.
The proposal is falsified if either of two things happens: the L1 ablation matches the full method on the rare-object subset, or the flip point along the camera baseline shows no dependence on parallax.
References
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- RobustNeRF: Ignoring Distractors with Robust Losses
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- DeGauss: Dynamic-Static Decomposition with Gaussian Splatting for Distractor-free 3D Reconstruction
- SUDS: Scalable Urban Dynamic Scenes
- 3D-Aware Video Generation
- NeRV: Neural Representations for Videos
- DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes
- An Exploration with Entropy Constrained 3D Gaussians for 2D Video Compression
- Gaussian Representations for Video