Methods that rebuild a 3D scene from casual photos assume the scene holds still. When people walk through the shot, current systems decide which pixels to ignore by asking whether the pixel's color matches what the static 3D model predicts. On a blank wall, that question gives the wrong answer in both directions. A person in a white shirt standing in front of a white wall looks almost identical to the wall, so they go undetected and leave ghostly geometry behind. A glossy floor that shines differently from each viewpoint, or a passing shadow, looks like a mismatch, so the method cuts a hole in a surface that never moved. We propose asking a different question: does the shape seen in this frame agree with the static 3D model? A per-frame estimate of depth and surface orientation from a single-image predictor is reliable on exactly the flat, featureless surfaces where color fails. A moving object always sits in front of the static surface it hides, while a highlight or shadow changes color without changing shape.
The note turns this asymmetry into a detector with three outcomes, which are separate geometric and color tests on each pixel. A pixel whose observed shape disagrees with the scene, in the occluding direction, is a transient and is excluded from all losses. A pixel whose color disagrees but whose shape agrees is an appearance change, such as glare or a cast shadow. It is excluded from the color loss but still supervises geometry. Everything else is static. The claim is that this separation, and not merely adding shape priors as a smoothness regularizer, is what lets a single pipeline produce clean, hole-free geometry for low-texture interiors captured with people moving through them.
Core Claim
Prior distractor-removal work detects transients from appearance. RobustNeRF trims the highest color residuals over 16×16 patches. NeRF On-the-go predicts per-pixel uncertainty from DINOv2 features with an SSIM-based loss. 3DGS-HPC gates DINOv2 patch errors with a color mixture model. RobustSplat delays densification and bootstraps masks from coarse to fine. Each of these papers motivates the proposal with a failure it reports:
- NeRF On-the-go reports that specular, view-dependent surfaces are misread as distractors and hollowed out.
- 3DGS-HPC needs a dedicated calibration step just to stop blank walls being deleted.
- RobustSplat and Split4D both report moving shadows baked into the static background.
On the other side, static indoor methods supply the tool we reuse. MonoSDF supervises an implicit surface with single-image depth (after scale-and-shift alignment) and normals. DN-Splatter flattens 3D Gaussian Splatting primitives into oriented discs and aligns them to predicted normals with an edge-aware depth loss; 3D Gaussian Splatting represents a scene as many small colored ellipsoids rendered by projection. Both assume a static scene.
The closest analogue to our idea is in video generation rather than reconstruction. GeoVideo adds a motion-probability head that exempts moving pixels from a reprojection loss. That head is learned from 2D statistics and cannot tell a shadow from an object. Our detector is explicit, sign-aware and geometric. Our use of the "geometric inconsistency" direction is therefore a change of role: inconsistency between per-frame shape and the global model is treated as the measurement that identifies motion, rather than a defect to suppress.
Research Program
We build on a 3D Gaussian Splatting reconstruction with DN-Splatter-style disc primitives and normal and depth losses. We keep RobustSplat's delayed densification, because RobustSplat shows that disabling growth outright over-smooths the scene and that delaying it is the right compromise.
For each training frame $t$, we render the current static model's depth $\hat D_t$ and normals $\hat N_t$. We take a single-image prediction of depth $D^{m}_t$ and normals $N^{m}_t$ from the same off-the-shelf predictor family MonoSDF uses. We align the predicted depth with a per-frame scale $s_t$ and shift $b_t$, fit robustly only on pixels currently labeled static.
The geometric transient score at pixel $p$ is one-sided:
$$ g_t(p) = \frac{\max\big(0,\ \hat D_t(p) - (s_t D^{m}_t(p) + b_t)\big)}{\hat D_t(p)} \;+\; \gamma\,\big[1 - \hat N_t(p)\cdot N^{m}_t(p)\big]\,\mathbb{1}\big[\hat D_t(p) > s_t D^{m}_t(p) + b_t\big] $$
In this score, the first term measures how far the observed surface is in front of the modeled one, relative to the modeled depth. The second term adds normal disagreement only where the observation is in front. The weight $\gamma$ balances the two terms.
The one-sidedness is deliberate. Mirror-like reflections tend to make single-image depth predictors report surfaces behind the true one, and that error direction no longer triggers the detector. Scores are aggregated over superpixels rather than fixed 16×16 patches. This answers 3DGS-HPC's admitted over-masking of thin structures.
Separately, the color residual is passed through a mixture-model threshold, as in 3DGS-HPC, to flag appearance outliers. The training objective becomes
$$ \mathcal{L} = \sum_{t,p} \big(1 - m^{g}_t(p)\big)\Big[\big(1 - m^{a}_t(p)\big)\,\mathcal{L}_{rgb} + \lambda\,\big(\mathcal{L}_{depth} + \mathcal{L}_{normal}\big)\Big], $$
where $m^{g}_t$ is the thresholded geometric transient mask, $m^{a}_t$ is the appearance-outlier mask, and $\lambda$ weights the shape terms. Glare and shadow pixels therefore still pull the wall flat but do not imprint their color. Their color is supplied by other views. Masks are re-estimated as the static model sharpens, following RobustSplat's coarse-to-fine schedule.
The analysis component characterizes when single-image shape is trustworthy enough to act as a detector. We will measure detection precision against the predictor's own normal error, stratified by surface texture, specularity, and distance between the transient and the wall behind it. That last factor is the expected weak point: a hand resting flat on a wall barely changes depth.
Baselines
The naive reference point is plain 3D Gaussian Splatting trained on all pixels. This shows the full size of the distractor problem.
The strongest distractor-removal competitor is 3DGS-HPC. It is the only method in the file that explicitly targets preserving blank walls, and it reports a PSNR gain of 1.39 to 5.54 dB over plain 3D Gaussian Splatting on the RobustNeRF scenes. RobustSplat is equally relevant because it claims the best PSNR, SSIM and LPIPS on every On-the-go scene. However, no reported per-scene number was found on Lacuna for RobustSplat.
For reference on the On-the-go Patio scene, NeRF On-the-go reports 21.41 dB PSNR against RobustNeRF's 20.54 dB. On the RobustNeRF Robotics Lab scene, RobustNeRF reports 30.87 dB. On the geometry side, the strongest static method is DN-Splatter, which reports an F-score of 0.9243 on MuSHRoom against 0.5835 for plain 3D Gaussian Splatting.
All methods will be run from their public configurations on identical images and camera poses. Each gets the same iteration budget and the same single-image predictor wherever priors are used. Hyperparameter tuning gets an equal small grid per method on one held-out scene.
The decisive ablation is a "stacked" pipeline, RobustSplat masks plus DN-Splatter losses, which uses shape priors only as regularizers. We will compare it to our method with the geometric detector switched off, the one-sided test made two-sided, and the appearance-only branch removed.
Beating the baselines means two things. On scenes with distractors, we must improve surface F-score and normal accuracy over both 3DGS-HPC and the stacked pipeline on textureless and specular regions. We must also not lose more than a small margin in rendering PSNR on On-the-go and RobustNeRF.
Why not just…?
Why not just add DN-Splatter's normal and depth losses to RobustSplat or 3DGS-HPC? This is the real threat, and it is why the stacked pipeline is the central ablation. Stacking leaves detection appearance-based. The specular hollows that NeRF On-the-go reports and the camouflaged distractors would still be masked, or left unmasked, before any shape loss could act. A masked-out highlight on a floor would then get no geometric supervision at all. If the stacked pipeline matches us on the specular and camouflage subsets, the three-way separation adds nothing and the proposal reduces to a recipe. The stratified results will show this directly.
Why not just mask people with a pretrained segmenter and then run DN-Splatter on what remains? Split4D reports that tracker-dependent decomposition reaches only 0.592 mIoU on Multi-Human against 0.893 for motion-aware clustering. Semantic masks also miss non-semantic transients such as moved chairs and opened doors, which 3DGS-HPC and RobustNeRF note. We will still run a segmentation-mask-then-DN-Splatter baseline, because on scenes containing only people it may be competitive. If so, our contribution narrows to the shadow, glare and moved-object cases.
Why not just detect transients from disagreement between consecutive single-image depth maps, with no static model? Single-image depth predictions are relative and drift between frames; MonoSDF needs per-batch alignment for exactly this reason. Without a static model to compare against, there is no direction to the disagreement, so the in-front versus behind asymmetry that separates occluders from reflections is lost. Our two-sided ablation approximates this alternative.
Proposed Experiments
Controlled synthetic evaluation. Kubric is the synthetic generator already used by RobustNeRF and NeRF On-the-go. We will build rooms with textureless walls, glossy floors, moving lights that cast shadows, and distractors whose albedo is matched to the background. This gives exact geometry and exact transient labels. It lets us report transient-mask precision and recall separately for camouflaged, specular and shadow pixels, alongside Chamfer distance and normal error.
Real paired captures. We will record low-texture interiors twice, once empty and once with people walking through. We will reconstruct the empty pass with DN-Splatter as pseudo-reference geometry and score the cluttered-pass reconstructions against it. A laser-scanned subset in the style of ScanNet++ would be preferable where it can be obtained.
Rendering benchmarks. For comparability with prior work, we report PSNR, SSIM and LPIPS on On-the-go and RobustNeRF. We also report static geometry on clean MuSHRoom and Replica. Those clean runs check that the detector does not damage scenes with no distractors.
Decisive checks. The result that would validate the idea is that the gap to the stacked pipeline appears specifically on the camouflage and specular subsets and is near zero on textured, matte regions. That pattern is what the mechanism predicts. A uniform improvement everywhere would instead suggest a tuning artifact.
The result that would falsify it is detection precision collapsing wherever the single-image predictor's normal error exceeds the angular threshold. We will report that curve explicitly.
As a secondary check, we will feed our transient masks to a GeoVideo-style reprojection loss in place of its learned motion head on DL3DV-10K. We will then test whether geometry-derived exemptions raise the reported multi-view consistency score, which GeoVideo reports rising from 61.2 for CogVideoX-5B to 77.2 with its own head. This extension is optional and not required for the main claim.
References
- RobustNeRF: Ignoring Distractors with Robust Losses
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
- DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing
- GeoVideo: Introducing Geometric Regularization into Video Generation Model