Geometry Disagreement, Not Color Disagreement: Telling Moving Objects from Reflections in Casual 3D Capture

When you reconstruct a 3D scene from casual photos, anything that looks different across photos gets treated as a problem. Pedestrians and moved chairs are real problems, because they are not part of the static scene. A reflection on a glass door, a highlight on polished metal, or a moving shadow are different: the underlying surface is still there and still in the same place. Current methods that clean up moving objects decide what to discard by asking whether a pixel's color, or its deep-network features, disagree with the rest of the reconstruction. On specular and reflective surfaces that question gives the wrong answer. Both NeRF On-the-go and RobustSplat report that they fail under strong lighting change and on reflective surfaces, and they erase real static structure as a result.

We propose to ask a different question: does the shape disagree? A person walking through the frame changes the depth and surface orientation seen at that pixel. A highlight or a shadow changes only the color. We would estimate per-pixel depth and surface normals from each photo, compare them with the depth and normals rendered from the current static reconstruction, and discard a pixel only when the geometry disagrees. Pixels whose color disagrees but whose geometry agrees would be explained by an appearance term, and they would not be allowed to spawn new geometry. The synthetic-to-real literature matters for one specific step. Monocular geometry predictors are themselves unreliable on glass and mirrors, and real sensors cannot supply ground-truth depth for such surfaces. A renderer can. We would fine-tune the geometry predictor on synthetic reflective scenes. This follows MarrNet's finding that normal and depth maps rendered from CAD look like those estimated from real images, which makes them a bridge between domains that is largely free of lighting and texture effects.

Core Claim

The method builds on 3D Gaussian Splatting, which represents a scene as many small colored 3D blobs fitted to the photos. Our claim is that a geometric residual separates physical transients from view-dependent and illumination effects much better than a photometric or semantic residual can. The geometric residual is the disagreement between the depth and normals predicted from a photo and those rendered from the static blobs. Using it to gate the loss and the blob-spawning step ("densification") should preserve reflective static structure that current distractor-removal methods delete, without losing their ability to remove real moving objects. The second claim is that the geometry predictor only works for this purpose after fine-tuning on synthetic scenes with exact geometry for reflective and transparent materials. A predictor trained on real data alone will "see" the reflected world inside a mirror, and the method would then flag the mirror as a moving object.

For each training pixel $p$ in image $i$ we compute a geometric residual

$$g_i(p) = \lambda_d \,\big|\log \hat D_i(p) - \log D_i(p)\big| + \lambda_n \big(1 - \hat N_i(p)^\top N_i(p)\big),$$

where $\hat D_i, \hat N_i$ are the depth and unit normal predicted from the image, $D_i, N_i$ are those rendered from the current static reconstruction, and $\lambda_d, \lambda_n$ weight the two terms. The depth comparison is done after a per-image scale-and-shift alignment, because monocular depth is only defined up to scale. A pixel is treated as transient when $g_i(p)$ is high. When $g_i(p)$ is low but the color residual is high, the pixel is fitted by a per-image, spatially smooth appearance correction plus the blobs' view-dependent color. It contributes to the photometric loss, but its gradients are excluded from the rule that decides where to spawn new blobs.

Relation to Prior Work

Existing distractor-removal methods fall into two groups. The first rejects outliers on color alone: RobustNeRF keeps only the lowest-error half of pixels, and ForestSplats builds masks from superpixels plus photometric error, with no foundation model involved. The second rejects outliers using semantic features from large pretrained vision models: SpotlessSplats uses Stable Diffusion features, NeRF On-the-go and T-3DGS use DINOv2-based uncertainty (T-3DGS adds SAM2 masks), and 3DGS-HPC calibrates DINOv2 errors with a photometric mixture model over image patches. Both groups respond to appearance, so none can separate "the color changed because the surface is shiny" from "the color changed because something is in front of it."

These papers play different roles for us. NeRF On-the-go and RobustSplat motivate the proposal, because they report the specular failure. 3DGS-HPC constrains it. It documents semantic mismatch, where foundation models miss shadows and moved furniture, and it documents the rare object problem. Our geometric cue has the same weakness on objects seen in only a few views, so we will measure that failure rather than claim to fix it. RobustSplat also supplies a method we reuse: delayed blob growth. Its finding that disabling densification entirely over-smooths the scene tells us to gate densification rather than switch it off.

MarrNet and Pretrain, Self-train, Distill supply the transfer argument. Intermediate geometric maps cross from synthetic to real far better than raw appearance does, and synthetic geometry acts as an anchor against degenerate solutions. Unlike MarrNet, we do not reconstruct objects from a single image. We use synthetic-trained geometry only as a per-pixel judge inside a multi-view optimization, which means errors from the single-view prediction are checked against the reconstruction rather than propagated into it. Robust Category-Level 3D Pose Estimation from Diffusion-Enhanced Synthetic Data supplies a second tool. We would use its idea of placing CAD content in varied, counter-intuitive contexts, adapted here to vary materials, so the geometry predictor cannot learn "shiny means background."

The closest works connecting both directions (Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules, USplat4D, RobustNeRF's Kubric scenes) use synthetic data only as a test bed. None transfers synthetic supervision into the real-scene optimization.

Research Program

The work has three parts.

The first part builds a synthetic training set of indoor and outdoor scenes rendered with a physically based renderer. The scenes contain mirrors, glass, glossy metal, water, moving lights and cast shadows, and they are rendered with exact depth and normals of the physical surface, not the reflected content. We fine-tune an off-the-shelf monocular depth-and-normal predictor on these scenes, mixing in its original training data to avoid forgetting. Because the scenes are procedural, we can hold out entire material classes to test whether the predictor generalizes across materials or merely memorizes them.

The second part integrates the residual $g$ into a Gaussian Splatting pipeline built on RobustSplat's delayed-growth schedule. Pixel classification uses a fixed rule on $g$ with a patch-level aggregation borrowed from RobustNeRF and 3DGS-HPC. We deliberately avoid a learned classifier at first, so the contribution is the cue itself. Pixels with high color residual and low $g$ are absorbed by a low-frequency per-image appearance field and excluded from densification. That exclusion is how moving shadows are prevented from producing floaters.

The third part is analysis. We will characterize where the cue fails. We expect three failure modes: flat distractors lying on surfaces (a sheet of paper on a table has almost no geometric signature), rare objects, and early optimization, when rendered geometry is still poor. For the last case we will study a schedule that trusts the photometric cue early and the geometric cue once rendered depth stabilizes.

Baselines

The naive reference is vanilla 3D Gaussian Splatting with no distractor handling. It sets the floor, and every method should beat it by the margins already reported: 3DGS-HPC reports PSNR gains of 1.39 to 5.54 dB over native 3DGS on the RobustNeRF benchmark.

The strongest existing competitor is RobustSplat, which reports the best results across all scenes against SpotLessSplats, WildGaussians and T-3DGS on RobustNeRF and NeRF On-the-go. It is also the most direct competitor, because its densification-aware design addresses the same floater mechanism we target. We found no reported per-scene result on Lacuna for RobustSplat, so we will rerun its released configuration on every scene, including our specular subsets. We will also build our method on top of RobustSplat, so that the only difference is the geometric gate.

We will also run SpotlessSplats (SLS-mlp), which reports 22.81 dB PSNR on the On-the-go Fountain scene against 21.70 dB for 3DGS, as the representative semantic-feature method. We will run T-3DGS, which reports 28.08 dB against 24.86 dB for SpotLessSplats on its semi-transient dataset, as the strongest DINOv2-plus-SAM2 method. We will run 3DGS-HPC as the calibrated patch-classification method. All methods get the same image sets, the same camera poses, a matched iteration budget, and the same hyperparameter-search budget on one held-out scene per dataset.

Three ablations isolate the key ingredients. The first replaces $g$ with the photometric residual inside our otherwise identical pipeline. The second replaces the synthetic-fine-tuned geometry predictor with the same predictor before fine-tuning. The third removes the densification exclusion for appearance-only pixels.

Beating RobustSplat means three things together. On specular-region metrics, masked PSNR must improve by a margin larger than the across-seed standard deviation, and the false-discard rate must drop substantially. Full-image PSNR, SSIM and LPIPS on the standard benchmarks must be no worse. Transient recall must not fall.

Why not just…?

Why not just use an off-the-shelf monocular depth predictor, with no synthetic fine-tuning? This is the most serious threat. If it works as well, the synthetic-to-real part of the proposal is unnecessary and the contribution shrinks to the geometric gate alone. We expect it to fail specifically on mirrors and glass, where real-trained predictors tend to report the depth of the reflected content, which is exactly what our gate would misread as a transient. The second ablation settles this on the reflective subset. If the unfine-tuned predictor matches the fine-tuned one there, we will report that and drop the synthetic training claim.

Why not just add per-image appearance embeddings, as in-the-wild methods such as WildGaussians do? Per-image embeddings absorb global exposure and weather changes. They do not tell a local highlight apart from a local occluder, so they cannot decide which pixels to discard. The photometric-residual ablation, which keeps the appearance field but removes $g$, measures how far appearance modeling alone gets.

Why not just mask with SAM2 or DINOv2 uncertainty? 3DGS-HPC documents that these features miss shadows and moved objects and are fragile early in optimization, and NeRF On-the-go reports that its DINOv2 uncertainty flags specular surfaces. The T-3DGS and SpotlessSplats baselines test this directly on our specular subsets.

Why not just turn off densification or rely on a robust loss? RobustSplat shows that disabling densification over-smooths fine detail. RobustNeRF's trimmed loss discards valid pixels even on clean views and breaks when a distractor matches the background color. Neither addresses the specular case, because a highlight produces a large residual in both.

Proposed Experiments

We evaluate on four datasets. The RobustNeRF benchmark provides paired clean and cluttered captures; its reported numbers include 30.87 dB for RobustNeRF against 25.22 dB for Mip-NeRF 360 on one scene. The NeRF On-the-go dataset has 12 casual captures with 5–30% occlusion, and NeRF On-the-go reports 21.41 dB on Patio against RobustNeRF's 20.54 dB. We also use Photo Tourism and the T-3DGS semi-transient dataset.

We add two new resources. One is a held-out synthetic test set in Kubric style with reflective materials, moving objects and moving shadows, for which ground-truth transient masks exist. The other is a set of manual annotations of static specular and glass regions in the real scenes. We report full-image PSNR, SSIM and LPIPS. We also report PSNR restricted to annotated specular regions, the false-discard rate (the fraction of annotated static specular pixels classified as transient), and transient recall. On RobustNeRF, ground-truth transient masks come from differencing the clean and cluttered captures.

Four checks decide whether the idea holds. First, on the synthetic test set, the geometric residual must separate transient from specular pixels with clearly higher area under the ROC curve than the photometric or DINOv2 residuals. If it does not, the core claim fails. Second, on real specular regions, false discards must fall compared with RobustSplat while transient recall stays level. Third, the synthetic fine-tuning ablation must show its gain concentrated on mirror and glass pixels, which is the test of the synthetic-to-real claim. Fourth, a stress test with flat distractors and rare static objects will show where the method breaks. We will report that result regardless of outcome.