Scenes captured casually, with a phone walking around a courtyard, contain two kinds of change. Things move: a pedestrian crosses, a car parks. Light moves too: a cloud passes, a shadow sweeps across a wall. Real-time 3D reconstruction methods such as 3D Gaussian Splatting (3DGS) represent a scene as millions of small colored blobs that are drawn directly onto the screen, with no neural network evaluated per pixel. Their "distractor-free" variants detect moving objects by flagging image regions that the current 3D model fails to explain. These variants cannot tell the two kinds of change apart. A passing shadow and a passing person both look like large errors, so both get thrown away, or neither does. RobustSplat says in its limitations that it "does not explicitly handle illumination changes." NeRF On-the-go reports that glossy surfaces are mislabeled as distractors.
We propose a cheap physical test that separates the two cases. It borrows the central trick of albedo-invariant shape recovery: dividing two images of the same surface point cancels the surface's paint, or albedo, and leaves only lighting and geometry. If an observed patch differs from the model's prediction only because the light changed, the ratio between observation and prediction carries no surface texture. If a different object now covers the patch, the ratio carries the background texture, inverted. Checking whether the observed image still contains the predicted texture therefore gives a lighting-blind distractor detector. It runs as a per-patch image operation during training and adds nothing to rendering time. As a secondary question, we test whether this detector, unlike residual-based filters, leaves small steganographic watermark perturbations alone. Prior work has never examined whether hidden payloads survive distractor-robust training.
Core Claim
Under the Lambertian image model $I = \rho L (N\cdot S)$, where $\rho$ is albedo, $L$ light intensity, $N$ the surface normal and $S$ the light direction, the ratio of two co-registered observations cancels $\rho$. Comparing Images under Variable Illumination uses this to build a purely geometric descriptor. Unbiased Photometric Stereo for Colored Surfaces turns it into linear constraints on normals. Both assume perfect co-registration and a static scene, and both use the ratio to recover shape. We use the ratio for a different purpose: as a diagnostic of whether co-registration holds.
Let $R_j$ be the 3DGS model's rendering of training view $j$, and $I_j$ the captured image. Within a 16×16 patch $P$, we regress observed log-image gradients on predicted ones:
$$\beta_P = \arg\min_{\beta}\ \sum_{p\in P} w_p\,\big\|\nabla \log I_j(p) - \beta\,\nabla \log R_j(p)\big\|_1 ,$$
where $w_p$ down-weights dark pixels in which division is unstable. The two cases separate as follows. A global or smoothly varying lighting change multiplies the image by a slowly varying factor, so the texture gradients survive and $\beta_P \approx 1$. A cast shadow adds one boundary edge but keeps the interior texture, and the robust ($L_1$) fit ignores that edge. An occluder replaces the surface, so its gradients are statistically unrelated to the predicted background texture and $\beta_P \approx 0$.
The detector therefore asks "is the predicted texture still there?" Residual-based filters ask "is the error large?", and a shadow and a person can both produce a large error. We are not aware of any work that uses ratio invariance inside a distractor filter (Gap 1 of the file).
The test fails in a known place: patches with no texture, where $\nabla \log R_j$ is near zero and $\beta_P$ is undefined. There we fall back to the photometric statistics of 3DGS-HPC. Specular highlights add light rather than multiplying it, so they are a second known weakness. We report both regimes separately instead of hiding them.
Research Program
We build on two existing systems. From RobustSplat we take its delayed-densification schedule, because $R_j$ is meaningless before coarse static geometry exists. From 3DGS-HPC we take its patch-wise mask pipeline. We replace the distractor score with a combination of $\beta_P$ and HPC's score, gated by a texture-confidence term, and we make it lighting-aware. Patches classified as relit are kept in the loss. Their low-frequency ratio $I_j/R_j$ is absorbed by a per-image smooth multiplicative shading field, a coarse grid upsampled to the image, so that the shared Gaussian colors are not corrupted. Patches classified as occluded are masked, as in HPC.
The shading field exists only during training. The exported model is a plain 3DGS file that renders in an unmodified rasterizer. This matters because All That Glitters Is Not Gold and SecureGS show that auxiliary networks cut frame rates from over 100 FPS to about 45–48 FPS.
The steganography strand is deliberately narrow. 3DGS watermarking methods such as 3D-GSW and CompMarkGS are tested against JPEG compression, cropping and pruning, but never against distractor-robust training. Yet an owner capturing in the wild would want to embed the mark during that training. Watermark perturbations are small, high-frequency deviations from the training images. Squared-residual trimming in the style of RobustNeRF is built to discard exactly such deviations, which matches the "tension" noted in Gap 3. Our test is nearly blind to them, because a small additive perturbation leaves $\beta_P \approx 1$. This predicts that robustness under the steganographic trilemma, the three-way trade-off between payload capacity, visual fidelity and survival of the payload, depends on which filter is used. We test that prediction as a stress test, not as a new hiding scheme.
Baselines
The naive reference point is vanilla 3DGS with no distractor handling, and Mip-NeRF 360 trained on the same captures. RobustNeRF reports a distractor-scene PSNR of 25.22 dB for that Mip-NeRF 360 setting, rising to 30.87 dB with its trimmed loss. That pair calibrates how much is at stake.
The strongest competitor is 3DGS-HPC. It is the fastest mask-free distractor filter in the file (13–17 minutes, about 2 GB, on an RTX 4090), and it reports PSNR gains of 1.39 to 5.54 dB over native 3DGS. It also shares our patch granularity, so it is the cleanest point of comparison.
RobustSplat is the second strong competitor, because its densification control is orthogonal to the choice of mask. The file reports that it achieves the best PSNR, SSIM and LPIPS across scenes, but gives no reported number for RobustSplat on Lacuna. We will therefore rerun it ourselves.
NeRF On-the-go is included as the uncertainty-based reference. It reports 21.41 vs 20.54 PSNR against RobustNeRF on Patio.
All methods run on the same images, with the same initial point cloud from Structure-from-Motion, the same iteration count where applicable, and the same GPU. Each receives an equal threshold-tuning budget on one held-out scene per dataset.
The key ablation keeps everything in our pipeline but swaps $\beta_P$ for HPC's original score. A second ablation removes the shading field, so that relit patches are kept but nothing absorbs the lighting change.
Beating HPC means three things together: higher test PSNR on lighting-varying scenes, a static-patch false-rejection rate on relit patches at least halved, and no PSNR loss on the constant-lighting On-the-go and RobustNeRF scenes. Training time must stay within roughly 20% of HPC's.
Why not just…?
Why not just add a per-image appearance embedding to absorb lighting changes? This is the real threat. A global per-image color transform handles exposure and overall sky changes well. It cannot represent a cast shadow covering part of a wall. It also gives the optimizer a second way to explain away a large, uniformly colored distractor as "appearance." Our shading field is a constrained version of this idea, applied only to patches the ratio test has cleared. The decisive experiment adds a per-image affine color embedding to HPC and compares it on scenes with injected local shadows. If it matches us there, the ratio test is unnecessary, and we will report that.
Why not just segment distractors with SAM or DINOv2? 3DGS-HPC documents that semantic segmenters recognize a "person" but miss a moving shadow or a shifted chair, and that missed frames turn into permanent floaters. More fundamentally, a semantic model gives no principled answer to whether a shadow is a distractor. That question concerns image formation, not object categories. We still include a DINOv2-feature variant through HPC's own perceptual term, so the comparison is fair.
Why not just raise the residual threshold or enlarge RobustNeRF's trimming patches? Loosening the threshold keeps shadows only by also keeping faint distractors, because residual magnitude cannot separate the two. We sweep the threshold for HPC and RobustNeRF-style trimming and plot static-patch retention against distractor-pixel leakage. Our claim is that $\beta_P$ dominates this curve, not merely that it reaches one better operating point.
Why not just disable densification? RobustSplat already tested this and found it yields "overly smooth results." We therefore keep its delayed schedule rather than revisit the question.
Proposed Experiments
The first experiment is a pure false-positive check. DiLiGenT-MV, the multi-view photometric stereo benchmark used by PS-GS, photographs static objects under many light directions. Every pixel is static and only the lighting changes, so any patch a filter rejects is an error. We run every filter on per-light image sets and report the rejected fraction. Our prediction is that HPC and RobustSplat reject large fractions on the shadowed sides of the objects, while our test rejects few patches outside textureless regions. This experiment directly falsifies the core claim if $\beta_P$ fails here.
The second experiment uses controlled hybrids. We take Mip-NeRF 360 scenes, insert pasted distractors, and multiply selected images by smooth shading changes and projected shadow mattes. This gives ground-truth masks for both kinds of change. We report mask precision and recall separately for occluders and for lighting changes, plus test-view PSNR, SSIM and LPIPS.
The third experiment uses the RobustNeRF benchmark and the NeRF On-the-go dataset, where lighting is roughly constant. It checks that nothing is lost where the existing methods are already strong.
The fourth experiment uses Phototourism, which combines crowds with real lighting shifts and has no reported number found on Lacuna for HPC. Results there are qualitative and serve as a held-out measure of PSNR on held-out views.
For latency, we report training wall-clock time and memory against HPC's 13–17 minutes and 2 GB, and we confirm that the exported model's frame rate matches vanilla 3DGS.
For the steganography stress test, we embed a 3D-GSW-style bitstring during distractor-robust training on the On-the-go scenes under each of three filters: residual trimming, HPC, and ours. We report bit accuracy, cover PSNR, and the fraction of watermark-bearing patches each filter masks. 3D-GSW reports over 90% bit accuracy under JPEG, cropping and removal of 20% of Gaussians on clean captures; that is the target to retain. A null result here, where all filters preserve the mark equally, would simply close Gap 3 as a non-issue, and is worth reporting either way.
References
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- Comparing Images under Variable Illumination
- Unbiased Photometric Stereo for Colored Surfaces: A Variational Approach
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- All That Glitters Is Not Gold: Key-Secured 3D Secrets within 3D Gaussian Splatting
- SecureGS: Boosting the Security and Fidelity of 3D Gaussian Splatting Steganography
- 3D-GSW: 3D Gaussian Splatting for Robust Watermarking
- CompMarkGS: Robust Watermarking for Compressed 3D Gaussian Splatting
- RobustNeRF: Ignoring Distractors with Robust Losses
- PS-GS: Gaussian splatting for multi-view photometric stereo