Keep the Shadow, Fill the Void: Illumination-Aware Distractor Removal for Gaussian Splatting

Methods that rebuild a 3D scene from casual video assume the world holds still. When people or cars move through the capture, the leading fixes treat every pixel that disagrees across frames as an outlier and throw it away. That discards two different kinds of pixels as if they were one. The first kind shows a static surface that a moving object's cast shadow darkens. The surface is visible there, only dimmer, so the pixel can be kept if the darkening is modeled. The second kind shows the moving object itself, and behind it lies surface that may never appear in any frame. Only these pixels need to be invented by a generative model. Current pipelines drop both kinds, so they lose recoverable data and leave holes. Where a generative fill is attempted, it usually copies nearby shadow stains into the invented content.

We propose to split rejected pixels into three classes inside a 3D Gaussian Splatting reconstruction. 3D Gaussian Splatting represents a scene as millions of small colored, semi-transparent blobs that are rasterized into images. The three classes are static-and-lit, static-but-shadowed, and occluded. Shadowed pixels are explained by a per-frame multiplicative shadow term applied to the static render and kept as supervision. Only regions occluded in every frame go to a generative prior, and that prior fills them in shadow-free space. The fill is then lifted into 3D and must stay consistent under the parallax of the training cameras, meaning the apparent depth-dependent shift of points as the camera translates. The claim is testable. Compared with outlier rejection, the scheme should shrink the unobserved area, remove the dark stains baked into static surfaces, and produce fills that stay coherent as the camera moves.

Where the gap sits

Four lines of work each hold one piece of the problem. The distractor-free reconstruction methods motivate the proposal because they document the loss:

  • RobustNeRF trains only on the lowest-error half of pixels.
  • NeRF On-the-go predicts per-pixel uncertainty from DINOv2 features, which are general-purpose image features from a self-supervised vision model.
  • RobustSplat delays the step where Gaussians are split and cloned, and bootstraps transient masks from coarse to fine resolution.
  • 3DGS-HPC classifies 16×16 patches using a photometric mixture model combined with DINOv2 distances.

All four discard shadows as outliers. RobustSplat states that it does not handle illumination variation. 3DGS-HPC notes that permanently occluded regions become empty holes and names diffusion-based completion as future work. We adopt RobustSplat's delayed-growth schedule and 3DGS-HPC's patch statistics as our starting pipeline. What we change is what happens after a pixel is rejected.

The shadow-modeling line supplies the illumination mechanism. D²NeRF splits a monocular video into static and dynamic radiance fields and adds a shadow field, a multiplicative darkening on the static branch. This is the closest existing idea to our shadowed class. It lives inside a two-branch NeRF, it has no mechanism for never-observed regions, and it is not combined with the outlier statistics that make Gaussian splatting robust to clutter.

GI-GS takes a heavier route. It path-traces direct and indirect light through a buffer of per-pixel depth, normals and material values rasterized from Gaussians. It assumes a static, fully observed scene, and Split4D shows that without illumination modeling, removing objects leaves inconsistent shadows behind. GI-GS constrains our design as much as it inspires it. Full multi-bounce transport depends on clean geometry, and that is exactly what is missing near disocclusions. We therefore model only the first-order effect that dominates in these captures: a moving occluder blocks the main light.

The disocclusion line supplies the fill and its failure modes:

All three produce plausible 2D content, but none writes that content back into a 3D representation with physical shadowing. Finally, TransFill supplies the parallax tool we reuse. It fills holes from a reference view by splitting the scene into depth layers and warping each layer with its own homography, a 3×3 image-to-image transform. We use a version of this to lift a 2D fill across training views.

Core Claim

A large share of the pixels that distractor-free methods discard are recoverable shadowed observations. Keeping them, and filling only the truly unseen remainder in a de-shadowed space under a parallax-consistency constraint, should improve reconstruction in three separate regions. In shadow-swept regions it should remove stains. In never-observed regions it should produce fills that are consistent across views. In the region seen in every frame it should cost nothing.

Research Program

The shadow model is deliberately small. For frame $t$ and pixel $p$, the observed color is modeled as

$$C_t(p) \approx s_t(p)\,\hat{C}(p),$$

where $\hat{C}(p)$ is the static Gaussian render and $s_t(p)\in(0,1]$ is a per-frame shadow map. The shadow map is smooth, and it may only scale all color channels together, with at most a small learned bluish tint to account for skylight filling in the shadow.

We tell shadowed pixels apart from occluded ones with a single test. A shadow scales brightness but keeps the local texture edges and DINOv2 features of the underlying surface. An occluder replaces both. Pixels that the 3DGS-HPC patch statistics flag as outliers but that pass this test join the shadowed class and supervise the static scene through the equation above. Pixels that fail the test are marked occluded. Accumulated over all frames, the occluded mask defines the never-observed set $\Omega$.

For $\Omega$ we render the static scene from an anchor training view with the shadow term removed. We inpaint that de-shadowed image with an off-the-shelf diffusion inpainter, conditioned on its lit neighborhood. We then lift the fill into new Gaussians using depth-layered reprojection in the style of TransFill, one homography per depth cluster followed by a per-pixel refinement.

The lifted Gaussians are trained with a parallax-consistency loss. Rendered into every other training camera that sees $\Omega$ at a different depth offset, they must agree with independent inpaintings made from those views. The agreement is measured after aligning each view with the depth-layer warp, not by raw pixel difference. The loss is designed to reject fills that look right from one view and shear or float from the others. Once the fill is in 3D, its shadowing at each training frame is predicted by the shadow maps already estimated there. This gives a self-check: shadowed renders of the filled region should match whatever partial observations border it.

Baselines

The naive reference is vanilla 3D Gaussian Splatting trained on all pixels. This shows the size of the problem. 3DGS-HPC reports gains of 1.39 to 5.54 dB PSNR over native 3DGS on distractor benchmarks.

The strongest existing distractor-free methods are RobustSplat and 3DGS-HPC. We use both unmodified, which also serves as our "discard everything" condition. They are the right competitors because they share our renderer and schedule, so any difference comes from how rejected pixels are handled. On the RobustNeRF benchmark and on NeRF On-the-go we found no reported absolute number on Lacuna for either. RobustSplat reports only relative wins over SpotLessSplats, WildGaussians and T-3DGS. We will therefore rerun both from released code on identical splits and iteration counts. For reference, published anchors on these benchmarks are:

  • RobustNeRF: 30.87 dB PSNR versus 25.22 dB for MipNeRF360 on a distractor scene.
  • NeRF On-the-go: 21.41 versus 20.54 for RobustNeRF on Patio.

D²NeRF is the shadow-aware competitor. It reports 36.58 PSNR on the Kubric "Pillow" scene against 20.09 for NeuralDiff. We run it on the same monocular Kubric sequences.

The generative competitor is inpaint-then-reconstruct. We take the 3DGS-HPC masks, inpaint each masked training frame independently in RGB with the same diffusion inpainter we use, and retrain 3DGS on the inpainted frames.

All methods get the same camera poses, the same GPU budget of roughly one RTX 4090 at matched wall-clock time, and the same number of hyperparameter trials per method.

Three ablations isolate our ingredients. The first drops the shadowed class, so that everything flagged is discarded or filled. The second inpaints in RGB instead of de-shadowed space. The third removes the parallax-consistency loss.

Beating the baselines means the following. In the shadow-swept region, our method should be significantly better, with non-overlapping confidence across scenes. In the never-observed region, it should be better on both appearance and cross-view consistency. In the region seen in every frame, it should be no worse than 3DGS-HPC to within about 0.2 dB.

Why not just…?

Why not just mask dynamic objects with a segmentation model, inpaint each frame, and reconstruct? This is the real threat. On never-observed regions it may match us on single-view appearance. The file gives two reasons to expect it to fall short. 3DGS-HPC reports that semantic segmenters miss moving shadows, which then bake into the static scene. Per-frame 2D fills are also chosen independently, and the file's gap analysis expects 2D fills without 3D grounding to drift across frames, so they should disagree under parallax. The inpaint-then-reconstruct baseline tests this directly. If it matches us on stain removal and on cross-view consistency, the shadowed class adds nothing, and we would narrow the paper to the de-shadowed fill.

Why not just use D²NeRF's shadow field? It already models multiplicative shadows, so our first contribution is not a new representation. It is a new place to put that representation: inside a clutter-robust Gaussian pipeline, where it turns rejected pixels back into supervision. D²NeRF also leaves permanently occluded regions as voids. The head-to-head on Kubric will show whether its two-branch NeRF recovers shadow-swept albedo as well as our classification does. If it does, our advantage reduces to never-observed completion and speed.

Why not just run a diffusion enhancer on the rendered frames, as DiffusionHarmonizer does? An image-space repair does not change the 3D scene. Our consistency metric renders translated held-out views and checks agreement across them, so any view-to-view drift from post-hoc repair will show up there. The FreeOrbit4D and Beyond the Visible results argue that generation without a geometric scaffold loses geometric control. Beyond the Visible reports optical-flow motion accuracy of 0.02 for unconstrained Veo3.1 versus 0.72 with its scaffold. We will include a post-hoc image-space refiner as an additional baseline if a DiffusionHarmonizer checkpoint is available. If not, we will use a generic single-step image diffusion refiner, reported as such.

Why not just use a robust loss? RobustSplat shows that robust losses alone still let densification spawn floaters. A robust loss also discards shadowed pixels rather than using them, so it cannot recover anything. Our first ablation isolates exactly this difference.

Proposed Experiments

Evaluation uses three datasets, the Kubric synthetic scenes, the RobustNeRF benchmark and NeRF On-the-go, and splits every test image into four regions. The first is seen in every frame. The second is shadow-swept but visible. The third is occluded in some frames only. The fourth, $\Omega$, is never observed.

On Kubric all four regions come from ground truth. On the RobustNeRF benchmark, the paired clean captures show surfaces that the cluttered training frames never reveal, which gives a natural never-observed test set. For each region we report PSNR, SSIM and LPIPS. We add two diagnostics. The first is shadow bias: the mean luminance ratio between the static render and clean ground truth in shadow-swept regions, where a value below one indicates baked stains. The second is fill consistency: after aligning rendered test views with the depth-layer warp, the error between fills of $\Omega$ seen from views that translate away from the training path. The consistency metric is designed to expose fills that are coherent from one view but not from others. On NeRF On-the-go, which has no clean ground truth for $\Omega$, we report fill consistency and a blind preference study.

The decisive check has three parts. First, the shadowed class must rescue a measurable fraction of rejected pixels. We will report the fraction on each dataset, and the idea succeeds only if it is non-trivial. Second, keeping those pixels must lower shadow bias relative to RobustSplat and 3DGS-HPC. Third, de-shadowed inpainting plus the parallax loss must beat per-frame RGB inpainting on fill consistency without losing appearance scores in $\Omega$. If the second part fails, the shadow model is wrong. If the third fails while the second holds, the paper becomes a clutter-robust shadow-recovery method without its generative half.