Tracking Illumination Before Rejecting Outliers in Streaming Gaussian Splatting

When a camera records a real place over minutes or hours, two different things make pixels disagree with the 3D model being built. People and cars move through the frame. The light also changes: clouds pass, the sun drops, shadows sweep across walls. Current reconstruction systems that tolerate moving objects find them by looking for pixels the model renders badly. A lighting change also makes the model render badly, so these systems confuse the two. They erase real walls as if they were passers-by, or they bake shadows into the geometry. RobustSplat's authors state that their method "does not explicitly handle illumination changes" (RobustSplat). NeRF On-the-go mistakes specular static surfaces for distractors (NeRF On-the-go).

We propose to separate the two causes by how they behave over time. Illumination changes the whole image coherently and, most of the time, slowly. Moving objects are local and short-lived. The method keeps a small, low-dimensional model of the current lighting, mainly a color transform applied to the rendered static scene. It updates this model online with an algorithm that has guarantees for following a moving target, and only then decides which pixels belong to transient objects. Fitting a color transform to a fixed rendering is ordinary regression, which is a convex problem. That makes it one of the rare places in 3D reconstruction where online learning theory applies directly rather than as a metaphor.

Core Claim

In streaming 3D Gaussian Splatting (3DGS, a scene representation made of many small colored 3D blobs that are rendered by projecting them onto the image), the step that labels distractors should operate on residuals after illumination has been tracked online, not on raw photometric error. The illumination tracker should come from dynamic-regret online learning, meaning learning that competes against the best per-moment parameters rather than one fixed setting. It should run an ensemble over forgetting rates, because the speed of lighting change is unknown and changes over time.

The claim is falsifiable on three counts:

  • Fewer wrong erasures. Under controlled lighting drift, the approach should erase far fewer static pixels than distractor filters without tracking.
  • The ensemble matters. The gain should grow with the frequency of abrupt lighting changes. That is the regime where a single fixed forgetting rate, which is what Adam's moving averages amount to, should fail.
  • The temporal prior matters. The approach should beat a careful robust fit made independently for each frame, mainly at high occlusion ratios.

Mechanism and Relation to Prior Work

Let $\hat{I}_t(p)$ be the static 3DGS rendering at pixel $p$ for frame $t$, and let $I_t(p)$ be the observed color. The illumination state $a_t \in \mathbb{R}^d$ parameterizes a per-frame color transform $A(a_t)$. This is an affine map on color, optionally mixed with a few smooth spatial basis functions so it can represent gradual shading across the image. It deliberately does not represent sharp cast shadows. The per-frame loss is

$$f_t(a) = \sum_{p} w_t(p)\,\big\| A(a)\,\hat{I}_t(p) - I_t(p) \big\|^2 ,$$

where $w_t(p)\in[0,1]$ is the current inlier weight. With the rendering held fixed, $f_t$ is a convex quadratic in $a$. The tracker is judged by dynamic regret against the per-frame best transform $a_t^*$:

$$\sum_{t=1}^T f_t(a_t) - \sum_{t=1}^T f_t(a_t^*) .$$

The difficulty of this tracking problem is measured by the path length $P_T=\sum_t\|a_t^*-a_{t-1}^*\|$, the total distance the best lighting parameters travel over the sequence.

This is the setting of the discounted-to-dynamic reduction of Xie, Zhang and Zhao. For curved losses such as linear regression, it gives dynamic regret of $O(d\log T+\sqrt{dTP_T})$ without knowing $P_T$ in advance. The forgetting rate is unknown and is not constant across a sequence, so we aggregate discounted online-gradient experts over a geometric grid of discount factors using the meta-learner of Yang, Yang and Zhang. That method pays only a $\sqrt{\log T}$ factor over a perfectly tuned discount, and it has so far been tested only on the ijcnn1 dataset. Both papers supply the method; this proposal is their first use on a 3D-vision parameter stream.

The non-convex parts of the system are the Gaussians themselves and the inlier weights. For those we make no theoretical claim, and we say so in the paper. The guarantee covers only the illumination stream that the masking step consumes.

Distractor labeling then runs on the lighting-corrected residual $A(a_t)\hat{I}_t - I_t$. We use the patch-wise photometric mixture model and DINOv2 classifier of 3DGS-HPC unchanged, combined with the delayed densification of RobustSplat. That combination is the robust-geometry backbone.

Several lines of prior work motivate the proposal without solving it:

  • WildGaussians already uses affine color correction that can be baked into the Gaussians. However, it learns a free code per image offline, with no temporal structure. Its DINOv2 masking therefore cannot benefit from knowing the light at frame $t$ resembles frame $t-1$.
  • CogniMap3D handles motion in streams with flow-based cues.
  • VIPeR adapts place recognition to seasonal drift with heuristic memories.

Neither CogniMap3D nor VIPeR separates lighting from transients inside a radiance model. The constraining evidence comes from 3DGS-HPC's observation that semantic features miss moving shadows. That observation is why we keep sharp shadows on the transient side: they are local, and the low-dimensional transform cannot absorb them.

Research Program

We build a streaming 3DGS mapper in which frames arrive in order with known or SLAM-estimated poses. Geometry is updated incrementally. Every frame passes through three steps:

  1. The ensemble illumination tracker fits the lighting transform on the current weights.
  2. The masker recomputes the weights on the corrected residuals.
  3. One alternation is repeated before the geometry update.

The analysis has two strands. The first measures empirical dynamic regret of the tracker against $P_T$ computed from the true lighting schedule, to check that the theory's scaling shows up in practice. The second characterizes when temporal tracking helps beyond per-frame fitting. Our hypothesis is that the temporal prior matters most when transients cover a large fraction of the frame. In that case a per-frame fit has too few inlier pixels to pin down $a_t$, while the tracker's prediction from frames before $t$ can initialize the weights before the fit.

Benchmarks and Protocol

No benchmark in our sources combines temporally ordered capture, lighting drift and transient objects with ground truth for all three, so we construct one.

The controlled benchmark relights the 12 NeRF On-the-go sequences and the RobustNeRF scenes, including the Robotics Lab scenes with paired distractor-free ground truth. Relighting applies known illumination schedules to the frames in capture order. There are three schedule types: smooth drift, step changes, and mixed. These cover a range of path lengths. Exact transient masks come from the ground-truth pairs. Because the schedules are applied in color space, we also need a check that does not depend on our own synthetic lighting model. For that we add one family of schedules that includes spatially non-affine shading. It tells us whether the tracker degrades gracefully when its lighting model is wrong.

Real-data evaluation uses two sources:

  • Oxford RobotCar (Oxford Radar RobotCar), short temporally ordered segments with traffic and lighting change, evaluated on held-out frames.
  • Photo Tourism, run in its standard unordered offline mode. This checks that nothing breaks when the temporal prior is uninformative.

Metrics follow the established protocol. We report PSNR, SSIM and LPIPS on held-out frames, with the held-out frame's lighting estimated on half of that image, as in the in-the-wild NeRF and WildGaussians protocol. Mask quality has two parts: the false-erasure rate on static pixels and recall on true transients. Floaters are counted against ground-truth geometry. We also report empirical tracking regret as a function of $P_T$.

Baselines

The naive reference is native 3DGS trained in the same streaming order with no masking and no lighting model. It shows how much any method recovers.

The strongest existing competitor is the robust-geometry backbone itself: 3DGS-HPC masking with RobustSplat's delayed densification. On the original, un-relit data these are the best reported distractor methods in our sources. 3DGS-HPC reports PSNR gains of 1.39 to 5.54 dB over native 3DGS on the RobustNeRF data. RobustNeRF reports 30.87 dB against Mip-NeRF 360's 25.22 dB there. NeRF On-the-go reports 21.41 versus 20.54 PSNR on Patio. We found no reported result on Lacuna for any of these methods under illumination drift, which is exactly the regime we build. We therefore re-run them ourselves under the identical streaming schedule, iteration budget and hyperparameter search.

The second strongest competitor is WildGaussians, run streaming with its per-image appearance codes and DINOv2 masks. It is the existing method that already models appearance. We also run the offline batch version of WildGaussians and of our method as an upper reference.

Beating these baselines means two things on the relit benchmark. The first is a lower false-erasure rate on static pixels at matched transient recall. The second is higher held-out PSNR than both, on step-change and mixed schedules. On un-relit data, our method should show no loss relative to the backbone, since lighting tracking should then cost nothing.

Three ablations isolate the ingredients:

  • a single fixed discount factor in place of the ensemble, swept over the grid;
  • tracking with masking on raw rather than corrected residuals;
  • an independent robust per-frame fit of the same lighting transform, with no temporal sharing.

Why not just…?

Why not just use Adam on the appearance code, which already forgets the past through moving averages? Adam behaves like one discounted learner with a fixed forgetting rate (Xie et al.). A single fixed discount degrades when the true rate of change is mismatched, while the ensemble nearly matches the oracle-tuned rate (Yang et al.). That evidence comes from ijcnn1 rather than vision, so it is only suggestive here. The fixed-discount sweep settles the question.

  • If the best single rate is close to optimal across all schedules, the ensemble is unnecessary. We would then report that and keep the simpler tracker.
  • We expect the ensemble to win specifically on mixed schedules, where slow drift and sudden steps alternate.

Why not just fit the lighting transform robustly on each frame independently? This is the most serious threat. A trimmed least-squares fit of a handful of parameters, in the spirit of RobustNeRF, may be nearly as good when transients are small. We include it as an ablation and stratify all results by occlusion ratio, from 5% to 30% of pixels in the On-the-go sequences.

  • Our claim survives only if the temporal tracker's advantage grows with occlusion ratio.
  • If the per-frame robust fit matches the tracker everywhere, the contribution reduces to "correct illumination before masking." That is still useful, but the online-learning component would be shown to be unnecessary, and we would say so.

Why not just rely on DINOv2 semantic masks, which are largely lighting-invariant? WildGaussians and NeRF On-the-go already do this. 3DGS-HPC documents why it is not enough. Semantic features miss moving shadows, which are not objects. They are also fragile early in training, when they delete textureless static walls. The masking-on-raw-residual ablation, together with the WildGaussians baseline, measures directly how much of the false-erasure problem semantic features already solve under lighting drift.

Decisive Checks

The proposal stands or falls on four results from the relit benchmark:

  • False erasures. Static-pixel false-erasure rate versus the strongest backbone at matched transient recall. A clear reduction on step and mixed schedules is required.
  • Ensemble versus best fixed discount. The gap between the ensemble and the best fixed discount, as a function of how often steps occur.
  • Tracker versus per-frame fit. The gap between the tracker and the robust per-frame fit, as a function of occlusion ratio.
  • Regret scaling. Whether empirical tracking regret grows roughly like $\sqrt{TP_T}$, as the bound predicts, across schedules of different path length.

A null result on the third check would redirect the work toward the simpler per-frame design. A null result on the second would remove the ensemble. Either outcome is informative.