Telling Moving Objects Apart from Moving Light in Uncalibrated Photometric Reconstruction

Photometric stereo recovers fine surface detail by watching how an object's brightness changes as the light moves. Methods that reconstruct scenes from many photos make the opposite assumption: they treat any change in brightness between frames as a sign that something in the scene moved, and they throw it away. Each family therefore breaks the other's premise. Recovering shape from casual handheld capture, where the light is uncontrolled and people or tools pass through the frame, needs both at once. Today, uncalibrated multi-view photometric stereo collapses when objects move, and methods that remove transient objects collapse when the lighting changes. RobustSplat states this second failure outright: its error spikes from changing light cannot be told apart from error spikes caused by moving objects.

The proposal rests on one observation. When the light changes but the scene does not, every pixel's new brightness must be explained by the same handful of global lighting numbers, applied to normals and albedo that the static surface already has. A moving object cannot be explained that way, because it puts new geometry and new color at specific pixels. So we fit a very low-dimensional, physically constrained lighting model to each frame against the current static surface. Whatever that fit cannot explain is the motion signal. That leftover error then decides which pixels are allowed to shape the geometry, and which are allowed to trigger refinement of a memory-bounded volumetric surface. In short, the lighting model is kept deliberately too weak to absorb a moving object, and that weakness is the detector.

Core Claim

A per-frame lighting fit with roughly nine parameters is rigidly coupled to a shared static signed distance field (a 3D function whose zero level marks the surface). The error left over after that fit separates dynamic objects from illumination change well enough to support three things together: uncalibrated photometric refinement, transient removal, and coarse-to-fine volumetric refinement in scenes where both the light and the contents change. Flexible per-image appearance codes and 2D segmentation masks cannot do this, because they either absorb transients or ignore lighting physics.

Concretely, let $I_t(p)$ be the observed intensity at pixel $p$ in frame $t$. Let $\mathbf{x}_p$ be the surface point hit by that pixel's ray in the current static field, $\rho(\mathbf{x}_p)$ its albedo, $\mathbf{n}(\mathbf{x}_p)$ its normal, $\mathbf{s}(\cdot)$ the second-order spherical-harmonic basis (nine smooth angular functions), $v_t(p)$ a visibility term ray-traced through the field, and $\boldsymbol{\ell}_t$ the unknown nine-dimensional lighting of frame $t$. The lighting-explained residual is

$$r_t(p) = I_t(p) - v_t(p)\,\rho(\mathbf{x}_p)\,\mathbf{s}(\mathbf{n}(\mathbf{x}_p))^\top \hat{\boldsymbol{\ell}}_t, \qquad \hat{\boldsymbol{\ell}}_t = \arg\min_{\boldsymbol{\ell}} \sum_p \psi\big(I_t(p) - v_t(p)\,\rho\,\mathbf{s}(\mathbf{n})^\top \boldsymbol{\ell}\big),$$

where $\psi$ is a robust penalty. With nine parameters fit over hundreds of thousands of pixels, $\hat{\boldsymbol{\ell}}_t$ cannot bend to fit a local intruder. The normalized $|r_t|$ is therefore the input to the transient mask, not raw photometric error.

Position Relative to Prior Work

Two lines of work each solve half of the problem.

The first supplies our method. High-Quality RGB-D Reconstruction via Multi-View Uncalibrated Photometric Stereo and Gradient-SDF jointly optimizes poses, spherical-harmonic or point lighting, and a voxel distance field that stores gradients, so that it can project onto the true surface and subdivide only where geometry varies. It is the representation we build on, but it assumes a static scene. A Differential Volumetric Approach to Multi-View Photometric Stereo supplies the ray-traced cast-shadow visibility inside an octree. It also constrains our design: it uses calibrated LEDs, and its global linear system reaches 150 GB of RAM. We therefore avoid global solves and use stochastic first-order optimization over a narrow band of voxels around the surface.

The second line motivates the gap. RobustSplat delays the growth of Gaussian primitives (small 3D blobs used to represent the scene) until the background converges, and it bootstraps transient masks from coarse to fine image scales. We reuse both ideas: delayed refinement of the voxel band, and coarse-scale mask bootstrapping. The difference is that its masks are computed on raw photometric error, which is exactly what fails under changing light. MonoRec flags moving objects from inconsistency in a multi-frame cost volume. It would likewise flag a static wall whose shading changed.

PS-GS and Neural Multi-View Self-Calibrated Photometric Stereo handle uncalibrated or partially known lighting for isolated static objects. Single-shot color multiplexing, as in Near-Light Color Photometric Stereo, sidesteps motion only by assuming a single homogeneous material. We assume neither.

Research Program

The system has three stages that alternate.

Bootstrap. A coarse static field is initialized from sensor depth where available, as in the Gradient-SDF work, or otherwise from a temporal-median, low-resolution multi-view fit. Lighting is fit per frame at a coarse image scale, where both small transients and fine shading detail are suppressed.

Residual-gated geometry. Pixels with high normalized residual are down-weighted in the photometric and depth losses. They are also excluded from the statistic that decides which voxels to subdivide. This mirrors delayed growth, but the gate is now lighting-compensated, which addresses the seed-sensitivity problem that coarse-to-fine schemes share (Gap 4 in the file).

Refinement. The narrow-band voxels are subdivided. Albedo and normals sharpen. Lighting is refit, and masks are refined across image scales.

Volumetric context enters through the distance field itself: an Eikonal constraint, which keeps the field a true distance function, and the ray-traced visibility. Pixels shadowed by static geometry under a new light are thus predicted rather than misread as motion. Shadows cast by the moving objects themselves onto the static background are genuinely caused by dynamics, and our model treats them as transient. We will measure how much background this sacrifices rather than claim to model it.

Memory is a hard constraint. All runs must fit on a single 24 GB GPU. We report peak memory alongside PS-GS's 6.5 GB figure.

The main analysis characterizes when the residual test is identifiable. It degrades when a moving object's shading happens to lie in the span of the spherical-harmonic basis, for example a large, smooth, uniformly colored intruder. It also degrades when the light changes in ways outside that span: near-field point lights and sharp specular highlights. We will sweep intruder size, texture, and light distance on synthetic data to map where the test breaks.

Baselines

The naive reference is uncalibrated Gradient-SDF photometric reconstruction (High-Quality RGB-D Reconstruction via Multi-View Uncalibrated Photometric Stereo and Gradient-SDF) run unchanged on the corrupted sequences. It shows how badly ignoring dynamics hurts. Beating it is necessary but not interesting.

The strongest existing competitor on the transient side is RobustSplat (RobustSplat), because it is the best documented method on the On-the-Go and RobustNeRF distractor benchmarks. The file reports only that it leads on PSNR, SSIM, and LPIPS there; no reported result found on Lacuna for its exact per-scene numbers. We run it in two forms. The first is as published. The second adds per-image appearance embeddings, giving it a fair chance to absorb lighting. Both forms use the same frames, poses, and wall-clock budget. Geometry is extracted for comparison via depth rendering and fusion.

The strongest method on the photometric side is PS-GS (PS-GS). On clean DiLiGenT-MV it reports a normal mean angular error of 3.68° on BEAR, in 0.7 hours with 6.5 GB. As further reference points on clean static data, Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset reports a mean shape error of 0.52 mm, and Uncertainty-Aware Deep Multi-View Photometric Stereo reports an average F-score of 0.935. Both are calibrated, so they are upper references, not competitors. PS-GS is run on the corrupted sequences with its released settings and the same budget.

Two pipeline baselines are given the best available 2D masks and then passed to Gradient-SDF UPS. The first uses semantic segmentation masks. The second uses ground-truth masks, an oracle that bounds what masking alone can achieve.

The ablation that isolates the key ingredient replaces our residual with raw photometric error inside the otherwise identical system. A second ablation replaces the nine-parameter lighting with a per-frame lighting MLP of increasing capacity, to test whether low capacity is really what makes detection work.

Beating the competitors means three things on corrupted sequences: lower Chamfer distance and normal error than every non-oracle baseline, closing at least half the gap to the ground-truth-mask oracle, and staying within the memory budget.

Why not just…?

Why not just give RobustSplat per-image appearance embeddings so it absorbs lighting? This is the most serious threat. An unconstrained per-image code can explain shading changes, but it can equally explain a local intruder. It also never yields normals or albedo, so it cannot use lighting change as a cue for shape. Our capacity ablation directly tests the claim that the lighting model's rigidity, not the architecture around it, is what separates motion from light. If the appearance-embedding variant matches our transient masks, we will say so, and the contribution narrows to the photometric shape refinement.

Why not just mask dynamics with a 2D segmentation network and run uncalibrated photometric stereo on what remains? Segmenters miss categories outside their training set, such as tools, cables, and the operator's own hand, and they blur boundaries. The file notes that external mask errors propagate into 3D without recourse, and that unguided multi-stage fusion underperforms joint optimization (F-score 0.741 or 0.461 for standalone parts versus 0.935 jointly, in Uncertainty-Aware Deep Multi-View Photometric Stereo). The ground-truth-mask oracle tells us how much of our gain comes from better masks versus joint refinement.

Why not just use a robust loss or intensity trimming instead of an explicit residual model? On dense calibrated data, discarding the top and bottom 40% of intensities rivals complex reflectance models (A Benchmark Dataset and Evaluation for Non-Lambertian and Uncalibrated Photometric Stereo). This is therefore a credible baseline, and we include a trimmed-loss Gradient-SDF variant. Its weakness is that trimming acts on raw intensities: a strong light change pushes whole static regions into the trimmed tails, while a mid-gray intruder survives. The intruder-albedo sweep below is designed to expose exactly this.

Proposed Experiments

No existing benchmark combines uncalibrated lighting change with moving objects, so we build two.

The first is derived from DiLiGenT-MV. Each frame takes one view and one randomly chosen light from the 96 available. Light labels are withheld, which makes the sequence uncalibrated. Moving occluders, either rendered objects or real hand cut-outs, are composited with shadows rendered onto the scanned geometry. The laser-scanned ground truth remains valid for the static object.

The second is a Blender suite with known transients, textured multi-material surfaces, and controlled light motion. It supports the identifiability sweeps.

We add about ten real handheld RGB-D captures under a moving flashlight with people passing through. These are evaluated against a static, clutter-free scan of the same scene.

Two sanity checks confirm we do not regress where assumptions hold. On TUM RGB-D, which has static lighting, absolute trajectory error should stay near the 0.4 cm that Gradient-SDF reports. On clean DiLiGenT-MV, normal error is compared against PS-GS's reported 3.68° on BEAR.

Metrics are Chamfer distance, F-score, normal mean angular error, transient-mask IoU against compositing ground truth, the fraction of static pixels wrongly masked, and peak GPU memory.

The decisive checks are three. First, the transient IoU of the full system versus the raw-error ablation, as a function of how much the light changes per frame. The proposal predicts the raw-error masks get worse as light change grows while ours stay flat. Second, the lighting-capacity sweep, which should show detection degrading as the lighting model becomes flexible enough to fit intruders. Third, the intruder sweep, which locates the failure regime where an intruder's shading falls within the lighting basis. If the first check shows no gap, the core claim is false.