Depth Disagreement as a Motion Signal for Active-Light 4D Reconstruction

Methods that rebuild a 3D scene from many images assume nothing moves. When something does move, they either smear it into ghostly fog or, in the case of 3D Gaussian Splatting (a scene representation made of millions of small colored blobs), spawn blobs that stay floating in mid-air after the object has left. The current fixes decide what is "moving" from color: a pixel whose color the static model cannot explain is treated as a distractor. Color is a poor witness. A moving shadow, a specular highlight, or a change in exposure all look like motion, and on a blank white wall nothing looks like anything.

We propose to decide what moved from geometry, measured directly. A structured-light sensor is a projector that casts a known pattern plus a camera rigidly bolted beside it, and it measures depth by triangulation. It produces a metric depth map every frame, even on textureless surfaces, and in infrared it is blind to visible shadows. Because projector and camera move together, the depth it reports along each camera ray depends only on what physically sits on that ray. It does not depend on where the rig is or how the room is lit. We can therefore render the current static reconstruction into each frame and compare. Where measured depth is closer than the static model predicts, something has entered the ray. Where measured depth is farther, the static model contains blobs in what is actually empty space. Where the sensor returns nothing, we have no evidence and should not guess. We use this per-ray depth disagreement, fused with semantic features from an ordinary RGB camera, to separate static background from moving objects and to keep and label the moving objects rather than discarding them.

Core Claim

Per-frame active depth from a rigid projector-camera rig gives a test of "did the geometry on this ray change?" that does not depend on lighting or pose. Using that test to gate both distractor classification and blob creation in a dynamic Gaussian scene will remove three known failure modes together. The first is shadows and specular surfaces being mislabeled as moving, which NeRF On-the-go and RobustSplat both acknowledge. The second is floating blobs left behind by passing objects. The third is collapse on textureless backgrounds, where passive 4D methods like Split4D have no correspondence signal. The claim is specifically about using depth as a decision signal, not as an extra reconstruction loss. That distinction is what the experiments are built to test.

Let $D_{SL}(x,t)$ be the structured-light depth at camera pixel $x$ and time $t$, and $\hat D_{s}(x,t)$ the depth rendered from the current static model at the estimated camera pose. We form the normalized disagreement

$$ r(x,t) = \frac{D_{SL}(x,t) - \hat D_{s}(x,t)}{\sigma(x,t)}, \qquad \sigma(x,t) \approx \frac{D_{SL}(x,t)^2}{f\,b}\,\sigma_d , $$

where $f$ is focal length, $b$ the projector-camera baseline, and $\sigma_d$ the disparity noise of the decoder. The quadratic growth of depth noise with distance follows directly from triangulation: depth is inversely proportional to disparity. Strongly negative $r$ marks an occluder in front of the static surface. Strongly positive $r$ is evidence of free space and deletes the static blobs on that ray up to the measured depth. Pixels with no valid return, or with $|r|$ inside the noise band, carry no geometric vote.

Research Program

Sensor. The sensing setup is an infrared dot-projector rig with a co-mounted RGB camera behind an IR-cut filter. The Intel RealSense D435 used by NSL is the concrete target. Putting the pattern in infrared sidesteps a problem the file raises: projected stripes or dots would corrupt DINOv2 features, which were trained on unpatterned images. The RGB stream stays natural, and the pattern stream is used only for geometry.

Per-frame depth decoding. We decode per-frame depth with the neural feature-matching decoder of NSL, which matches learned features between the IR image and the reference pattern and reports binocular MAE of 0.0237 m against 0.3568 m for template matching on its synthetic set. We deliberately drop NSL's monocular refinement stage. That stage oversmooths thin structures, and the disagreement test needs raw triangulated depth with an honest noise estimate. We also need a validity mask for dark, specular, or multipath pixels, which NSL and Micro-Baseline Structured Light both identify as failure cases.

Representation. The scene is two Gaussian sets. The static set is built in the style of RobustSplat, whose delayed blob growth we keep. The dynamic set follows Split4D: each blob carries a linear velocity and a learned identity vector, trained chronologically. The disagreement $r$ changes three places in the optimization. First, static blobs may only be created or cloned at pixels whose geometry agrees with measurement. Second, positive $r$ applies a free-space penalty that removes static blobs in front of the measured surface. Third, negative $r$ seeds dynamic blobs at the measured 3D points, so that moving objects on textureless backgrounds get geometry the photometric loss could never supply.

Fusion with the semantic branch. Geometry has its own blind spot: an object moving parallel to a surface it touches, such as a sheet of paper sliding on a desk or a hand flat on a table, changes depth by less than the noise band. For these cases we keep a semantic branch that predicts per-pixel distractor uncertainty from DINOv2 features, as in NeRF On-the-go. Its weight is driven by the geometric evidence. Where $|r|$ is large and the return is valid, the geometric vote dominates. Where the return is invalid or $|r|$ sits in the noise band, the semantic vote decides. This is the cross-modal fusion at the heart of the proposal: two signals that fail in different places, combined according to which one is trustworthy at each pixel. It is not concatenating RGB features onto points in the manner of PointFusion.

Instance identity. For instance identity on dynamic blobs we use Split4D's contrastive grouping on unassociated per-frame 2D masks, with one change: masks are only allowed to supervise blobs inside regions flagged dynamic. Temporal association is then checked by metric 3D overlap of measured points rather than by Hungarian matching. That addresses the static-world failure of Panoptic Lifting and PCF-Lift, both of which break when objects move between views. Split4D's own reported weakness is blob sharing at human-object contact zones, and metric depth gives a direct constraint there: at contact, the two surfaces are separated along the ray by measured depth even when their colors blend.

RGB-IR timing offset. The RGB and IR streams are not exposed at the same instant. We estimate a per-sequence time offset jointly with the blob velocities, so that semantic masks are compared against the dynamic blobs at their RGB exposure time. Uncorrected offset is the misalignment failure the file attributes to naive projection fusion.

Baselines

The naive reference point is static 3D Gaussian Splatting trained on the RGB stream with the structured-light depth as an added depth loss and no dynamic handling. This shows how much damage motion does when it is ignored.

The strongest existing distractor-removal method is RobustSplat (RobustSplat), which delays blob growth and bootstraps transient masks from coarse to fine resolution. It is the strongest competitor because its target is exactly the floating-blob failure. The file reports that it beats SpotLessSplats, WildGaussians, and T-3DGS on the RobustNeRF benchmark and the NeRF On-the-go dataset, but no reported result was found on Lacuna for its exact per-scene numbers. For reference, RobustNeRF reports 30.87 dB PSNR against 25.22 dB for Mip-NeRF 360 on its distractor scenes, and NeRF On-the-go reports 21.41 PSNR on Patio against 20.54 for RobustNeRF. These numbers come from RGB-only data and are not directly comparable to our captures.

For the dynamic-object side, the strongest method is Split4D. It reports 0.893 mIoU on Multi-Human against 0.592 for the tracker-based SA4D. No reported result was found on Lacuna for any method on active-light dynamic scenes, so all numbers on our data will be produced by us.

Fairness rules apply to every baseline. Each receives the same RGB frames, the same camera poses, the same training iterations, and equal hyperparameter search budgets. Each also gets a "+depth" variant that adds the structured-light depth as an L1 depth loss. Without that variant, any win could be credited simply to having more sensor data.

The key ablation swaps the two roles of depth. One version uses depth only as a reconstruction loss. The other uses depth only as the gate for blob creation, free-space deletion, and distractor classification, with no depth loss. A second ablation removes the semantic branch, measuring what the geometric gate misses on its own. A third replaces structured-light depth with per-frame scale-aligned Depth Anything v2 depth inside the identical pipeline.

Beating the field means three things on held-out clean views and labeled regions. Our method must exceed RobustSplat+depth in static-background PSNR. It must cut the false-dynamic rate on shadow and specular regions by a clear margin. It must exceed Split4D+depth in dynamic-object mIoU on textureless-background sequences. Static-scene quality may not drop on sequences that contain no motion.

Why not just…?

Why not just add a depth loss to Split4D or RobustSplat? This is the most serious threat, and the "+depth" baselines exist to answer it. A depth loss pulls surfaces toward measured depth. It does not change what drives blob creation or distractor labeling in either method: both still decide from photometric residuals, so a moving shadow still triggers densification and transient masking. The depth-loss versus depth-gate ablation settles the question. If depth-as-loss matches the gated system on the shadow, specular, and floater metrics, our central claim is wrong, and the contribution reduces to a dataset.

Why not just use monocular depth instead of a projector? Monocular foundation models give relative depth that has to be rescaled every frame. NSL reports that they oversmooth thin structures, and on uniform walls they have no cue at all. Scale drift between frames would itself produce false disagreement. The monocular-substitution ablation measures this directly. We expect it to hold up on textured scenes and fail on the textureless and thin-structure subsets. If it does not fail there, the hardware is unnecessary for those regimes, and we will report that.

Why not just subtract the IR pattern image from a background frame? With a stationary rig, plain frame differencing on the IR image is a strong detector of geometric change, and we include it as a baseline on the static-camera sequences. It cannot work once the rig moves, because every pixel's content changes with pose. The disagreement test is the version of background subtraction that survives camera motion: it compares against a rendered static model instead of a stored frame.

Why not just run a video tracker and reconstruct the rest statically? Split4D already reports that tracker-based lifting (SA4D, 0.592 mIoU) loses to native 4D grouping (0.893) under fast motion and occlusion. We include SA4D-style masking with static splatting as a baseline. We do not expect active light to rescue tracker drift.

Proposed Experiments

No existing benchmark combines projected patterns, moving objects, and semantic labels, so we build one in two parts.

The first part is synthetic. We use indoor layouts in the style of Replica and ScanNet (ScanNet), render them with an IR dot pattern under an NSL-style physically based simulation, and add moving people and objects. Each sequence has controlled nuisance factors: moving visible-light shadows with no geometry change, specular and dark objects, textureless walls, sliding-on-surface motion, and hand-object contact. Ground truth covers depth, static-only renders, and per-frame instance masks.

The second part is real. We record handheld D435 captures, each paired with a distractor-free capture of the same scene following the RobustNeRF protocol, so that static-background quality can be scored on clean views.

Metrics cover four groups. Static quality is PSNR, SSIM, and LPIPS on clean views, plus depth MAE of the static model. Floaters are scored as the total opacity of static blobs lying in measured free space. Classification is scored as the false-dynamic rate on shadow and specular regions and the miss rate on sliding-motion regions. Dynamic objects are scored by mIoU over time and by boundary error at contact zones.

Three decisive checks determine the outcome. The first is the shadow test. A light moves while geometry stays fixed. Photometric and DINOv2-based methods should flag the shadows as motion, and ours should not; if ours does, the gate is not doing its job. The second is the textureless-wall test. An object crosses a blank wall. Split4D+depth should lose the object's trajectory, and ours should keep it with mIoU close to its value on textured backgrounds. The third is the sliding-motion test, the expected failure of the geometric gate. Here, recovering performance from the semantic branch is what justifies the fusion. If the geometric gate alone matches the full system on this test, the semantic branch is dead weight and we will drop it.