Depth cameras that emit their own infrared light, such as continuous-wave time-of-flight sensors, give metric geometry for moving scenes that a single color camera cannot pin down. They work by flashing modulated light and measuring the phase delay of its return. Sunlight and halogen lamps flood the same pixels with unmodulated infrared photons. That extra light adds random photon-counting noise ("shot noise") and can saturate pixels entirely. The fastest current way to reconstruct dynamic scenes from such a sensor is 3D Gaussian splatting, which represents the scene as millions of small colored blobs that are rasterized rather than ray-marched, and it is especially fragile to this noise. Splatting grows its scene by splitting blobs wherever the training error pushes hard on blob positions. Noise pushes hard everywhere. Under bright ambient light the method should therefore fill the scene with spurious floating blobs, which corrupt the depth and also slow rendering, because frame rate falls as primitive count rises.
The useful fact is that a time-of-flight camera measures its own ambient light. The four raw phase-shifted exposures it captures per frame contain a constant offset that is mostly ambient flux. Under Poisson photon statistics, the size of that offset tells you each pixel's noise variance for free. This note proposes to use that sensor-reported variance to weight every raw measurement in the reconstruction loss. Saturated pixels are dropped from the loss. Blob splitting is decided by a noise-debiased statistic instead of raw gradient magnitude. Together these should keep dynamic reconstruction accurate, and keep the blob count and frame rate stable, as ambient light rises from dim indoor levels toward sunlight.
Core Claim
The closest work is Time of the Flight of the Gaussians, which reconstructs dynamic scenes with Gaussian splatting supervised by raw time-of-flight phasors. It trains in 40–60 minutes and renders above 100 Hz, roughly 100× faster than the volumetric methods TöRF and F-TöRF. Its stabilizers are fixed heuristics: a 10× lower reflectivity learning rate and low-reflectivity initialization. It is indoor-only, and it neither models ambient noise nor adapts to it.
RobustSplat supplies the failure mechanism this proposal targets. Large photometric residuals cause splitting and cloning that leave permanent floaters, and RobustSplat's remedy is to delay densification on a schedule. Our claim is narrower and different. The splitting decision should be conditioned on per-pixel noise that the sensor itself reports. A global schedule or a global loss weight cannot do this, because ambient flux varies across the image, for example a sunlit window next to a shaded wall.
The rest of the program follows from that one ingredient. Pixels where ambient light dominates automatically lose influence on the time-of-flight term, so the color-image loss takes over. This is the graceful hand-off to passive triangulation that TöRF describes, but at splatting speed.
Research Program
The image formation model follows Li et al. The renderer predicts raw quad intensities $\hat I_k$ for each phase offset $k$, including an explicit per-pixel ambient term $\hat B$. For the observed quads $I_k$ we form a whitened loss:
$$ \mathcal{L}_{\text{ToF}} = \sum_{p \notin \mathcal{S}} \sum_{k=0}^{3} \frac{(I_k(p) - \hat I_k(p))^2}{g\,\bar I(p) + \sigma_r^2} $$
The symbols are defined as follows. $\bar I(p)$ is the mean of the four observed quads at pixel $p$, which is dominated by ambient light when it is bright. $g$ is the sensor gain converting counts to photoelectrons. $\sigma_r^2$ is read noise. $\mathcal{S}$ is the set of pixels at or near full-well saturation. The denominator is simply the Poisson variance plus read noise, so the loss is a Gaussian approximation to the true photon likelihood. Both $g$ and $\sigma_r$ are calibrated once per sensor from flat-field captures.
The second change replaces the splitting statistic. Standard splatting averages the magnitude of per-view positional gradients. Because it averages magnitudes, zero-mean noise inflates the statistic instead of cancelling. We instead accumulate the gradient mean $\bar{\mathbf g}$ and the gradient variance $s^2$ over the $N$ views that observe a blob. A blob is split only when $\|\bar{\mathbf g}\|^2 - s^2/N$ exceeds the threshold. This removes the expected contribution of noise from the squared mean gradient, so persistent structural error triggers growth and random fluctuation does not. With the whitened loss, $s^2$ is itself on a calibrated scale.
Motion handling is inherited unchanged from Li et al.: the deformation network and optical-flow guidance. Keeping it unchanged means any gain is attributable to the noise model, not to a new motion model.
There is no ambient-corrupted dynamic time-of-flight benchmark in the file. We therefore build two.
The first is synthetic. We take the path-traced F-TöRF Blender scenes, which have ground-truth radiative transport and include the axial speed test with motions of up to 127 pixels. We add physically sampled ambient flux at controlled levels to the raw quads, along with Poisson sampling and clipping at full well. This gives an ambient-level sweep with exact depth ground truth.
The second is real. We capture dynamic tabletop and room-scale sequences with an OPT8241-class sensor, chosen to match F-TöRF, rigidly paired with a color camera. Each scene is shot under three conditions: dark, halogen, and window sunlight. Each condition gets a long-exposure static reference for depth on the static background. We will release both.
Baselines
The naive reference is Time of the Flight of the Gaussians run unmodified on the ambient-corrupted quads, using the released hyperparameters. Its indoor operating point is known: 40–60 minutes of training and above 100 Hz rendering. The file reports only that it achieves "lower depth MSE on the majority of synthetic scenes" than NeRF baselines, with no number, so no reported result was found on Lacuna for its depth MSE under ambient corruption. We will measure that degradation ourselves, and this curve is the first deliverable.
The strongest existing method on the robustness question is the same system augmented with RobustSplat-style delayed densification. RobustSplat is the best documented answer to residual-driven floaters, although the file gives no per-scene numbers for it either. We tune its delay length on held-out synthetic scenes with the same budget we give our own threshold.
For the volumetric reference, the file's only geometric number is TöRF's depth MSE of 0.005 on DinoPear, measured indoors. We report TöRF and F-TöRF on the synthetic sweep, while noting they are 60–72-hour methods.
The isolating ablations run our method twice: once with only the whitened loss and the standard split rule, and once with only the debiased split rule under an unwhitened loss.
Every method gets the same frames, the same GPU, the same iteration count, and an equal hyperparameter search. Beating the baselines means two things at ambient levels where the naive baseline's depth error has at least doubled relative to the dark condition. First, depth error must be statistically lower across scenes. Second, the final blob count and frame rate must stay within 20% of their dark-condition values.
Why not just…?
Why not just denoise the time-of-flight frames first? This is the most serious threat. Very Power Efficient Neural Time-of-Flight shows that a U-Net on raw low-exposure quads reaches 10.13 cm MAE, against 44.06 cm for BM3D, so learned 2D denoising is strong. Two problems remain. It was trained on static indoor scenes, and it outputs depth, which TöRF argues discards the phase and amplitude information that raw supervision exploits. We will retrain that U-Net on our synthetic ambient sweep, feed its output to the same splatting pipeline, and compare directly. If it matches us, the honest conclusion is that per-frame denoising suffices, and the note's contribution shrinks to the benchmark.
Why not just turn down the time-of-flight loss weight or mask bright pixels with a threshold? A tuned global weight or hard mask is the simplest version of our idea. A global weight fails when ambient varies within an image, and a hard mask throws away the partially informative pixels that are noisy but not saturated. The whitened-loss-only ablation, compared against a best-tuned global weight with a saturation mask, settles whether continuous per-pixel weighting matters. We will report it even if the gap is small.
Why not just delay densification as RobustSplat does? Delay treats every residual as transient. Ambient shot noise is not transient: it persists for the whole capture. A delay therefore postpones floaters rather than preventing them, or, if made long, starves real structure. The tuned-delay baseline tests exactly this. The decisive evidence is primitive count against training iteration under constant sunlight.
Why not just use the color camera alone? Monocular dynamic reconstruction is ill-posed, and TöRF recovers geometry from two views only because of the metric time-of-flight signal. We include a color-only splatting run. We expect it to lose indoors and to converge toward our method only where ambient light makes the time-of-flight term uninformative, which is the regime where our weighting already defers to color.
Proposed Experiments
The central experiment is the synthetic ambient sweep. We plot depth MSE, LPIPS on held-out views, final blob count, and rendering frame rate against ambient-to-signal ratio for every method above.
The decisive check is whether the naive baseline's blob count rises with ambient level while ours stays flat. That is the predicted floater mechanism. If blob count does not rise for the baseline, the densification half of the proposal is unnecessary and we report that.
A second check targets the split rule directly. In static regions under constant sunlight, the fraction of splits triggered there should drop sharply with the debiased rule.
On real captures, we evaluate depth against the static long-exposure references on background pixels and evaluate held-out color views everywhere.
Finally, we report training wall-clock. Our changes add only per-blob gradient moment accumulation, so training should stay within the 40–60 minute envelope Li et al. report.
References
- Time of the Flight of the Gaussians: Optimizing Depth Indirectly in Dynamic Radiance Fields
- TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis
- Flowed Time of Flight Radiance Fields
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Very Power Efficient Neural Time-of-Flight