Methods that build 3D models from video of scenes with moving things in them mostly use one signal to find the moving parts: wherever the current 3D model fails to reproduce a pixel, that pixel is assumed to belong to a moving object (a "transient") and is masked out or modeled separately. This works on clean, well-lit footage. On real sensors it fails in a predictable way. In dim scenes, photon shot noise and electronic read noise produce large reconstruction errors in dark static regions. The reconstruction method then either deletes that static background as "transient" or grows spurious geometry to explain the noise. The same noise makes monocular depth predictions flicker, and those predictions are the priors the dynamic model relies on for geometry. We propose to replace the learned or ad hoc decision of "is this error due to motion?" with a statistical test against a physically parameterized sensor noise floor. The test asks whether a patch's error is larger than the camera's own signal-dependent noise could explain.
The idea is simple to state. Real camera noise has a known form: its variance grows linearly with the true brightness, from shot noise, plus a constant, from read noise. If we divide each pixel's reconstruction error by the noise standard deviation the model predicts at that brightness, static regions should look like unit-variance white noise however dark they are. Moving objects should look like spatially coherent outliers. That one normalized residual then does two jobs. It decides which pixels get transient treatment, and it decides where temporal depth-consistency penalties may be applied without smearing the edges of moving objects.
Core Claim
The claim is that transient detection in dynamic 3D Gaussian splatting should be calibrated by sensor physics rather than by image statistics. 3D Gaussian splatting represents a scene as many small colored 3D blobs that are rendered and optimized against the input frames. A signal-dependent noise model, fit online to the scene's own static pixels, makes the static/transient decision invariant to illumination level. It also gives a principled place to apply temporal depth regularization: we impose it only where the static hypothesis is accepted.
Concretely, we work in linear (pre-gamma) intensity. That means raw frames where available, or frames unprocessed back to linear space otherwise. For pixel $p$ with observed intensity $I_p$ and rendered intensity $\hat I_p$, we compute the whitened residual
$$z_p = \frac{I_p - \hat I_p}{\sqrt{a\,\hat I_p + b + c_t}},$$
where $a$ is the shot-noise gain, $b$ is the read-noise variance, and $c_t$ is a model-error variance that is large early in optimization and shrinks as the static model converges. Under the hypothesis "this patch is static and the residual is pure sensor noise," the sum $Q = \sum_{p \in P} z_p^2$ over a patch $P$ of $k$ effective pixels is approximately chi-squared with $k$ degrees of freedom:
$$Q \sim \chi^2_k \quad \text{(static hypothesis)}.$$
We call a patch transient when $Q$ exceeds the chi-squared quantile at a chosen false-alarm rate. The threshold therefore has a meaning, a false-alarm probability, that does not change with ISO, exposure, or scene brightness. Demosaicing correlates neighboring pixels, so we either test on the raw Bayer mosaic or estimate effective $k$ from the residual autocorrelation on accepted static patches.
The parameters $a$ and $b$ come from camera metadata when it exists. Otherwise they are fit during optimization by robust (median-based) regression of residual variance on rendered intensity over currently accepted static patches. This regression is constrained to an affine variance–brightness relationship, so it cannot absorb a moving object, whose error is not explained by brightness. That constraint is the key difference from a learned per-pixel uncertainty, which can absorb anything. It also addresses the offline-calibration dependency criticized in Improving 3D Reconstruction Through RGB-D Sensor Noise Modeling. That work parameterizes depth noise by distance and incidence angle from calibration scans and uses it to weight static fusion. We borrow its idea that noise models should set fusion weights, but we fit the model in the loop and use it for a static-versus-dynamic decision that the paper never faced, since its scenes sit on turntables.
Relation to Prior Work
The proposal is motivated by two lines of work that do not talk to each other. On the reconstruction side, RobustSplat delays Gaussian densification so the static background settles first, then bootstraps transient masks coarse-to-fine from residuals. 3DGS-HPC labels 16×16 patches as static or transient using a Gaussian mixture fit to photometric error plus DINOv2 features. Both decide from the distribution of errors in the image itself. Neither knows that a dark wall is expected to have a large error. Our patch test is closest in form to 3DGS-HPC: we keep patch aggregation but replace the mixture with a hypothesis test whose null distribution comes from the sensor. We reuse RobustSplat's delayed densification as the mechanism that makes the model-error term $c_t$ shrink over time.
On the sensor side, Optical Flow in the Dark models shot plus read noise on raw low-light images. It shows that learning on raw data beats enhancing first, lowering endpoint error on its most underexposed VBOF setting from 10.88 to 6.09. Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation shows that simulating the actual sensing process beats post-hoc noise. These works supply the noise model and the warning against enhancement pre-processing, but none of them reconstructs a dynamic scene.
For the temporal depth side, Video Depth Anything stabilizes per-frame depth with a temporal gradient matching loss. DVD gets consistency from a video diffusion prior with windowed affine alignment. Both assume depth changes smoothly over time, which is false at the boundary of a moving object. MoDGS supervises monocular dynamic Gaussians with an ordinal depth loss to sidestep scale drift. It treats the depth network as a black box and does not ask which depth predictions are trustworthy under noise. Robust Consistent Video Depth Estimation aligns depth with deformable splines but removes moving objects with Mask R-CNN. That limits it to semantic categories, and it discards the foreground. Our contribution relative to these is not a new depth network. It is a gate: temporal depth consistency is enforced on rendered depth only over patches the noise test accepts as static, and the external video-depth prior is down-weighted where the whitened residual says the input frame was too noisy to trust.
Research Program
We build the method on top of a monocular dynamic Gaussian splatting pipeline in the style of MoDGS. The pipeline uses static Gaussians for background and deformable Gaussians for moving content, with a video depth prior from Video Depth Anything replacing single-image depth.
The first component is the noise-calibrated patch test above. It routes accepted-static patches to the static Gaussians and rejected patches to the deformable set. This replaces the residual-mixture or semantic masks in existing pipelines. The second component is online noise self-calibration. We alternate between fitting $(a, b)$ on accepted patches and re-testing. We initialize from metadata when available and from a single global variance otherwise.
The third component is gated temporal depth regularization. On accepted-static patches we penalize the frame-to-frame change in rendered depth after warping by camera motion, which is the temporal gradient matching idea applied to the reconstruction rather than to a network. On rejected patches we impose no temporal smoothness, so depth may jump at object boundaries. We also weight the video-depth prior per pixel by the inverse noise variance, so the dark, noisy regions where depth predictions flicker most contribute least.
The analysis component asks whether the decision really is illumination-invariant. We plot the empirical false-transient rate on known-static pixels as a function of brightness and noise level for our test and for each baseline. Our prediction follows directly from the chi-squared null: our curve should be flat near the chosen false-alarm rate. Image-statistics methods should show false transients concentrated in dark regions, with the effect growing with ISO.
Baselines
The naive reference is vanilla 3D Gaussian splatting, plus a vanilla dynamic Gaussian model with no transient handling, run on the same frames. It tells us how much any transient handling helps under noise.
The strongest static-scene competitor is 3DGS-HPC. It is the closest method in design, since it also decides per patch from photometric error, and it reports a 1.39 to 5.54 dB PSNR gain over native 3DGS on RobustNeRF. For it we will first reproduce the reported range on clean RobustNeRF before testing under noise. We include RobustSplat as a second strong competitor because it reports the best PSNR, SSIM, and LPIPS across NeRF On-the-go scenes, though no exact number was found in the file.
The strongest dynamic competitor is MoDGS, which reports 22.64 PSNR on DyNeRF against 19.55 for the next baseline. We will run it with both its original depth prior and with Video Depth Anything, so that the depth prior is held fixed between it and our method.
All methods get the same input frames, the same camera poses, and the same iteration budget. Each baseline also gets the same hyperparameter search budget per noise level that we use for our false-alarm rate. That gives every baseline a fair chance to tune its threshold to the noise level, which is exactly the advantage our method claims to remove. Beating a baseline means higher PSNR and lower LPIPS on held-out views of static regions and lower static-region depth flicker at the high-noise settings. It also means not losing more than about 0.3 dB on clean data, because a method that only helps when noise is extreme and hurts otherwise is not useful.
Two ablations isolate the key ingredient. In the homoscedastic ablation, the denominator is a single learned constant, so the noise model has no brightness dependence. In the learned-uncertainty ablation, a per-pixel variance head is trained jointly in the style of uncertainty-weighted NeRFs. The method only earns its claim if it beats both: the first shows signal dependence matters, the second shows physical constraint matters. A third ablation removes depth gating, applying temporal depth regularization everywhere, to measure boundary smearing.
Why not just…?
Why not just denoise the frames first and run RobustSplat or MoDGS? This is the most serious alternative. Optical Flow in the Dark found that enhance-then-estimate pipelines raised flow error from 6.09 to 10.88 on underexposed footage because enhancement breaks the consistency of pixels across frames. AURORA-KITTI similarly found disjoint denoising inferior to joint treatment. But those results concern enhancers tuned for appearance and single-image depth completion. A modern multi-frame video denoiser could do better, and it could hallucinate or erase low-contrast moving objects in ways that hurt transient detection less than expected. We will therefore run this baseline with a strong off-the-shelf video denoiser in front of each competitor. If it matches us on held-out views and transient detection, the honest conclusion is that the noise model belongs in a pre-processor, and we would report it as such.
Why not just let the residual mixture model or a learned uncertainty adapt to the noise level? 3DGS-HPC's Gaussian mixture already adapts its variance to the overall error distribution, and a learned per-pixel variance can model anything. The problem is that both adapt globally or freely. A single mixture variance cannot be simultaneously right for bright and dark regions when noise variance grows with brightness. A free per-pixel variance can raise its value over a moving object and absorb it into the static model. The homoscedastic and learned-uncertainty ablations test exactly this claim. The brightness-binned false-transient plot shows where each fails. If the learned head matches ours, the physical constraint adds nothing and we will say so.
Why not just mask moving objects with a segmentation model? Semantic masks catch people and cars but not moving shadows, reflections, or shifted furniture. 3DGS-HPC reports these failures. Split4D shows tracking-based decomposition reaching 0.592 mIoU against 0.893 for native decomposition on Multi-Human. Low light makes segmentation worse, not better. We will include a SAM2-mask variant of the dynamic pipeline in the dark-scene experiments to show this concretely rather than by argument.
Proposed Experiments
No benchmark in the file pairs dynamic scenes with calibrated low-light raw noise, so the core evaluation is semi-synthetic. We unprocess RobustNeRF, NeRF On-the-go, and DyNeRF frames to linear space and add Poisson-Gaussian noise. The noise uses $(a, b)$ settings spanning roughly well-lit to very dark exposures, chosen to match ranges like VBOF's underexposure levels. We then reprocess the frames to sRGB for baselines that expect it. RobustNeRF provides clean transient-free views for scoring.
For temporal depth, we use the Bonn RGB-D Dynamic sequences, which have ground-truth depth and moving people. On these we measure AbsRel and a static-region flicker metric: the frame-to-frame variance of camera-motion-compensated depth on ground-truth-static pixels. We also measure boundary sharpness on moving people to detect smearing. For the Video Depth Anything prior on Bonn there is no reported result found on Lacuna for its noisy-input behavior, so we will measure it ourselves at each noise level. For real-sensor validation, we will capture a small set of raw handheld dynamic scenes at several ISO settings with a tripod-locked clean exposure for static reference. This is a modest but necessary addition, since every other test depends on synthetic noise.
The decisive check is the brightness-binned false-transient curve on semi-synthetic RobustNeRF at the darkest setting. If our test holds near the nominal false-alarm rate across brightness bins, while 3DGS-HPC and RobustSplat, tuned per noise level, show false transients concentrated in dark static regions, the core claim stands. If the denoise-first pipeline or the learned-uncertainty ablation produces an equally flat curve with equal view quality, the claim fails. In that case the right takeaway is that existing tools already suffice.
References
- Improving 3D Reconstruction Through RGB-D Sensor Noise Modeling
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- Optical Flow in the Dark
- Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos
- DVD: Deterministic Video Depth Estimation with Generative Priors
- MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors
- Robust Consistent Video Depth Estimation