Frequency-Aware Outlier Rejection for Reconstructing Static Scenes Past Moving Distractors

Neural scene reconstruction methods build a 3D model of a place from ordinary photos, and they assume nothing in the scene moves. Real captures break that assumption: people walk through, cars pass. The most widely used fix, the robust loss of RobustNeRF, throws away the pixels the model currently fits worst, on the theory that moving objects are what the model cannot explain. We argue this rule has a built-in confusion. Coordinate networks fit smooth content first and fine detail much later, a well-documented tendency called spectral bias. Early in training, therefore, a sharp static edge or fine texture also produces a large error. Rejection by error size cannot tell "moving object" from "fine static detail the model has not reached yet," and the fine detail gets discarded along with the pedestrians.

The proposal is to measure each pixel's error separately in each spatial frequency band. Each band's error is compared against how much error that band normally has at that point in training. A pixel is rejected only if its error is unusual for its own band, and only in bands the model can currently represent. In two sentences: moving distractors are outliers relative to what the model should already have learned, not relative to raw error. Because spectral bias makes "what should be learned by now" depend predictably on frequency, the outlier test should be run per frequency band and tied to the model's current bandwidth.

Core Claim

Residual-based distractor rejection loses fine static detail because residual size is confounded with spatial frequency, and this loss grows with the Fourier feature scale. Normalizing residuals within frequency bands, and restricting outlier decisions to bands the representation has already unlocked, should recover fine static detail without letting distractors back in. This should also make distractor-robust reconstruction much less sensitive to the Fourier scale hyperparameter.

The two research directions play unequal roles here. Spectral bias and Fourier feature mapping supply the mechanism and the diagnosis. Moving the static-scene assumption to real captures supplies the application and the benchmarks. Full 4D reconstruction of the moving objects is out of scope. We only want a clean static model.

Background and Positioning

Tancik et al. showed that mapping input coordinates through random sinusoids, with a scale $\sigma$, lets a coordinate network learn high frequencies at all. The same work showed that $\sigma$ controls a sharp trade-off between blur and noisy artifacts. Their own coordinate-regression results show how large the effect is: 25.57 versus 19.32 PSNR on 2D image regression with and without the mapping. NeRF relies on the same kind of encoding. Nerfies introduced coarse-to-fine annealing of these encodings to keep a deformation network out of bad solutions. We reuse that annealing as a method component, as an explicit clock for "which bands the model can currently represent."

On the distractor side there are four approaches. RobustNeRF trims the highest-residual half of pixels and smooths the mask over 16×16 patches. NeRF On-the-go predicts per-pixel uncertainty from DINOv2 features. SpotlessSplats uses Stable Diffusion features plus pruning of underused Gaussians. 3DGS-HPC calibrates perceptual cues with a photometric mixture model. The residual-based methods motivate this proposal, because none of them conditions its threshold on frequency. The feature-based methods constrain it, because they avoid the confound entirely, so we must show a gain over them or complementarity with them.

The closest prior work is RobustSplat. It delays Gaussian densification and bootstraps transient masks from coarse image resolution to fine. Computing masks at low resolution partly low-pass filters the residual, so RobustSplat already contains a crude version of our intuition. It differs from our proposal in three ways. Its schedule is a fixed resolution ramp that is not tied to what the representation has learned. It judges one full-band residual at each resolution rather than normalizing per band. It has never been tested as a fix for Fourier-scale sensitivity.

DASH is related in spirit. It classifies primitives as static or dynamic by how well a linear motion model converges. We likewise use convergence behavior as the signal, but per frequency band of the image residual rather than per primitive's motion.

Method

Let $r_k(p,t)$ be the residual at pixel $p$ and training iteration $t$ after band-pass filtering both the render and the photo into band $k$ of a Laplacian pyramid, from $k=0$ (coarsest) to $K$ (finest). We track robust statistics of $|r_k|$ over all pixels in the current batch: the median $m_k(t)$ and the median absolute deviation $s_k(t)$. These are running estimates of how much error band $k$ "normally" carries at time $t$. They are measured from data rather than predicted from theory, which sidesteps the infinite-width caveat of the kernel analysis in Tancik et al.. The outlier score is

$$ z(p,t) = \max_{k \le k^*(t)} \frac{|r_k(p,t)| - m_k(t)}{s_k(t)}, $$

where $k^*(t)$ is the finest band currently admitted by the Nerfies-style annealing schedule on the Fourier encoding. A pixel is excluded from the loss when $z$ exceeds a threshold. We then apply RobustNeRF's 16×16 patch smoothing. Bands finer than $k^*(t)$ are still supervised but never used to flag outliers.

Large distractors differ from the background in color and shape, so they remain obvious in coarse bands, where the static scene has already converged. Fine static texture shows up only in fine bands, where normalization removes the early-training offset. An edge case we will report honestly is a distractor whose difference from the background is purely fine-scale, such as a thin moving wire. Such an object could slip through until its band is admitted.

We build this first on a Mip-NeRF-360-style coordinate network, where the Fourier encoding and annealing are explicit. We then transfer it to 3D Gaussian Splatting (3DGS). Gaussians have no Fourier encoding, but they do fill in fine detail late through densification. So we replace the annealing clock with the empirical plateau of each band's median $m_k(t)$. The transfer tests whether the idea depends on Fourier features specifically or on coarse-to-fine learning generally.

Baselines

The naive reference is the unmodified backbone trained on distractor-laden data: Mip-NeRF 360 for the coordinate network and vanilla 3DGS for splatting. RobustNeRF reports 25.22 dB PSNR for Mip-NeRF 360 against 30.87 dB for RobustNeRF on its synthetic distractor scene. SpotlessSplats reports 21.70 PSNR for 3DGS against 22.81 for its method on the On-the-go Fountain scene. These numbers bound the size of the problem.

The most direct competitor is RobustNeRF's trimmed loss. It shares our backbone and our residual-only signal, so any gain over it is attributable to band normalization. On the On-the-go Patio scene, NeRF On-the-go reports 20.54 PSNR for RobustNeRF against 21.41 for NeRF On-the-go. The strongest existing residual-based method is RobustSplat. The file reports that it "consistently achieves the best results across all scenes," but gives no per-scene numbers; there is no reported per-scene result found on Lacuna for RobustSplat, so we will rerun it. For feature-based filtering we use SpotlessSplats and 3DGS-HPC. For 3DGS-HPC the file reports only a 1.39 to 5.54 dB improvement over 3DGS on the RobustNeRF data, at roughly 13 to 17 minutes on an RTX 4090.

Every method runs on the same images and camera poses, with the same backbone and iteration budget. Each gets the same threshold search: one sweep of trim fraction or threshold, tuned on a single held-out scene and then frozen for all others.

Three ablations isolate the key ingredient. The first uses band normalization without the annealing gate, testing whether the gate matters. The second uses Nerfies-style annealing with the plain trimmed loss, testing whether annealing alone explains any gain. The third transplants RobustSplat's resolution-cascaded masks into our coordinate backbone, separating "coarse-to-fine masking" from "per-band normalization."

We count it as beating a baseline only if three conditions hold together. PSNR, SSIM and LPIPS on clean held-out views must improve over the trimmed loss and the transplanted cascade. The gain must be concentrated in fine-band PSNR. Distractor recall on Kubric must not drop.

Why not just…?

Why not just use semantic features, as NeRF On-the-go and SpotlessSplats do, which never look at residual size? This is a real threat, because feature-based masks avoid the frequency confound by construction. The evidence against relying on them alone is that 3DGS-HPC documents perceptual thresholds erasing textureless static walls and missing shadows. NeRF On-the-go also reports reflective surfaces being flagged as distractors. Our experiments compare against these methods directly and also combine our residual score with their feature score. If features alone match us on fine-band error, the contribution narrows to "a better residual signal for feature-free or feature-fragile settings," and we will say so.

Why not just use RobustSplat's coarse-to-fine mask cascade, which already delays fine-scale decisions? It may capture most of the benefit, which is why the transplanted-cascade ablation exists. The distinguishing prediction is that a fixed resolution ramp still judges a full-band residual at each stage. It should therefore still discard fine static texture that is slower to converge than the schedule assumes, and this should show up most at large Fourier scales. If the cascade matches us across the scale sweep, our per-band statistics add nothing and the proposal fails.

Why not just tune the Fourier scale or the trim fraction per scene? Per-scene tuning needs clean ground-truth views, which distractor-laden captures do not have. The claim under test is that the best scale under distractors differs from the best scale on clean data. Our scale sweep measures this gap directly, and a method that closes it removes the need to tune.

Why not just mask moving objects with an off-the-shelf 2D segmenter or tracker? Split4D reports that tracked 2D masks drift under fast motion and occlusion, outperforming a mask-lifting baseline 0.893 to 0.592 mIoU. Segmenters also miss unnamed distractors such as shadows. We include a SAM-masked static baseline on Kubric, where ground-truth masks let us score mask quality separately.

Proposed Experiments

The first decisive check targets the mechanism, on the synthetic Kubric benchmark, which has ground-truth distractor masks. We train the trimmed-loss baseline and our method while sweeping the Fourier scale across several octaves. We record the fraction of static pixels with high local frequency that are wrongly excluded, over training and at convergence. The hypothesis predicts that for the trimmed loss, this false-exclusion rate rises with Fourier scale and is concentrated early in training. For our method it should stay flat. If the trimmed loss shows no frequency-dependent false exclusion, the premise is wrong and we stop there.

The second check targets reconstruction quality on the RobustNeRF dataset and the 12 NeRF On-the-go scenes. We use the coordinate-network backbone and then the 3DGS transfer, reporting PSNR, SSIM, LPIPS, and PSNR restricted to the finest two pyramid bands. The prediction is that the gain is small in the overall metrics and large in the fine-band metric.

The third check tests whether the method holds up under clutter and lighting changes, on Phototourism scenes. Those scenes contain the moving shadows that all residual methods struggle with. We expect our method to inherit the failure, and we will report it as such rather than claim a fix.

The final check is sensitivity: the spread in held-out PSNR across the Fourier-scale sweep, for each method. A distractor-robust method whose quality no longer depends strongly on the Fourier scale would be the clearest sign that the two research directions have actually been connected.