Methods that build 3D scenes from casual photos assume the world holds still. When a person walks through the frame, a shadow slides across a wall, or a cable sways, the optimizer either averages the conflicting views into ghostly floaters or wrongly adds geometry to explain them. Current fixes decide, pixel by pixel or patch by patch, whether a pixel is "static" or "transient," and they throw away the pixels they distrust. This proposal argues that the pixel is the wrong unit of trust. A moving shadow corrupts the smooth, low-frequency brightness of a wall but leaves the wall's sharp edges intact. A thin swaying wire corrupts only fine edges and leaves the coarse color unchanged. If the image error is split into frequency bands, the reconstruction can keep the trustworthy band of each pixel and discard only the corrupted one.
The core idea, in two sentences: decompose every rendered and real image into frequency bands with a cheap wavelet transform (a transform that splits an image into a coarse approximation plus horizontal, vertical, and diagonal detail layers at several scales), and learn a separate "trust this or not" weight for each band at each location rather than one weight per pixel. Estimate these weights from coarse to fine, but let finer bands overturn coarse decisions. This avoids the early, irreversible mistakes that coarse-to-fine pipelines are known for, and it makes sharpness-boosting frequency losses usable on scenes that contain moving distractors.
Core Claim
Frequency-domain losses sharpen 3D Gaussian Splatting (a scene representation made of millions of small colored 3D blobs that are rasterized into images). FreGS penalizes Fourier amplitude and phase differences and reports Mip-NeRF360 PSNR of 27.85 against 27.21 for plain 3DGS. Wavelet-GS separates low- and high-frequency content with wavelets. Both are evaluated only on static scenes. The research file also records that high-frequency coefficients are extremely sensitive to misalignment. That observation motivates the proposal: on a casual capture with moving distractors, a FreGS-style loss should amplify transients, because it injects large gradients exactly where a moving object creates edges the static scene cannot explain. Those gradients feed the splitting heuristic that RobustSplat identifies as the source of frozen mid-air Gaussians.
We claim that band-specific robust weighting resolves this tension. It recovers the sharpness gains of frequency supervision on distractor-laden captures. It also fixes two documented failure modes of current distractor removal: thin static structures and micro-distractors, both of which 3DGS-HPC concedes its 16×16 patches cannot handle, and illumination-type transients such as shadows, which RobustSplat does not address.
Research Program
The objective replaces the per-pixel photometric loss with a weighted loss over wavelet sub-bands:
$$\mathcal{L} = \sum_{b} \sum_{x} w_b(x)\, \frac{\left| W_b(\hat{I})(x) - W_b(I)(x) \right|}{s_b},$$
Here $W_b$ is the wavelet sub-band $b$ (a Haar transform at three levels, giving one coarse band and nine detail bands), $\hat{I}$ is the rendered image and $I$ the captured photo, $x$ indexes locations in that band, $s_b$ is a running per-band scale, and $w_b(x)\in[0,1]$ is the learned trust weight. The per-band normalization by $s_b$ follows the coefficient rescaling used by Joint Sub-bands Learning with Clique Structures, which we reuse as a method to stop low-frequency energy from dominating the loss. A Haar transform at three levels costs a few additions per pixel. This sidesteps the transform-overhead concern raised in the file for FFT-based losses.
The trust weights are estimated in three steps. The first step borrows from RobustNeRF, which treats distractors as outliers and trims the highest-error half of residuals with a spatial smoothing kernel. We apply that trimming separately inside each band, so "outlier" is judged against that band's own error distribution.
The second step propagates weights from coarse to fine. A coarse band's weight acts as a prior on the finer bands at the same location, but only as a soft prior. A finer band can reopen a location if its residual behaves like static content. This is the lesson of BEAMER, which shows that committing to a single coarse hypothesis traps dense matchers near discontinuities. It is also the lesson of LoFTR, whose fine stage cannot recover from a coarse match that falls outside its window. Here the coarse-to-fine idea from matching serves as a design constraint, not as the application.
The third step separates static from transient content using how residuals change over time. A thin static structure and a thin distractor look identical at any single iteration: both produce large high-band residuals before they are fitted. They differ over time. The static structure's residual shrinks as consistent views pull Gaussians into place, while a distractor's residual persists or jumps between views. We track an exponential moving average of each band's residual per training image and let the trend, not the instantaneous value, drive $w_b$.
Densification, the step that splits and clones Gaussians, reads its gradients only from trusted bands. This extends RobustSplat's decoupling of densification and dynamics from a global delay to a band-specific gate. The method never models the moving objects themselves. As in RobustSplat and 3DGS-HPC, the target is the clean static scene.
Baselines
The naive reference point is vanilla 3DGS with its standard L1-plus-SSIM loss and no distractor handling. It defines how much damage the transients cause. A second, deliberately naive reference is FreGS's frequency loss applied unchanged to distractor data. It tests the premise that unweighted frequency supervision makes transient artifacts worse. If FreGS does not degrade relative to vanilla 3DGS on these captures, the motivation weakens and we will report that.
The strongest existing competitor is RobustSplat, which delays Gaussian growth and bootstraps transient masks from low resolution upward. It is the closest prior work because it already combines coarse-to-fine reasoning with transient removal without external segmentation models. Its cascade, however, refines the resolution of a single pixel mask, not the frequency content being trusted. It is reported as best across all NeRF On-the-go and RobustNeRF scenes, but no reported result found on Lacuna for RobustSplat's exact PSNR on either dataset. We will therefore rerun it from its public description. 3DGS-HPC is the second strong competitor, reporting a 1.39 to 5.54 dB PSNR gain over native 3DGS on the RobustNeRF dataset with 13–17 minutes of training on an RTX 4090. As a NeRF-based reference on the RobustNeRF-style "Patio" scene, NeRF On-the-go reports PSNR 21.41 against RobustNeRF's 20.54. We also include a 3DGS port of RobustNeRF's trimmed loss.
Fairness rules apply to every method. All methods share the same 3DGS codebase, initialization from the same structure-from-motion points, the same iteration count, the same held-out clean evaluation views, and an equal hyperparameter sweep of the same size on one validation scene per dataset.
Three ablations isolate the key ingredient. The first is pixel-shared trust: identical pipeline, but one weight per pixel broadcast to all bands. This is the decisive ablation, since it removes band specificity and keeps everything else. The second replaces soft coarse-to-fine priors with hard gating, which tests the irreversibility argument. The third removes the residual-trend signal, which tests the thin-static-structure claim.
Beating the baselines means matching or exceeding RobustSplat on full-image PSNR, SSIM, and perceptual distance (LPIPS) while showing a clear gain on the subsets where the hypothesis predicts it. Those subsets are high-band reconstruction error and edge accuracy on thin structures, and full-image error on Photo Tourism scenes with shadow and lighting transients.
Why not just…?
Why not just mask transients with Segment Anything or DINOv2 features? 3DGS-HPC documents that semantic models miss transients that are not objects, such as moving shadows and lighting shifts. They are also fragile on the blurry renders of early optimization, sometimes deleting textureless static walls. ForestSplats likewise reports better generalization and large memory savings by avoiding such models. Shadow-type transients are exactly the case where band-wise trust should help most, because the low band is corrupted while edges remain usable. A segmentation mask can only discard the whole region.
Why not just combine existing pieces: RobustNeRF-style trimming to get a pixel mask, then FreGS's frequency loss on the surviving pixels? This is the real threat, and it is exactly the pixel-shared-trust ablation. If that ablation matches the full method, the contribution reduces to "frequency loss plus masking," and we will say so. Our prediction is that it loses specifically on shadows, where it discards edge information it could have kept, and on thin structures, where a pixel mask erases them. The controlled injection experiment below is designed to separate the two cleanly.
Why not use a spatial Laplacian pyramid instead of wavelets? A Laplacian pyramid is itself a band-pass decomposition, so this alternative is not wrong in principle. The claim is about trusting bands separately, not about Haar wavelets as such. We will run a Laplacian-pyramid variant with identical band-wise weighting. If it ties, the proposal stands with the cheaper transform. What we argue against is RobustSplat-style resolution cascades, which still issue one decision per pixel. MDPG explains that unweighted multi-scale spatial losses stay dominated by low-frequency energy at every scale.
Why not detect moving content with an off-the-shelf tracker or optical flow? Flow says nothing about illumination transients, and trackers drift. Dynamic 3D Gaussians reports a 15.7-pixel median trajectory error for PIPs on PanopticSports, an order of magnitude worse than joint optimization. At that error level, any flow-derived mask would be badly misplaced around thin structures.
Proposed Experiments
The main evaluation uses the twelve NeRF On-the-go sequences, with occlusion from 5% to 30%, the four RobustNeRF scenes with paired clean views, and Photo Tourism landmarks for lighting and crowd transients. Metrics are PSNR, SSIM, and LPIPS on clean held-out views. We add per-band reconstruction error on those views, edge F-score restricted to thin-structure regions, and the number of Gaussians lying outside the static scene's surface as a floater count. Mip-NeRF360 serves as a no-distractor sanity check: the method should match FreGS there, showing that the trust weights do not discard good signal when nothing moves.
The decisive check is a controlled injection study on the clean RobustNeRF views. We add synthetic smooth multiplicative shadows, which are purely low-band, and synthetic thin moving lines, which are purely high-band, at known locations. This lets us measure whether the learned $w_b$ goes low only in the corrupted band. Three predictions should hold together. Band-wise trust should beat pixel-shared trust mainly under shadows and thin lines and tie under large opaque distractors. Hard coarse-to-fine gating should erase thin static structures that soft gating keeps. Naive FreGS should produce more floaters than vanilla 3DGS. If the first prediction fails, band specificity buys nothing and the idea should be dropped.
References
- FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization
- Wavelet-GS: 3D Gaussian Splatting with Wavelet Decomposition
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution
- RobustNeRF: Ignoring Distractors with Robust Losses
- Handling Multiple Hypotheses in Coarse-to-Fine Dense Image Matching
- LoFTR: Detector-Free Local Feature Matching With Transformers
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- ForestSplats: Deformable Transient Field for Gaussian Splatting in the Wild
- MDPG: Multi-domain Diffusion Prior Guidance for MRI Reconstruction
- Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis