Methods that build a 3D scene from photographs usually assume nothing in the scene moves. When that assumption breaks, current systems do one of two things. They throw the offending pixels away as "distractors," or they let the whole scene deform over time. Both choices ignore that a region resisting a static fit can be resisting for different reasons. A tourist walks through one photo. A leaf sways a few centimetres. A window reflects the sky differently from each viewpoint. The sun moves between captures. Or the static model simply has not yet grown enough detail there. Each cause calls for a different, small change to the model, and mistaking one for another produces the known failures of this literature: shiny surfaces reconstructed as holes, foliage mislabeled as moving, and fine static texture discarded along with the pedestrians.
Computational linguistics met the same problem decades ago. When a grammar-based parser failed on a sentence, some systems stopped treating the sentence as garbage. They asked which missing rule or dictionary entry would have let the parse succeed, and collected statistics over a large corpus to decide which components were actually at fault. We propose to carry that discipline into 3D reconstruction. When a region of a Gaussian-splatting scene fails to fit, the system pools evidence across all images to decide whether the failure is real. It then tries a small menu of typed model extensions for that region, freeing one degree of freedom at a time. It keeps the extension that best predicts held-out views per added parameter. The output is a cleaner static reconstruction together with an explicit, checkable account of why each region failed.
Core Claim
Diagnosing the cause of a static-fit failure before deciding how to handle it will preserve static detail and view-dependent surfaces that distractor rejection destroys. It will do so while using fewer added parameters than a model that switches on every kind of extra capacity everywhere. The new element is the combination of two pieces borrowed from parsing-failure research, applied to multi-view reconstruction. The first is collection-wide blame assignment, which separates systematic failures from noise. The second is local "wildcard" hypothesis testing over a typed inventory of explanations, scored on held-out views. Neither piece exists in current distractor-free reconstruction.
What Carries Over From Parsing, and What Does Not
The motivating line of work treats a parser halt as evidence that the grammar is incomplete. Linguistic Knowledge Acquisition from Parsing Failures breaks down failed charts to propose missing grammar rules and lexical entries. It validated this by deliberately deleting a prepositional-attachment rule and checking that the system re-proposed it. We borrow exactly that controlled-deletion protocol for evaluation.
The same paper also supplies a warning. It produced between 1 and 72 candidate hypotheses per sentence and needed handcrafted elimination heuristics to prune them, a problem Hypothesis Selection in Grammar Acquisition also confronts. Our design responds in two ways. We keep the space of explanations small and typed. We replace hand-written elimination rules with a single held-out scoring criterion.
Two further techniques provide the method. The wildcard idea comes from Computer Aided Correction and Extension of a Syntactic Wide-Coverage Lexicon and Chart Mining-based Lexical Acquisition with Precision Grammars. In both, a suspect word's constraints are temporarily removed, and the features the grammar needs to finish the derivation reveal the missing entry. Our analogue removes one specific constraint from a scene region. For example, the region's color may be allowed to vary with viewing direction, while everything else stays frozen.
The aggregation idea comes from Error Mining in Parsing Results (also published as Fouille d'erreurs sur des sorties d'analyseurs syntaxiques). That work correlates word sequences with parse failures across millions of words. It iteratively spreads each failed sentence's blame over the words it contains, so that innocent words appearing next to guilty ones are cleared. A rendered pixel has the same structure: its error is shared by every Gaussian along the camera ray. This makes the iterative blame distribution a direct fit for deciding which 3D clusters are actually responsible for failing pixels.
One part does not transfer. Parsers have discrete grammars, and the hypotheses are symbolic. Ours are continuous model extensions whose value must be measured, not proved.
Relation to Reconstruction Work
Three systems treat static-fit failure as something to remove. RobustNeRF optimizes only the lowest-residual half of pixels with spatial smoothing. NeRF On-the-go predicts per-pixel uncertainty from DINOv2 features (DINOv2 is a self-supervised image feature extractor) to mute transient pixels. SpotlessSplats masks transients using Stable Diffusion features and prunes under-used Gaussians.
All three share a single response to failure, which is to down-weight it. Their papers report the consequences. Reflective surfaces get misread as distractors, foliage confuses the detectors, and aggressive rejection erodes fine static texture. SpotlessSplats additionally observes that in Gaussian splatting, color residuals are noisy because densification itself creates temporary errors. That observation is the strongest argument that a single residual threshold cannot tell "moving object" apart from "not yet fit."
WildGaussians adds a per-image appearance module for lighting. That module is a second hypothesis type, but it is applied globally and unconditionally rather than chosen per region.
On the dynamic side, DAS3R learns a per-Gaussian "staticness" weight guided by a pairwise motion mask, and DynaSplat separates static from deforming Gaussians using deformation variance and optical flow. Both reduce the question to a single static-versus-dynamic axis. Our proposal widens that axis into a small set of causes and makes the choice among them explicit.
HOSNeRF attaches objects to a human skeleton when they start moving. It is the closest structural analogue, since a static element there acquires new degrees of freedom. However, it relies on external masks and pose estimators rather than on the reconstruction's own failures, so we treat it as context and not as a baseline.
Research Program
We start from standard 3D Gaussian splatting, which represents a scene as many small colored blobs rendered by projection, trained on a static model. Training proceeds as usual, with the system logging diagnostics after each densification phase. Gaussians are grouped into spatial clusters. For each cluster, the system records a residual matrix over images, accumulated through the rendering weights. Those weights tell us how much of each pixel each Gaussian contributed.
Error mining. The error-mining stage runs the iterative blame redistribution of Sagot and de la Clergerie over this matrix. A cluster is "suspect" if it keeps absorbing blame after co-occurring clusters have taken their share. A second filter guards against densification noise: a cluster's suspicion must persist across two consecutive densification phases. The residual's pattern over images is kept as a signature. Concentrated in a few images suggests a transient. Varying smoothly with viewing direction suggests a reflection. Low-frequency and shared by everything in an image suggests lighting. Small and temporally correlated suggests sway. Uniformly high everywhere suggests under-fitting.
Wildcard hypothesis testing. For each suspect cluster, the system tries six typed extensions, each freeing exactly one degree of freedom for that cluster alone:
- higher-order view-dependent color, for specular surfaces;
- a per-image appearance offset, for lighting;
- a per-image transient opacity mask;
- a low-dimensional smooth deformation, for sway;
- a per-frame rigid motion, for a moving body;
- extra densification, for under-fit static geometry.
Each extension is fitted for a few hundred iterations with the rest of the scene frozen. It is then scored on held-out images in which the cluster is visible but not flagged:
$$S(h) = \Delta \mathcal{L}_{\text{held}}(h) - \lambda\, k_h \log n_c$$
Here $\Delta \mathcal{L}_{\text{held}}(h)$ is the reduction in held-out photometric loss for that cluster under hypothesis $h$, $k_h$ is the number of parameters $h$ adds, and $n_c$ is the number of pixels the cluster covers. The penalty is a standard Bayesian-information-style complexity charge. The held-out requirement is what keeps the transient mask from winning every time: masking lowers training loss, but it predicts nothing new about unflagged views.
The winning extension is committed and training resumes. Because the inventory is fixed at six and only suspect clusters are tested, the cost grows with the number of suspects, not combinatorially. This avoids the hypothesis explosion that forced handcrafted elimination in the grammar-acquisition work.
Analysis. We study three things. The first is whether the diagnosed type matches ground truth when it is known. The second is how much of the gain comes from typing versus from aggregation. The third is where the method fails: we expect trouble when two causes overlap in one cluster, such as a reflective car that is also moving. That case mirrors the parsing literature's finding that diagnosis degrades when several errors co-occur.
Baselines
The naive reference point is vanilla 3D Gaussian splatting trained on contaminated captures. It shows how much damage the static assumption does. For context, on the Fountain scene SpotlessSplats reports 21.70 PSNR (peak signal-to-noise ratio, in dB) for standard 3D Gaussian splatting.
The strongest existing method in our Gaussian setting is SpotlessSplats, which reports 22.81 PSNR on the same Fountain scene. It is the strongest competitor because it already combines feature-based transient detection with pruning on the same representation we use. Beating it in average PSNR is necessary but not sufficient. We must also beat it on masked PSNR over reflective and fine-texture regions, where it is expected to lose detail.
For the RobustNeRF benchmark, the reference figures are 30.87 dB for RobustNeRF against 25.22 dB for Mip-NeRF 360. For uncertainty-based masking, NeRF On-the-go reports 21.41 PSNR on Patio against RobustNeRF's 20.54. For lighting variation we compare against WildGaussians. No reported per-scene result was found on Lacuna for WildGaussians on the NeRF On-the-go dataset, and none for its aggregate Photo Tourism PSNR, so we will rerun it. On unposed video with large movers we compare against DAS3R, which reports more than 2 dB PSNR improvement over SpotlessSplats and WildGaussians on DAVIS, Sintel, and Point Odyssey.
All baselines will be run from released code on identical image splits, camera poses, and resolution. Each gets the same total GPU-hour budget, including our diagnosis passes, and an equal hyperparameter sweep of twelve configurations tuned on a shared validation split.
Ablations. Three ablations isolate the two borrowed ingredients. The first is a kitchen-sink model: every extension type is switched on for every Gaussian, matched to our model's final parameter count, and trained jointly. This tests whether typing matters or capacity alone suffices. The second replaces error mining with a per-pixel residual threshold. This tests whether ray-level blame redistribution matters. The third scores hypotheses on training loss instead of held-out loss. This tests whether the held-out criterion is what prevents masking from dominating.
Why not just…?
Why not just switch on every kind of extra capacity everywhere? This is the real threat. Combining WildGaussians-style appearance modules, uncertainty masks, view-dependent color, and a deformation field in one model is simple and may work. Our argument is that unassigned capacity gets absorbed by the wrong explanation. An appearance embedding can soak up a transient, or a deformation field can bend geometry to fake a reflection, because nothing ties each degree of freedom to its cause. The matched-parameter kitchen-sink ablation settles this. Suppose it comes within 0.3 dB of our method on average PSNR and on masked reflective-region PSNR, and its implicit assignments (which module dominates each region) match the injected ground truth as well as ours. In that case typed diagnosis is not worth its complexity, and we would report that.
Why not just mask movers with a segmentation foundation model before reconstruction? Pre-masking only answers "is this a person or car." It cannot detect reflections or under-fitting at all, and it cannot verify 3D inconsistency. DAS3R reports false positives from learned motion masks on dense branches and swaying foliage, and SpotlessSplats notes that appearance-based detection fails when distractors resemble the background. Our injection benchmark includes unlabeled-category movers and camouflaged movers specifically to measure this, and a SAM-masked Gaussian-splatting run is included in that comparison.
Why not just use a robust loss and accept some lost detail? Trimmed and uncertainty-weighted losses are cheap and strong on average PSNR. But RobustNeRF itself acknowledges the loss of fine detail. WildGaussians and NeRF On-the-go report that shiny surfaces are misread as distractors and that persistently occluded regions become holes. Average PSNR hides these failures, which is why we report masked metrics on reflective and high-frequency regions. If robust losses match us there too, the diagnosis adds nothing.
Proposed Experiments
Controlled-deletion benchmark. The first experiment mirrors the deliberate rule deletion in the grammar-acquisition work. We take clean captures and inject one known failure type per region. Specifically, we composite moving objects, apply a periodic displacement to vegetation, apply per-image global lighting changes, and render with reduced view-dependent capacity in reflective regions. The injections are made both in synthetic scenes and through image compositing on the clean halves of the RobustNeRF paired captures. We report confusion matrices of diagnosed versus true type, which is a capability no baseline offers. The decisive check is whether reflections are classified as reflections rather than as transients at a rate well above the kitchen-sink model's implicit assignment.
Natural benchmarks. The second set of experiments uses the RobustNeRF benchmark, the NeRF On-the-go dataset (12 scenes with 5% to 30% occluded pixels), and Photo Tourism. We report PSNR, SSIM (structural similarity), and LPIPS (a learned perceptual distance) on clean held-out views, along with masked PSNR on hand-annotated reflective and fine-texture regions.
Unposed video. The third experiment uses DAVIS for unposed video with large movers against DAS3R. Here we test whether rigid-motion hypotheses can replace its learned staticness weight.
Efficiency. Across all experiments we report the number of added parameters and the wall-clock overhead of diagnosis. The parsing analogues took over ten CPU days for chart mining, so cost is a legitimate concern and must be bounded.
The proposal succeeds only if three things hold together: it beats SpotlessSplats on average PSNR, it clearly improves reflective-region and fine-texture metrics, and it beats the matched kitchen-sink ablation. Average PSNR alone would not be enough.
References
- Linguistic Knowledge Acquisition from Parsing Failures
- Hypothesis Selection in Grammar Acquisition
- Computer Aided Correction and Extension of a Syntactic Wide-Coverage Lexicon
- Chart Mining-based Lexical Acquisition with Precision Grammars
- Error Mining in Parsing Results
- Fouille d'erreurs sur des sorties d'analyseurs syntaxiques
- RobustNeRF: Ignoring Distractors with Robust Losses
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- SpotLessSplats: Ignoring Distractors in 3D Gaussian Splatting
- WildGaussians: 3D Gaussian Splatting In the Wild
- DAS3R: Dynamics-Aware Gaussian Splatting for Static Scene Reconstruction
- DynaSplat: Dynamic-Static Gaussian Splatting with Hierarchical Motion Decomposition for Scene Reconstruction
- HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video