Fast 3D reconstruction models now turn a handful of photos into a 3D scene in under a second. They do this by predicting a cloud of 3D Gaussians, small colored blobs whose positions, sizes and opacities together render the scene. The best of them refine that first guess in a few learned steps. At each step they render the current Gaussians, compare the result with the input photos, and push the pixel error back into 3D to adjust the blobs. That loop quietly assumes every pixel error is fixable, meaning some arrangement of Gaussians could reproduce the photo exactly. Learning theory calls this the realizability assumption: the true answer lies inside the model class. Real photos break it. Mirrors, glass, view-dependent highlights, motion blur and sensor noise produce errors that no arrangement of Lambertian-ish blobs can explain. A refiner that tries anyway invents "floaters", spurious Gaussians hung in mid-air to paint over a reflection.
We propose to split each rendering error into two parts. The realizable part is the error that a consistent 3D change to the Gaussians would remove in every view. The unrealizable part is error that such a change cannot remove without hurting other views. We learn this split using synthetic data, where it can be labeled exactly, and we refine only on the realizable part. Here the synthetic-to-real gap is used as supervision rather than being only a nuisance. A simulator can render the same scene twice, once with effects Gaussians can represent and once with effects they cannot, and the difference between the two renders is a pixel-exact label of what is unrealizable.
Core Claim
Recurrent refiners fail on real scenes largely because they treat misspecification residuals as geometric residuals. ReSplat renders intermediate Gaussians, computes 2D error maps against the inputs, and routes them to 3D through cross-attention over a fixed four-step loop. Learning Efficient Fuse-and-Refine refines feed-forward Gaussians in a sparse voxel scaffold. Neither distinguishes the two sources of error; the research file describes both as treating all residuals equally.
We claim that gating the refinement signal with a predicted realizability map will do three things:
- reduce floaters and depth error on reflective and transparent real scenes;
- leave quality on ordinary scenes unchanged;
- beat gates based on residual magnitude or on the backbone's own confidence.
The last point matters most, because it separates the idea from a robust loss.
Magnitude and confidence gates should fail for known reasons. A large residual can be a large but perfectly fixable misplacement, so down-weighting by size also suppresses useful signal. Confidence is miscalibrated in exactly the wrong way. A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction measured seven backbones on 13 datasets. They underestimate 3D error by a median factor of 2.4×, and the audit reports the most overconfident pixels as the ones labeled most certain. The audit also finds a residual scene-to-scene gap that dataset-level recalibration cannot close. Our gate is predicted per scene and per pixel from cross-view evidence, which targets exactly that gap.
The learning-theory framing does real work in the design, not just in the naming. Under realizability, one inconsistent observation can eliminate a hypothesis; this is the version-space elimination behind Analysis of a greedy active learning strategy. When realizability fails, a discrepancy may be irreducible noise rather than evidence against the hypothesis, and elimination stalls. The analogue for us is this: an update to the Gaussians should be accepted only if it lowers error on views it was not computed from, by more than a noise floor. That is the logic of held-out validation under bounded noise, echoing the Massart-noise fast rates in Localization of VC classes.
A second warning comes from Learning from Synthetic Data: Limitations of ERM. It proves that plain empirical risk minimization, meaning minimizing average training loss, reinforces its own errors once more than half the data inherits the model's mistakes. A refinement loop that repeatedly fits its own renders runs a similar risk of feeding on itself. That paper's guarantees are limited to the realizable case and to stylized problems. We borrow it as motivation and a design constraint, not as a theorem we inherit.
Research Program
Synthetic data with a realizability label. We will build a paired synthetic corpus from Objaverse objects composed into room-scale layouts. Each view is rendered twice. One render is "representable": diffuse or low-order view-dependent shading, static geometry, a pinhole camera, and no noise. The other is "full": mirrors, glass, strong specular highlights, defocus and motion blur, rolling-shutter-style distortion, and sensor noise. The per-pixel difference between the two renders is the target for a small realizability head. This matches the catalog of simulator shortcomings in Learning to Estimate Scenes from Images, but inverts its role: effects the simulator can toggle on and off become labels.
Gated recurrent refiner. We start from the ReSplat architecture. At each of its $T=4$ steps, the realizability head predicts a map $g \in [0,1]$ per input pixel. It reads the residual maps from all input views plus the current rendered depth. The residual passed into cross-attention becomes $g \cdot r$, where $r$ is the rendered-minus-observed error.
Self-supervised correction on real video. On real video without labels, we add a probe signal during training. One input view is held out from the update computation. The gate is rewarded when the gated update reduces error on that held-out view, and penalized when it admits residuals whose correction raises held-out error. This cross-view falsification test needs no ground truth, so the gate can adapt to real optics that the simulator missed.
Training data. Training mixes the synthetic corpus with RealEstate10K and DL3DV, as GS-LRM co-trains on Objaverse and RealEstate10K. The initial Gaussians come from DepthSplat, matching ReSplat's setup.
Inference cost. The probe is used only in training. Inference adds one small head per step, and we will report the latency overhead against ReSplat's 100×-over-3DGS speed.
Baselines
Do-nothing reference. Single-pass DepthSplat with no refinement: Swin-based multi-view stereo fused with Depth Anything V2 features. It shows how much any refinement buys. The file reports DepthSplat as nearly 2 dB PSNR above MVSplat on DL3DV, but gives no absolute PSNR.
Strongest existing method. Ungated ReSplat. It is the only method in the file that already closes the loop on rendering error, and it reports beating DepthSplat by 3.5 dB PSNR on DL3DV 8-view synthesis. No absolute DL3DV PSNR for ReSplat was found on Lacuna. We will therefore retrain it ourselves on identical data, including our synthetic corpus, with the same step count, GPU budget and tuning sweep.
Post-hoc refiner. Fuse-and-Refine applied to GS-LRM. It is reported to improve RealEstate10K PSNR by about 2.37 dB over GS-LRM, and is the strongest alternative that does not iterate.
Per-scene optimization reference. Zip-NeRF's 31.22 PSNR on DL3DV-140, as reported in DL3DV-10K, and 3DGS fine-tuning initialized from DepthSplat for 500 and 4,000 iterations. These are an upper-quality, high-cost reference, not a like-for-like comparison.
Ablations isolating the key ingredient. Each replaces the learned realizability map $g$ while keeping the architecture and budget fixed:
- a magnitude gate, i.e. a robust loss such as Charbonnier or trimmed residuals;
- a confidence gate using backbone confidence after the audit's two-parameter power-law recalibration;
- $g$ trained on synthetic labels only, without the probe signal;
- $g$ trained with the probe only, without synthetic labels.
What counts as beating ungated ReSplat. A statistically reliable reduction in depth error and floater count on DL3DV scenes annotated as reflective or transparent, with PSNR on non-annotated scenes within 0.2 dB of ReSplat. The magnitude and confidence gates must not match that result.
Why not just…?
Why not just run a few hundred steps of per-scene 3DGS optimization on the feed-forward output? Gradient descent on the photometric loss has exactly the flaw we target: it will happily place floaters to explain a reflection. It also costs minutes, whereas ReSplat reports surpassing 4,000-iteration 3DGS in four learned steps. We include 500- and 4,000-iteration fine-tuning, stratified by the DL3DV annotations, to test whether optimization overfits unrealizable pixels more than our gate does.
Why not just use a robust loss that down-weights large residuals? This is the real threat. If geometric errors are usually small and optical ones large, magnitude alone would separate them. Our hypothesis is that early in refinement, geometric misplacements produce the largest residuals, especially at depth discontinuities, so trimming them slows correction. The magnitude-gate ablation settles this directly. If it matches the learned gate, we will report that the cheap version suffices.
Why not just weight residuals by recalibrated confidence? The calibration audit shows that global power-law recalibration cuts median overconfidence to 1.11×. It also shows scene-level drift remains and the ordering of confidences, not just their scale, is where downstream weighting goes wrong. The confidence-gate ablation measures whether this global fix is enough per pixel.
Why not just train on more real data such as DL3DV-10K? Real data does not make Gaussians able to represent mirrors; DL3DV-10K itself catalogs reflections and transparency as persistent failure cases. Real data also provides no label for which residual is unrealizable. We still train on real data; the question is whether the gate adds value on top of it.
Proposed Experiments
Benchmarks and metrics.
- DL3DV-140 and DL3DV 8-view synthesis, stratified by the dataset's reflection and transparency annotations;
- RealEstate10K and ACID for generality;
- ScanNet for metric depth, which the file lists among the audit's evaluation sets;
- held-out synthetic scenes where the realizable/unrealizable split is known exactly.
We report PSNR, SSIM and LPIPS on novel views, and depth Abs Rel on ScanNet. Floaters are counted as Gaussians with opacity above a threshold lying off the ground-truth surface by more than a set fraction of scene depth. On synthetic scenes we report gate accuracy against the known split.
Decisive check 1. On annotated reflective and transparent DL3DV scenes, the gated refiner should reduce floater count and depth error relative to ungated ReSplat, while PSNR on training views rises less. That pattern is the signature of declining to fit what cannot be fit.
Decisive check 2. On synthetic scenes, the gate should have high agreement with the true unrealizable residual. It should also beat the magnitude and confidence gates on the pixels where they disagree.
Decisive check 3. Remove the synthetic realizability labels and keep only the probe objective. If performance on real scenes holds, the synthetic-to-real contribution is weaker than claimed, and the proposal reduces to cross-view validation, which we would report as such.
References
- ReSplat: Learning Recurrent Gaussian Splats
- Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting
- Analysis of a greedy active learning strategy
- Localization of VC Classes: Beyond Local Rademacher Complexities
- Learning from Synthetic Data: Limitations of ERM
- Learning to Estimate Scenes from Images
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
- DepthSplat: Connecting Gaussian Splatting and Depth
- DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision