Supervised Trust Gates for Foundation Geometry in Feed-Forward Gaussian Splatting

Feed-forward 3D reconstruction networks take a few photos of a scene and, in one pass of a neural network, output a full 3D model. Here the model is a cloud of small colored Gaussian blobs, a "3D Gaussian Splatting" representation that renders new viewpoints very fast. These networks estimate depth by matching pixels across the input photos. Matching works well on textured surfaces. It fails at object silhouettes, on shiny or blank surfaces, and in regions only one camera sees. The current fix is to borrow geometry from a large pretrained 3D foundation model (VGGT, which predicts a 3D point for every pixel) and add a training loss that pulls the predicted blobs toward those points. That fix helps a lot. It is also indiscriminate: it applies the prior everywhere with equal weight, including places where matching was already right and the prior is wrong.

We propose to make the choice between the two depth sources explicit and per-pixel. A small gate inside the network decides, at each pixel, how much to trust multi-view matching versus the foundation prior. The gate decides which feature stream flows through the network's skip connections, and it also decides where the auxiliary depth loss toward the prior is applied. The key ingredient is that the gate is not left to discover this on its own. During training it is taught directly which source was correct, using a label computed by checking which depth reprojects the input image more faithfully into held-out target views. At test time no target views are needed, because the gate predicts trust from matching-confidence features alone. The claim is that most of the remaining error in prior-regularized feed-forward splatting comes from applying the prior uniformly, and that a supervised trust gate removes that error at depth boundaries and in scenes where the prior is wrong.

What exists and what is missing

The regime is sparse-view feed-forward splatting. pixelSplat predicts Gaussians from image pairs with an epipolar transformer that attends along matching lines. MVSplat replaces that with a plane-swept cost volume, a stack of matching scores over candidate depths. A lightweight 2D U-Net with ordinary skip connections then predicts the remaining Gaussian parameters. We build directly on MVSplat. Its cost volume already yields a per-pixel matching confidence, and its U-Net skips are exactly where an ungated mix of features happens today.

The motivating evidence comes from Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting. That paper shows that depth discontinuities at silhouettes "explode" into torn, floating Gaussians when unprojected. Its remedy, PM-Loss, aligns predicted Gaussian centers to a VGGT pointmap with the Umeyama algorithm (a closed-form rigid-plus-scale fit) and applies a one-directional Chamfer loss (a nearest-point distance between two point sets). It reports at least 2 dB PSNR gain over MVSplat and DepthSplat on RealEstate10K and DL3DV. The same paper also constrains our design. It states that PM-Loss is a training-only loss bounded by pointmap quality, so prior errors propagate into the student. This limitation is what we target. Gen3R takes the opposite route and jointly denoises VGGT geometry with a video diffusion transformer. It shows that generative coupling removes floaters, but at the cost of training a full video DiT. We do not attempt generation of unseen regions. Our "prior" is the foundation pointmap, and we treat the generative-hallucination question as out of scope.

The mechanism is borrowed from Attention Gated Networks. That work multiplies U-Net skip features by a sigmoid map computed from coarser decoder features, raising pancreas Dice on CT-150 from 0.814 to 0.840 for about 7% extra inference time. Its gate is learned only implicitly from the segmentation loss and knows nothing about geometry. The same paper warns that gates can saturate near zero and stall gradients. Depth-completion work such as Gated Cross-Attention Network for Depth Completion and AGG-Net gates image features by sparse depth. It stays in 2D image-to-image completion, never outputs 3D primitives, and never arbitrates between two competing geometry sources.

Our approach differs from these in three ways. The gate is driven by matching confidence rather than image appearance. It is supervised by a reprojection-derived label rather than learned implicitly. It controls both feature routing and the extent of the auxiliary depth loss, so the regularizer stops acting where the prior is unreliable. The first property also covers Flash3D, which anchors Gaussians on a monocular foundation depth. Flash3D cannot correct that anchor when the depth is overestimated, which is the error-inheritance failure in a single-image setting.

Core mechanism

For each context-view pixel $p$, the network has two depth hypotheses. The first, $d_{mv}(p)$, comes from MVSplat's cost volume. The second, $d_{pr}(p)$, comes from the VGGT pointmap after Umeyama alignment to the context cameras. Each has an encoder feature stream, $F_{mv}$ and $F_{pr}$. A gate network sees the cost-volume entropy, the gap between the two depths, and the decoder feature at that scale. It outputs $g(p)\in[0,1]$. Each skip connection then carries $g\,F_{pr} + (1-g)\,F_{mv}$, and Gaussian depth is placed using the same mixture of the two hypotheses.

The gate label $g^*(p)$ is 1 when warping the context pixel into the target views with $d_{pr}$ gives lower photometric error than warping with $d_{mv}$, and 0 otherwise. Pixels where both errors are high, which are likely occluded in the targets, are excluded from the gate loss. The full objective is

$$ \mathcal{L} = \mathcal{L}_{render} + \lambda_g \, \mathrm{BCE}(g, g^*) + \lambda_d \sum_p g(p)\, \big\| \mu(p) - x_{pr}(p) \big\|, $$

The terms are as follows. $\mathcal{L}_{render}$ is MVSplat's usual target-view image loss. BCE is binary cross-entropy. $\mu(p)$ is the predicted Gaussian center from pixel $p$, and $x_{pr}(p)$ is the aligned prior point. The weights $\lambda_g$ and $\lambda_d$ balance the terms. The last term is a gated version of PM-Loss: where the gate says "trust matching," the prior exerts no pull.

To guard against the saturation issue Schlemper et al. report, the gate is initialized at 0.5 and its BCE term is warmed up before the gated depth loss is switched on.

Research program

The build is a modification of the public MVSplat pipeline. It adds a VGGT pointmap preprocessing pass, which runs offline as PM-Loss does, a second shallow encoder for pointmap features, gate modules at each U-Net skip scale, and the label computation above.

The analysis has three parts. The first is aggregate novel-view quality. The second is error localized to depth-boundary bands, defined as pixels within a few pixels of a ground-truth or VGGT depth edge. This band is where PM-Loss claims its gains, and it is where we predict ours concentrate further. The third is robustness to a bad prior. We deliberately corrupt the VGGT pointmaps with depth bias, edge dilation, and random patch swaps, then measure how each method degrades. Our expectation follows from the loss form. A uniformly weighted prior loss should degrade roughly in proportion to corruption, while a gate that learns to reject a disagreeing prior where matching is confident should degrade much less.

We also inspect the gate maps directly, as an interpretable record of where foundation geometry beats multi-view matching. We expect high trust in the prior at silhouettes, low-texture walls, and single-view regions, and low trust on well-textured surfaces.

Baselines

The naive reference point is MVSplat itself, trained with its standard rendering loss and ungated skips. On RealEstate10K, MVSplat reports PSNR 26.39. We retrain it in our codebase so that all methods share data splits, resolution, training steps, and GPU budget, and we report our reproduction next to the published number.

The strongest existing competitor is MVSplat plus PM-Loss from Revisiting Depth Representations. It uses the same VGGT prior through a uniform loss and is the method our gating is meant to supersede. The file gives only "at least 2 dB" improvement and no exact PSNR on RealEstate10K, DL3DV, or DTU, so there is no reported result found on Lacuna for an exact PM-Loss score. We therefore rerun it with identical precomputed pointmaps and the same hyperparameter search budget as our method. Beating it means a higher full-image PSNR and a lower LPIPS on RealEstate10K and DL3DV, a larger margin inside boundary bands, and a clearly flatter degradation curve under prior corruption. A full-image tie with a boundary-band and robustness win would still support the core claim. A loss on all three would refute it.

Two ablations isolate the key ingredient. The first, learned fusion without gate supervision, keeps the dual encoders and the gates but drops the BCE term, so the gates learn only from rendering. The second, standard Schlemper attention gates, puts gates on MVSplat's U-Net skips driven by decoder features alone, with no prior and no geometry signal. A third control gives the PM-Loss baseline the dual-encoder input, so that our gains cannot be attributed to extra parameters.

Why not just…?

Why not just feed the VGGT pointmap into MVSplat as extra input channels and let the U-Net learn the fusion? This is the most serious threat. A large enough network might learn implicit trust from the rendering loss alone. The ablation without gate supervision tests exactly this. If it matches the supervised gate, we will report that the explicit label is unnecessary and that the contribution reduces to "add the prior as input." Our reason to expect a gap is that the rendering loss gives only weak, view-averaged credit assignment per pixel. The reprojection label instead tells the gate directly which source was right, which matters most in the boundary bands where the two depths disagree sharply.

Why not just weight PM-Loss by a fixed confidence heuristic, such as cost-volume entropy or VGGT's own confidence? This is cheaper and could plausibly capture much of the benefit. We include it as a baseline at the same budget. A fixed heuristic cannot detect a prior that is confidently wrong, because VGGT's confidence does not know its own bias. It also cannot detect matching that is confidently wrong, as on specular surfaces. The corruption stress test, where the injected prior errors keep VGGT's confidence unchanged, is designed to separate the two. If the heuristic tracks our gate there as well, the learned gate is not justified.

Why not just use PM-Loss as published, since it already gives at least 2 dB? Its own authors note that it inherits pointmap errors. On clean benchmarks where VGGT is good, PM-Loss may already capture most of the available gain, and we may see only small aggregate improvements. That is why the decisive evidence is the boundary-band and corrupted-prior analysis rather than headline PSNR alone.

Why not just use a generative model such as Gen3R to fix geometry? Gen3R addresses a different problem: hallucinating unseen content from one view. It requires training a video diffusion transformer. Its 1-view RealEstate10K PSNR of 20.51 is not comparable to the two-view reconstruction setting here. We do not claim to compete on unobserved-region synthesis.

Proposed Experiments

We train on RealEstate10K and DL3DV with two context views, following the MVSplat and PM-Loss protocols. We report PSNR, SSIM, and LPIPS over full images and over boundary bands, at both standard and extrapolation baselines, since PM-Loss reports its largest gains on extrapolation.

For geometry, we evaluate zero-shot on DTU using accuracy, completeness, and overall Chamfer distance from Gaussian centers against the structured-light scans of Large Scale Multi-view Stereopsis Evaluation. Since no numeric DTU result for MVSplat or PM-Loss is reported on Lacuna, all DTU numbers come from our own reruns.

The corruption sweep applies three error types at five strengths each to the pointmaps of both PM-Loss and our method, and plots degradation against strength. We measure gate calibration by comparing predicted $g$ against held-out reprojection labels on the test split. We also report overhead in inference time and memory against MVSplat's published speed of about 22 FPS.

The decisive checks are three. The first is whether the supervised gate beats both the unsupervised-gate ablation and the confidence-weighted PM-Loss inside boundary bands. The second is whether its degradation slope under corrupted priors is substantially below uniform PM-Loss. The third is whether these gains survive on DTU geometry rather than appearing only in rendered PSNR.