When a 3D scene is reconstructed from only three to ten photographs, the model has far too much freedom. It learns to reproduce the training photos exactly while placing geometry in the wrong places: foggy "floaters" in front of the cameras, flattened surfaces, and needle-shaped blobs. Multi-view stereo networks address exactly this kind of ambiguity. They match image features across views at many candidate depths and store the results in a cost volume, a grid of matching scores over pixels and depth hypotheses. Full cost volumes are expensive, because memory grows with resolution times the number of depth hypotheses. The best current sparse-view method that uses them, MVPGS, therefore runs a heavy pretrained stereo network once, offline, and then holds the 3D model to that network's depth maps, including its mistakes.
This note proposes a different arrangement. At every training step, the partially trained scene model supplies the depth guess. A tiny cost volume, only a handful of depth hypotheses per pixel, is built around that guess from frozen image features of the other input views. The narrow window is what keeps memory low, and it is the same trick Effi-MVS uses to fit high-resolution stereo into about 3 GB. The window also acts as a live, cross-view check on the geometry the model is currently learning. The claim is that verifying geometry against correspondence inside the optimization loop resists sparse-view overfitting better than freezing an external depth map. It should also cost a fraction of the memory of any full cost volume, provided the narrow window is protected from the known failure of coarse-to-fine search, which is locking onto a wrong early guess.
Core Claim
We use 3D Gaussian Splatting (3DGS), which represents a scene as millions of small colored ellipsoids that are rasterized into images. For a training view $v$ and pixel $p$, the Gaussians produce a rendered depth $d_p$. They also produce a distribution of compositing weights along the ray, whose spread $\sigma_p$ measures how unsure the model is about where the surface lies.
We place $K$ depth hypotheses (with $K \approx 8$) in the interval $d_p \pm c\,\sigma_p$. We warp frozen features from the other input views to each hypothesis, compute group-wise correlation, and apply a softmax to obtain a matching distribution $m_p$ over the window. The Gaussians' own ray weights, binned onto the same hypotheses, give $w_p$. The regularizer is
$$\mathcal{L}_{\text{match}} = \sum_{p} \kappa_p \, \mathrm{KL}(m_p \,\|\, w_p),$$
where $\kappa_p$ is a matching confidence (the ratio of the top peak to the second peak of $m_p$). Its purpose is to switch the loss off on textureless, specular, or occluded pixels, where correlation is unreliable.
The memory cost per step is proportional to $H \cdot W \cdot K$, not $H \cdot W \cdot D$ for a full search range of $D$ hypotheses (typically 48 to 192). The same signals also drive where Gaussians are added and removed. Gaussians that carry weight at depths the matching rejects are pruned. Gaussians are added where the matching peak lies far from any existing weight.
The known weakness of narrow windows is the one CVP-MVSNet inherits from its coarse stage: if the first guess is wrong, a residual search cannot escape. In our setting the "coarse stage" is a floater sitting in front of the camera, which is precisely the sparse-view failure we are trying to fix. We address it with scout pixels. In each step, a small random subset (about 1%) of pixels gets a full-range search. Wherever a scout finds a confident peak far outside the current window, that pixel's window is re-centered on the peak. Memory stays essentially constant, and the scouts give the optimization a way out of local traps.
Relation to Prior Work
Three groups of prior work bear on this proposal, each in a different role.
The work that motivates it is MVPGS. MVPGS shows that multi-view stereo depth from MVSFormer rescues sparse-view 3DGS on DTU, raising PSNR from 14.07 for plain 3DGS to 20.26 with 3 views. But its Consistent Structure Loss ties the Gaussians to a fixed, externally computed depth. The paper's own open issues are memory-heavy preprocessing and inherited stereo errors. We keep MVPGS's premise that correspondence is the right prior. We change where correspondence is computed (inside the loop, around the model's current geometry) and how much of it (a window rather than a full volume).
The papers that supply the method are Effi-MVS and CVP-MVSNet. Effi-MVS uses dynamic windows around the current depth estimate. CVP-MVSNet uses residual search with about 8 hypotheses, which gave 6× less memory than Point-MVSNet. MVSFormer++ supplies the idea of frozen DINOv2 features for matching. All three were built and evaluated with dense, well-chosen neighboring views, and all output depth maps rather than a renderable scene. Our proposal moves their memory trick into the sparse, per-scene optimization regime they never tested.
The work that constrains the design shows the limits of regularizers that use no correspondence. DropGaussian and DropoutGS are strong and cheap, but both acknowledge that they cannot resolve ambiguity under wide baselines, because they have no cross-view mechanism. Effective rank regularization fixes the shape of individual Gaussians but not where they sit. VGOS and InfoNeRF regularize density through smoothness or ray entropy. The ray-entropy prior is close in spirit to ours, since both push the ray weights toward one peak. The difference is that InfoNeRF decides where the peak goes by self-consistency alone, whereas matching evidence from the other views decides it here. FixingGS and ProSplat use 2D diffusion priors instead. These add heavy compute and, per ProSplat, can produce sharp but geometrically flat textures. We use no generative prior.
Research Program
We will build the regularizer as a plug-in to a standard 3DGS trainer. It has four parts:
- a frozen feature extractor (a DINOv2 backbone, with a lightweight pretrained stereo feature network as an alternative);
- window construction from the rendered depth and weight spread;
- group-wise correlation against the two to four nearest input views;
- the confidence-gated loss, together with the matching-driven rules for adding and pruning Gaussians.
The analysis part turns the open question of how few hypotheses suffice under sparse views into a measurement. We will sweep the hypothesis count $K$ over 1, 2, 4, 8, 16, and a full range. For each setting we will record reconstruction quality, floater mass, and peak memory. We will do this with and without scouts, and at 2, 3, 6, and 9 input views. The result is a memory-versus-overfitting curve with a location for the knee.
A third component is controlled corruption. We will feed deliberately wrong depth, generated by offsetting stereo depth on selected regions, both to MVPGS-style anchoring and to our in-loop method. This tests the central argument that live verification recovers from bad geometry where frozen supervision cannot.
Baselines
The naive reference is plain 3DGS trained only on the photometric loss. MVPGS reports 14.07 PSNR for it on DTU with 3 views. Anything we build must clear this by a wide margin to matter.
The strongest correspondence-based competitor is MVPGS itself, with offline MVSFormer depth, forward-warped pseudo-views, and monocular depth. It reports 20.26 PSNR on DTU with 3 views. We will run it from its released configuration on identical view splits and resolutions. We will report its peak memory including the MVSFormer preprocessing stage, because that stage is part of its real cost. Beating MVPGS means higher PSNR and lower depth error at matched training time, with lower total peak memory.
The strongest correspondence-free competitor is DropGaussian. It reports 20.76 PSNR on LLFF with 3 views, with essentially no added cost. DropoutGS is a second competitor in this group; it reports 19.35 on LLFF with 3 views and 20.22 on DTU. Because dropout and our loss act on different things, we will run DropGaussian both as a rival and as a base. We will add our loss on top of it and give both variants the same number of iterations and the same hyperparameter-search budget, 20 configurations each, tuned on held-out scenes. Beating it means a gain over DropGaussian alone on LLFF, and a larger gain on the wide-baseline splits where DropGaussian admits it fails.
For the diffusion-prior family we will use FixingGS, which reports an average of 20.89 dB PSNR over 3, 6, and 9 views on DL3DV-10K. It serves as a quality ceiling, not a strict target. We will count it a success if we come within about 0.5 dB at a small fraction of its training compute. We found no reported LLFF or DTU number for FixingGS on Lacuna.
Two ablations isolate the key ingredient. The first is a full-range in-loop cost volume, which shows what the narrow window costs in quality and saves in memory. The second is a $K=1$ variant that simply compares features at the rendered depth, and it is treated as a baseline in its own right (see below).
Why not just…?
Why not just use MVPGS's offline depth, which already works? It does work on DTU, and if our method cannot beat 20.26 PSNR there, the proposal fails on its home ground. The argument for doing better rests on the corruption experiment and on wide-baseline splits. In both, an offline depth map is wrong in some regions, and MVPGS's structure loss actively pulls the Gaussians toward the error. If MVPGS matches us even under injected depth errors, then live verification adds nothing, and we will say so.
Why not just apply Gaussian dropout? DropGaussian's 20.76 PSNR on LLFF with 3 views and zero overhead make it a real threat on forward-facing scenes with good view overlap. There, our method may gain little. Dropout's documented weakness is the absence of any cross-view signal under wide baselines. We therefore expect the difference to show up on the DL3DV wide-baseline and 2-view splits, and the stacked DropGaussian-plus-matching run settles whether the two effects are complementary.
Why not just penalize feature mismatch at the rendered depth, with a single hypothesis? This is the most dangerous alternative, because it is cheaper still and resembles common consistency losses. A single-point check can say that the current depth is bad, but not which direction to move. Near a floater, the gradient it provides is local and weak. A window turns rejection into a target distribution. The $K$ sweep answers this directly: if $K=1$ matches $K=8$ in both floater mass and PSNR, the window is unnecessary, and the contribution shrinks to the scouts and the confidence gating.
Why not just use a diffusion prior? FixingGS and ProSplat show strong wide-baseline quality. We do not claim to beat them on perceptual sharpness. Our claim concerns geometric fidelity per unit of compute. Depth error on DTU, where ground-truth scans exist, and training cost will show whether correspondence buys geometry that diffusion does not.
Proposed Experiments
The benchmarks are DTU with 3 views, LLFF with 2, 3, and 6 views, Realistic Synthetic 360° with 4 views, and DL3DV-10K wide-baseline splits with under 30% overlap. We report PSNR, SSIM, and LPIPS on held-out views. On DTU we also report Chamfer distance to the structured-light scans. We measure floater mass as the fraction of rendered opacity lying in front of the ground-truth surface. We also report peak GPU memory, including all preprocessing, and wall-clock training time.
Three checks are decisive:
- In-loop versus offline correspondence. On DTU with 3 views, the method should exceed MVPGS's 20.26 PSNR and reduce Chamfer distance, using less total peak memory than MVSFormer preprocessing.
- Recovery from bad geometry. Under injected depth corruption, our reconstruction error in the corrupted regions should fall back toward the clean-input level, while MVPGS's should not.
- The memory-versus-overfitting curve. The $K$ sweep should show a knee at a small $K$, with scouts moving that knee lower. That result would be the first direct measurement of how much hypothesis density sparse-view reconstruction actually needs.
If the knee sits at $K=1$, or if scouts make no difference, we will report that as the main finding.
References
- MVPGS: Excavating Multi-view Priors for Gaussian Splatting from Sparse Input Views
- Efficient Multi-view Stereo by Iterative Dynamic Cost Volume
- Cost Volume Pyramid Based Depth Inference for Multi-View Stereo
- MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo
- DropGaussian: Structural Regularization for Sparse-view Gaussian Splatting
- DropoutGS: Dropping Out Gaussians for Better Sparse-view Rendering
- Effective Rank Analysis and Regularization for Enhanced 3D Gaussian Splatting
- VGOS: Voxel Grid Optimization for View Synthesis from Sparse Inputs
- InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering
- FixingGS: Enhancing 3D Gaussian Splatting via Training-Free Score Distillation
- ProSplat: Improved Feed-Forward 3D Gaussian Splatting for Wide-Baseline Sparse Views