Clamp, Solve, Then Hallucinate: Boundary-Anchored Depth Completion for Sparse-View Gaussian Splatting

When a scene is rebuilt from only two or three photographs, many surfaces are visible in just one of them. A fast feed-forward reconstructor that finds depth by matching pixels across views has nothing to match in those regions, so it leaves holes or smeared geometry there. The usual remedy calls in a large generative model, meaning an image or video generator trained on huge photo collections, and has it invent the missing content. That model is expensive, and nothing stops it from also disturbing the parts of the scene that were measured reliably.

We propose to split the missing geometry into two parts. The first is a smooth part that a fixed numerical solver computes exactly from the reliable depth around the hole. The second is a residual (the leftover correction) that a small learned generator predicts, and it is zero on the hole's boundary by construction. The reliable geometry therefore cannot drift, the seam between measured and invented surfaces is continuous, and the learned model only has to supply what the smooth solution misses. We expect that leftover to be sparse and local: creases, edges and depth jumps.

The idea comes from physics-informed operator learning. There, fixing the known operator in the architecture and learning only the residual gives exact boundary satisfaction and large parameter savings. We move that idea into the regime where sparse-view reconstructors fail.

Core Claim

In each input view, a feed-forward sparse-view reconstructor such as MVSplat produces a per-pixel depth and a matching confidence. MVSplat places one 3D Gaussian (a small colored blob used as the scene primitive) per pixel and sets its opacity from that confidence.

Call $\Omega$ the set of pixels whose confidence is low, and $d_{\text{obs}}$ the trusted depth on the boundary of $\Omega$. We replace the depth inside $\Omega$ by the solution of an edge-aware elliptic boundary-value problem:

$$ \nabla\cdot\big(c_\theta(x)\,\nabla d(x)\big) = f_\theta(x) \ \text{ in } \Omega, \qquad d = d_{\text{obs}} \ \text{ on } \partial\Omega . $$

Here $d$ is the completed depth. $c_\theta$ is a positive conductance map that tells the solver where smoothing may cross, so dropping it near image edges lets depth jump at occlusion boundaries. $f_\theta$ is a source term that injects curvature and shape. A small conditional generator with parameters $\theta$ predicts both maps from the image and the cost-volume features. A fixed sparse linear solver computes $d$ and is differentiated implicitly.

With $f_\theta = 0$ and $c_\theta = 1$ the method reduces to harmonic interpolation of the hole. Because the boundary condition is clamped rather than penalized, the completed depth equals the trusted depth on $\partial\Omega$ exactly, whatever the network outputs.

The claim has three parts. First, this structure fills single-view-visible regions better than a network of equal size that predicts depth directly. Second, it reaches that quality with a generator one to two orders of magnitude smaller than the monocular-depth or diffusion models used today. Third, the benefit comes specifically from exact clamping plus the solver, not from extra capacity.

The generator is generative in the literal sense. It is trained with flow matching, a denoising-style objective that learns to transport noise into samples, over the pair $(c,f)$. It can therefore output several plausible completions of an ambiguous hole, and every one of them respects the boundary.

Why This Intersection, and What Is Already Done

The operator-learning literature supplies the method. Structure-Preserving Operator Learning encodes inputs onto finite-element degrees of freedom and clamps the boundary ones. It reports zero boundary error, against 18–43% for Fourier neural operators and DeepONets, and uses 3.5× fewer parameters. FINN splits PDEs into fixed numerical flux structure plus learned kernels, with roughly 90% fewer parameters. When the boundary condition was shifted, FINN was the only model that adjusted instead of reverting to its training pattern. Climate Modeling with Neural Diffusion Equations and physics-informed neural ODEs follow the same rule of fixing the known operator and learning the residual. Elliptic PDE learning is provably data-efficient explains why elliptic structure is cheap to learn: its solution operators have hierarchically decaying Green's functions. The same work warns that the guarantees weaken for discontinuous, wave-like problems. We take that warning seriously. The learned conductance $c_\theta$ is our answer to depth discontinuities, and whether it suffices is an empirical question the experiments test.

Sparse-view reconstruction supplies the failure mode. MVSplat hard-codes epipolar geometry, the constraint that a point seen in one image must lie along a known line in another, through plane-sweep cost volumes. It reaches 26.39 dB PSNR on RealEstate10K with 12M parameters, but it leaves holes where no second view sees a surface. That is precisely $\Omega$. RegNeRF regularizes unobserved depth with a soft smoothness loss and removes floaters, but it blurs structure. We share its assumption that natural surfaces are mostly smooth, but we impose it as an exact solve with learned exceptions rather than a penalty.

The generative-prior line motivates the work and also defines what we must beat. GS-GS refines renders from unseen poses with Stable Diffusion plus DPT depth losses. CAT3D and FlowR synthesize hundreds of views with large multi-view generators. OracleGS down-weights hallucinations that a VGGT attention oracle flags, which amounts to admitting that unconstrained generation corrupts geometry. GIFSplat is the closest structural relative. It predicts residual Gaussian updates in a feed-forward loop using cues from a frozen 2D enhancer, but its residuals are unconstrained and nothing protects well-measured regions.

None of these methods separates the fill into a solver-computed part and a learned part that vanishes on the boundary. That separation is what we add.

We are deliberately not proposing the "obvious" combination the file suggests, namely a Laplace–Beltrami smoothness loss on Gaussians. That is a soft regularizer of the RegNeRF kind, and a total-variation baseline would likely match it.

Research Program

We build the method on MVSplat's released pipeline. For each context view, we threshold cost-volume confidence to get $\Omega$ and run a few conjugate-gradient iterations of the multigrid solve on the pixel grid. At 256×256 this is a small sparse system, so latency should stay within the feed-forward regime. We then back-project the completed depth to Gaussian centers and lift the opacity of filled pixels, since those surfaces are real even though they were matched only once.

The generator is a compact U-Net in the 1–5M parameter range. It predicts $(c_\theta, f_\theta)$ from image features and the cost volume.

Supervision comes free from view dropping. On RealEstate10K and DL3DV training clips, we reconstruct with four to six context views to get confident depth over most pixels. We then keep only two views and train the generator so the clamped solve reproduces the denser depth inside the newly unmatched regions. The flow-matching objective acts on $(c,f)$, and a rendering loss on held-out views trains end to end through the solver.

A second phase tests whether the same decomposition helps a big generative pipeline. We feed our completed geometry as the initial reconstruction to a FlowR-style refinement and ask whether better-anchored initial geometry reduces the drift FlowR inherits from poor initializations.

We scope the claim carefully. The method addresses surfaces seen in exactly one view, which is the dominant hole type in two- to three-view feed-forward reconstruction. Regions seen by no view need view synthesis, and we do not claim to solve them.

Baselines

The naive reference is MVSplat unchanged, which leaves low-confidence pixels as reported. It reports 26.39 dB PSNR on RealEstate10K (MVSplat). We retrain it on our data splits so all methods share resolution, context-view selection and training budget. Beating it means a clear gain on the masked single-view-visible region, not just on whole images, where holes are a small fraction of pixels.

The strongest feed-forward competitor with generative cues is GIFSplat. It reports up to +2.1 dB over AnySplat and FLARE, but the file gives no absolute RealEstate10K or DL3DV PSNR, so no reported result on Lacuna fixes a target number. If code is available, we run it under the same splits, and we count its parameters including the DIFIX enhancer.

The strongest generative reference is FlowR, which reports 22.43 dB PSNR on DL3DV with 12 views. That is a denser-input, heavier-compute setting. We use it as an upper reference for quality and a lower reference for cost, and we compare at matched view counts where we can run it.

For soft geometric regularization, we reimplement a RegNeRF-style depth-smoothness loss on the same backbone. RegNeRF itself reports 18.89 dB on 3-view DTU as a per-scene optimizer.

A monocular-depth fill aligns an off-the-shelf monocular depth network to the confident pixels by least-squares scale and shift, then substitutes it inside $\Omega$. The file cites no specific monocular-depth paper beyond DPT's use in GS-GS, so we report its size alongside its accuracy.

Three ablations isolate the key ingredient. The first gives the same generator the same parameter budget but has it predict depth directly, with no solver. The second replaces the hard clamp with a boundary-matching loss. The third drops the learned terms, leaving pure harmonic fill with $f=0$ and $c=1$.

Beating the field means three things. On masked depth error and masked PSNR we must exceed MVSplat, the soft-loss variant and the direct-prediction ablation. Seam discontinuity must be exactly zero at the boundary and low just inside it. And we must at least match aligned monocular fill while using a far smaller network.

Why not just…?

Why not just align a monocular depth network to the confident pixels and paste it in? This is the most serious threat, and it may well match us on accuracy. Monocular depth is only defined up to scale and shift. A single global fit leaves visible seams where it meets cost-volume depth, and GS-GS still needs a diffusion loop on top of DPT depth. The experiment settles it directly: we measure seam discontinuity and masked depth error against this baseline. We also run a hybrid in which the monocular prediction enters only as an input to $f_\theta$ inside our clamped solve. If the pasted monocular fill wins outright, the honest conclusion is that clamped solving is best used to blend foundation-model depth, not to replace it, and we will report it that way.

Why not just generate dense views with a large multi-view model, as CAT3D or FlowR do? CAT3D acknowledges that its generated views are slightly inconsistent and leave floaters. FlowR needs about 91K tokens on an H100 and roughly 1.5 minutes for 200 views, and it depends on the quality of its initial reconstruction. Our phase-two experiment tests the complementary claim: that anchored initial geometry improves FlowR-style refinement, rather than claiming our method replaces it.

Why not just add a smoothness or total-variation loss on depth? RegNeRF shows that this suppresses floaters but blurs structure, and SPON shows that soft penalties leave large boundary residuals. The soft-loss and harmonic-only ablations measure how much of our gain comes from exact clamping versus learned conductance. If harmonic fill alone captures most of it, the paper becomes a simpler and still useful result.

Why not just filter hallucinations with an uncertainty oracle, as OracleGS does? Down-weighting tells the optimizer to ignore bad regions but supplies no geometry for them, and it runs SEVA plus VGGT. We will report OracleGS's compute alongside ours. We note that no Mip-NeRF 360 numbers for it are on Lacuna, so we can compare only where we can run it.

Decisive Checks

The central test is the direct-prediction ablation at matched parameters. We sweep generator size from 0.5M to 20M parameters and plot masked depth error. The hypothesis predicts that the solver-backed model saturates at a small size while the direct predictor needs far more capacity to approach it. If the two curves coincide, the physics-informed claim is false.

The second test targets discontinuities. We stratify hole pixels by distance to image edges. If learned conductance fails at occlusion boundaries, which is where elliptic theory predicts trouble, the gains should vanish near edges and survive in smooth interiors.

The third test checks transfer. We evaluate zero-shot on DTU and ACID, following MVSplat's protocol. FINN's boundary-shift result suggests that clamped structure should transfer better than a black-box fill. A lack of transfer would count against the idea.

Throughout, we report PSNR, SSIM and LPIPS on full images and on single-view masks, together with depth error against dense-view reconstructions, seam discontinuity, parameter count and per-scene latency.