Fast 3D reconstruction models can now turn a handful of photos into a 3D scene in under a second. They do this by predicting a cloud of small colored blobs called Gaussian splats in a single network pass, without fitting each scene by gradient descent. Their weakness is everything the cameras did not see. Behind a doorway or past the edge of the frame, the model has nothing to triangulate from, so it leaves holes or blur. The usual remedy is a diffusion model, an image generator trained to turn random noise into realistic samples step by step, which imagines the missing content. Run naively, that remedy brings back per-scene optimization that takes minutes. It also yields imagined pixels that disagree from one viewpoint to the next.
We propose that the generator should not imagine pixels at all in this loop. It should imagine a line sketch: the straight edges where walls meet, door frames, table edges, and stair noses that define a scene's structure. Line sketches are sparse, so a few-step generator can produce them quickly. Unlike imagined pixels, an imagined 3D line can be checked: if it is real structure, sketches generated from two different novel viewpoints should triangulate to the same 3D segment, and it should join up with lines actually detected in the input photos. Only lines that pass this cross-view vote are allowed to place new Gaussians in empty space and to steer a forward-only refinement loop. The result is a sub-second-to-seconds reconstructor that completes unobserved structure with an explicit consistency check, and it needs no test-time gradient descent.
Core Claim
In a recurrent feed-forward Gaussian reconstructor, the generative prior should act on 3D-verifiable line structure rather than on 2D appearance. Filtering generated lines by multi-view triangulation consensus before they seed or correct Gaussians should give better geometry and cross-view consistency in unobserved regions than pixel-space generative feedback, at comparable runtime. The claim is specific to unobserved regions. In observed regions we expect line cues to help mainly in textureless or repetitive scenes, and we treat that as a separate, smaller effect.
Position Relative to Prior Work
The closest existing system is GIFSplat. It predicts poses and coarse Gaussians, then applies three forward-only residual updates, and injects a generative prior by rendering a novel view, enhancing it with a frozen diffusion image restorer, and distilling the pixel difference into Gaussian-level cues. It supplies our refinement mechanism. We differ in what the generator produces and in how its output is admitted. GIFSplat's cue is an appearance delta with no 3D consistency test. Ours is a line sketch that must survive triangulation across generated views before it can influence the scene.
ReSplat: Learning Recurrent Gaussian Splats also supplies method. It shows that four rendering-error-driven recurrent steps beat single-pass prediction, but its feedback comes only from visible input views, so it cannot add content where no Gaussian exists. This motivates our explicit seeding step. DepthSplat and GS-LRM are the single-pass reference points: fast and strong on observed content, with no mechanism for extrapolation.
Two per-scene methods motivate the structural angle and set quality ceilings. MG-SLAM shows that detected 3D lines can bound planes and fill occluded surfaces with interpolated Gaussians. It does this by optimization and only under the assumption that structure follows three orthogonal axes. We keep the idea that lines drive completion but replace the axis assumption with a learned generator and a consensus filter. G4Splat tames video diffusion with plane masks and a 3D visibility grid, also by per-scene optimization. We borrow its visibility grid for evaluation but not its optimization.
NeuSD found that guiding a diffusion model with rendered normal maps separates geometry from appearance better than guiding it with RGB. This is the key piece of evidence that geometric rather than pixel guidance matters. It also gives the strongest competing design, normal-map feedback, which we test head-on.
On the line side, UPAL supplies a cheap joint point-line detector whose line head predicts a distance-to-nearest-line field. That field is the representation our generator outputs. LightGlueStick supplies line matching between input views and the evidence that lines matter in repetitive scenes: on 7Scenes Stairs, pose accuracy rises from 59.8% with points only to 79.1% with lines. Both papers also constrain the design. They document that line endpoints slide across viewpoints, so our consensus test compares infinite-line direction and offset plus segment overlap, never endpoint positions.
Research Program
Base reconstructor. We start from a GIFSplat-style pipeline, a coarse Gaussian initializer followed by a weight-shared residual head run for $T=3$ steps, and retrain it on RealEstate10K and DL3DV. To the per-view encoder we add UPAL's line distance fields from the input images. This is a small, separate change that we ablate.
Line-sketch generator. We train a conditional diffusion model, distilled to a few denoising steps so its latency is comparable to GIFSplat's frozen enhancer. Its input is three maps rendered at a novel camera from the current Gaussian state: the partial line distance field, depth, and an opacity map. Its output is a completed line distance field. Training targets come from running UPAL on real held-out frames of the training videos. The generator therefore learns where structural edges continue beyond what the input views covered, supervised by real future frames rather than by synthetic data.
Consensus lifting. At test time, at each refinement step we render $K$ novel views, typically four, that are placed to cover regions the visibility grid marks as unseen. We generate sketches for these views, extract segments from them, and lift each segment to a 3D line candidate. The lift uses the rendered depth where depth exists and pairwise triangulation between generated views where it does not.
A candidate line $\ell$ is accepted when its consensus score clears a threshold $\tau$:
$$s(\ell) = \sum_{k=1}^{K} \mathbb{1}\left[\, d_k(\ell) < \epsilon \,\right] + \lambda \, \mathbb{1}\left[\, \ell \text{ meets an input-view line} \,\right] \ge \tau .$$
Here $d_k(\ell)$ is the angular-plus-offset distance between $\ell$ projected into view $k$ and the nearest line in that view's generated sketch. The tolerance $\epsilon$ is set to absorb endpoint sliding. The bonus weight $\lambda$ rewards connection to observed structure, following the principle that visible structure should be fixed before anything is hallucinated, as argued by Yiming Li et al..
Seeding and feedback. Accepted lines that lie in empty space spawn a thin band of new Gaussians along them, which gives the residual head something to move and grow. For every accepted line, the gap between it and the depth or normal discontinuities in the current render is encoded as an extra feedback channel for the residual head. This channel plays the role GIFSplat gives its pixel delta.
Training. Everything is trained end to end through the forward-only loop on real video. The rendering loss on held-out frames includes frames that fall outside the input views' frustums, so the network is rewarded for completions that match what the camera later actually saw.
Baselines
Do-nothing reference. The GIFSplat-style recurrent model with its generative branch removed, run on our retrained base reconstructor. It shows how much of the unobserved-region error any prior closes.
Strongest feed-forward competitor. GIFSplat itself, with its pixel-delta diffusion cue. It is the only cited method that already combines forward-only refinement with a generative prior. The file reports it improving PSNR by up to +2.1 dB over AnySplat and FLARE on RealEstate10K, DL3DV, and DTU at second-scale inference. No absolute GIFSplat PSNR on these benchmarks was found on Lacuna. We will retrain it on identical data, at the same resolution, with the same number of refinement steps, the same number of generator calls per step, and a matched GPU-hour budget. Beating it means a statistically reliable gain in masked unobserved-region depth error and Chamfer distance, with no loss of observed-region PSNR, at no more than 1.5 times its wall-clock time.
Strongest single-pass baselines. ReSplat (recurrent) and DepthSplat. The file reports ReSplat beating DepthSplat by 3.5 dB PSNR on DL3DV 8-view synthesis; no absolute value was found on Lacuna. These two show whether the gains survive against strong reconstruction without any prior.
Per-scene quality ceiling. G4Splat, which reports a Chamfer distance of 6.61 and a PSNR of 23.90 on Replica with 5 views. We do not expect to beat it. Success means closing a stated fraction of the gap between the do-nothing reference and G4Splat on those two numbers while running orders of magnitude faster.
Key-ingredient ablations. Three ablations isolate the proposal's core. First, generated sketches injected without the consensus filter. Second, consensus filtering with endpoint-based rather than infinite-line distances. Third, the full pipeline with the generator replaced by a normal-map generator that uses an analogous depth-reprojection consensus test.
Why not just…?
Why not just distill normal maps instead of lines, as NeuSD suggests? This is the most serious threat. NeuSD already shows that geometric guidance beats RGB guidance, and dense normals carry more information than sparse lines. Our bet is that lines win on verifiability and cost: a 3D line can be triangulated from two sketches and matched to detected input lines, whereas checking dense normals across views requires consistent hallucinated depth, which is exactly what is in doubt. The third ablation above settles this, because it gives the normal generator the same consensus machinery. If normals match lines on unobserved Chamfer distance and cross-view consistency, we will report that the consensus filter is the contribution and that the line representation is not.
Why not just add line features to the encoder and skip the generator? Line cues at the input could explain most of the gains in textureless or repetitive scenes, which is what Gap 4 in the file suggests. Because they act only on observed pixels, they cannot fill space no camera saw. We evaluate this variant and report every metric split by the visibility mask. If it matches the full model inside unobserved regions, the generative component is not earning its cost.
Why not just inpaint the missing views in 2D, then run GS-LRM? ReSplat (degradation) reports that restoring images in 2D and then reconstructing causes multi-view inconsistency, and G4Splat reports that unguided diffusion produces conflicting geometry across angles. We still run this pipeline with GS-LRM as a baseline, since it is cheap. It is scored on the cross-view consistency metric below, where we expect it to fail most visibly.
Why not just optimize per scene with line losses, as in MG-SLAM? That approach is slower by design, and it relies on three orthogonal axes. We treat per-scene methods as the quality ceiling, not as competitors. To check the generality claim, we evaluate separately on the non-orthogonal and outdoor scenes in DL3DV.
Proposed Experiments and Decisive Checks
Training uses RealEstate10K and DL3DV. We evaluate with 2 and 5 input views on RealEstate10K and DL3DV-140. For geometry against ground-truth meshes we use Replica, and for real indoor scenes ScanNet, both zero-shot. Novel cameras are deliberately placed to extrapolate beyond the input frustums.
Every metric is reported separately for observed and unobserved regions. The split is defined by a G4Splat-style visibility voxel grid built from ground-truth depth. In both regions we measure PSNR, SSIM, and LPIPS, plus depth absolute relative error and Chamfer distance on Replica.
Two structural metrics complement these. Line recall measures how many ground-truth structural edges, detected by UPAL on real held-out frames, are reproduced in renders. Cross-view consistency is the reprojection disagreement of rendered depth discontinuities between pairs of novel views, which directly measures the seam problem raised by G4Splat. We also report wall-clock time on one GPU and the number of Gaussians.
The decisive check is a single comparison. In unobserved regions on Replica and DL3DV, full model versus GIFSplat versus the normal-map variant, on Chamfer distance and cross-view consistency at matched runtime. The proposal succeeds if the full model wins on both against both. It also needs a sanity check: the no-consensus ablation must lose clearly, otherwise the filter is decorative.
A secondary check is 7Scenes Stairs and textureless ScanNet subsets. There, the encoder-only line variant should account for most of any observed-region gain, which would separate the two effects cleanly.
References
- GIFSplat: Generative Prior-Guided Iterative Feed-Forward 3D Gaussian Splatting from Sparse Views
- ReSplat: Learning Recurrent Gaussian Splats
- DepthSplat: Connecting Gaussian Splatting and Depth
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
- MG-SLAM: Structure Gaussian Splatting SLAM With Manhattan World Hypothesis
- G4Splat: Geometry-Guided Gaussian Splatting with Generative Prior
- NeuSD: Surface Completion With Multi-View Text-to-Image Diffusion
- Unified and Efficient Point-Line Local Features
- LightGlueStick: a Fast and Robust Glue for Joint Point-Line Matching
- VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion
- ReSplat: Degradation-agnostic Feed-forward Gaussian Splatting via Self-guided Residual Diffusion