Sparse View Synthesis with Geometric and Generative Priors

This direction is split into other directions: sparse-view overfitting. Its last version stays readable and citable here.

Version 1 of 1237 papers263 conceptsfrozen 2026-10-04

Neural reconstruction methods like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) typically require a dense set of images to accurately capture a scene. When the number of viewpoints is reduced to a handful—the "sparse-view" regime—these models face a fundamental ambiguity: there are infinitely many 3D configurations that can explain a small set of 2D images.

The Overfitting Trap

The primary failure mode in sparse-view reconstruction is overfitting. In a dense-view scenario, every point in 3D space is constrained by multiple intersecting rays. In a sparse-view scenario, large regions of the scene may only be observed by one or two cameras.

As noted by Prune Truong and colleagues in SPARF, the optimization process "cheats" by placing density or 3D primitives (like Gaussians) in a way that satisfies the training pixels but lacks true 3D structure. This results in several distinct artifacts:

  • Floaters: Stray bits of geometry that hang in mid-air to block or create colors for specific views.
  • Billboarding: Primitives that flatten themselves against the camera plane, appearing correct from the training angle but disappearing or distorting when the camera moves.
  • Needle-like Artifacts: Elongated Gaussians that stretch toward the camera, a phenomenon Marko Mihajlovic addresses in SplatFormer.

Anchoring Geometry with External Priors

To prevent the model from inventing "fake" geometry, researchers introduce external constraints that anchor the 3D representation to physical reality.

One of the most effective "free" anchors comes from the Structure-from-Motion (SfM) process used to estimate camera poses. Kangle Deng and the DS-NeRF team demonstrated that the sparse 3D point clouds generated by SfM can serve as a depth supervisor. By forcing the neural field to match these points, the model is prevented from placing density in empty space.

More recent work like MVPGS by Wangze Xu extends this by using Multi-View Stereo (MVS) to provide a denser geometric initialization. Similarly, SolidGS introduces a "solidness factor" and monocular depth priors to ensure that Gaussian primitives form a continuous surface rather than a cloud of disconnected flakes.

Hallucinating Consistency with Generative Models

When geometric priors aren't enough, researchers turn to generative models to "fill in the blanks." If we only have three images of a chair, a geometric prior might tell us where the legs are, but it won't tell us what the back of the chair looks like.

Methods like Sp2360 and Bootstrap-GS use 2D diffusion models as a "hallucination engine." By rendering the current (incomplete) 3D model from an unobserved viewpoint and passing it through a diffusion model, the system generates a "pseudo-ground truth." This synthetic view provides the missing supervision needed to clean up floaters and complete the scene. Yifei Gao shows that reintegrating these synthetic views actually reduces the number of primitives required, as the model no longer needs redundant "fixer" Gaussians to satisfy sparse constraints.

Structural Regularization and Neural Generators

A third path involves changing the architecture of the 3D representation itself to be inherently more robust. Standard 3DGS optimizes each Gaussian independently, which is why they can easily become "floaters."

SplatFields and SplatFormer move away from independent optimization. Instead, they use a neural generator (like a Point Transformer) to predict the properties of the Gaussians. Because the neural network has its own inductive biases—such as spatial smoothness—it is much harder for the model to produce isolated, noisy artifacts.

Similarly, DIP-GS by Raja Giryes leverages the Deep Image Prior, using the architectural bias of a CNN to generate structured Gaussian parameters. This ensures that even when data is scarce, the resulting 3D scene retains the natural textures and structures inherent in the network's design.

Papers in this version

Showing 60 of 237 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sparse_view_synthesis_with_geometric_and_d7228187,
  title = {Sparse View Synthesis with Geometric and Generative Priors},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-view-synthesis-with-geometric-and-generative-priors/rdr_d722818771c44f228d71f69ece05cbd4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_sparse_view_synthesis_with_geometric_and_d7228187_v1,
  title = {Sparse View Synthesis with Geometric and Generative Priors},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-view-synthesis-with-geometric-and-generative-priors/rdr_d722818771c44f228d71f69ece05cbd4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}