Neural reconstruction methods like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) typically require a dense set of images to accurately capture a scene. When the number of viewpoints is reduced to a handful—the "sparse-view" regime—these models face a fundamental ambiguity: there are infinitely many 3D configurations that can explain a small set of 2D images.
The Overfitting Trap
The primary failure mode in sparse-view reconstruction is overfitting. In a dense-view scenario, every point in 3D space is constrained by multiple intersecting rays. In a sparse-view scenario, large regions of the scene may only be observed by one or two cameras.
As noted by Prune Truong and colleagues in SPARF, the optimization process "cheats" by placing density or 3D primitives (like Gaussians) in a way that satisfies the training pixels but lacks true 3D structure. This results in several distinct artifacts:
- Floaters: Stray bits of geometry that hang in mid-air to block or create colors for specific views.
- Billboarding: Primitives that flatten themselves against the camera plane, appearing correct from the training angle but disappearing or distorting when the camera moves.
- Needle-like Artifacts: Elongated Gaussians that stretch toward the camera, a phenomenon Marko Mihajlovic addresses in SplatFormer.
Anchoring Geometry with External Priors
To prevent the model from inventing "fake" geometry, researchers introduce external constraints that anchor the 3D representation to physical reality.
One of the most effective "free" anchors comes from the Structure-from-Motion (SfM) process used to estimate camera poses. Kangle Deng and the DS-NeRF team demonstrated that the sparse 3D point clouds generated by SfM can serve as a depth supervisor. By forcing the neural field to match these points, the model is prevented from placing density in empty space.
More recent work like MVPGS by Wangze Xu extends this by using Multi-View Stereo (MVS) to provide a denser geometric initialization. Similarly, SolidGS introduces a "solidness factor" and monocular depth priors to ensure that Gaussian primitives form a continuous surface rather than a cloud of disconnected flakes.
Hallucinating Consistency with Generative Models
When geometric priors aren't enough, researchers turn to generative models to "fill in the blanks." If we only have three images of a chair, a geometric prior might tell us where the legs are, but it won't tell us what the back of the chair looks like.
Methods like Sp2360 and Bootstrap-GS use 2D diffusion models as a "hallucination engine." By rendering the current (incomplete) 3D model from an unobserved viewpoint and passing it through a diffusion model, the system generates a "pseudo-ground truth." This synthetic view provides the missing supervision needed to clean up floaters and complete the scene. Yifei Gao shows that reintegrating these synthetic views actually reduces the number of primitives required, as the model no longer needs redundant "fixer" Gaussians to satisfy sparse constraints.
Structural Regularization and Neural Generators
A third path involves changing the architecture of the 3D representation itself to be inherently more robust. Standard 3DGS optimizes each Gaussian independently, which is why they can easily become "floaters."
SplatFields and SplatFormer move away from independent optimization. Instead, they use a neural generator (like a Point Transformer) to predict the properties of the Gaussians. Because the neural network has its own inductive biases—such as spatial smoothness—it is much harder for the model to produce isolated, noisy artifacts.
Similarly, DIP-GS by Raja Giryes leverages the Deep Image Prior, using the architectural bias of a CNN to generate structured Gaussian parameters. This ensures that even when data is scarce, the resulting 3D scene retains the natural textures and structures inherent in the network's design.