Imagine looking at an object from only two or three widely spaced angles. For any given pixel in one image, the corresponding 3D point must lie somewhere along that camera’s line of sight. When you have hundreds of dense, overlapping views, thousands of intersecting rays pinpoint the exact 3D coordinates of every surface. But when views are sparse, this optimization problem becomes severely under-constrained.
As Mijeong Kim et al. point out in their work on InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering, an optimization algorithm operating on sparse data faces a mathematical cliff: infinitely many 3D configurations can project back to create the exact same 2D images. This fundamental uncertainty is known as sparse-view geometric ambiguity.
Why Unconstrained Optimization Fails
When modern 3D reconstruction pipelines—such as Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS)—are given too few perspectives, they leverage their excessive degrees of freedom to take the path of least mathematical resistance. To satisfy the sparse training views, the optimizer does not bother building a solid, continuous surface. Instead, it distributes semi-transparent density or independent primitives throughout empty space.
This geometric collapse manifests as distinct families of reconstruction artifacts:
- Floaters and Fog: Disconnected, cloud-like clusters of density that appear correct from the input viewpoints but look like floating debris from any other angle.
- Hollow Shells and Needle Structures: Primitives that stretch into elongated, needle-like shapes pointing directly at the camera lenses, satisfying the color constraint of a pixel without establishing true depth.
- Background Collapse: In outward-facing setups with minimal view overlap, the lack of geometric constraints causes background structures to smear or disintegrate entirely, as observed by Sibo Wu et al. in GenFusion: Closing the Loop between Reconstruction and Generation via Videos.
To resolve this ambiguity, researchers must introduce priors that artificially restrict the solution space, forcing the model to choose physically plausible geometries over degenerate shortcuts.
Sharpening Geometry via Information and Depth Regularization
One intuitive way to eliminate floaters is to force the model to be decisive about where surfaces exist. In InfoNeRF, Mijeong Kim et al. introduce ray entropy minimization. Instead of allowing a camera ray to pass through a foggy, semi-transparent haze of low-density particles, the entropy loss penalizes intermediate transparency. It forces the volume density along a ray to be either completely concentrated at a single solid surface or entirely empty. This information-theoretic constraint mathematically disallows the formation of cloudy floaters.
Standard Optimization: Ray ───► [ Foggy Density ] ───► [ Foggy Density ] (Ambiguous)
With Ray Entropy: Ray ───► [ Empty Space ] ───► [ Solid Surface ] (Sharp)
Another strategy anchors the optimization using monocular geometric cues. While predicting absolute depth from a single image is notoriously difficult, predicting relative depth relationships within a local patch is highly reliable.
In RDG-GS: Relative Depth Guidance with Gaussian Splatting for Real-time Sparse-View 3D Rendering, Chenlu Zhan et al. shift the optimization target away from noisy absolute depth values. By enforcing that the reconstructed 3D scene respects the relative spatial orderings predicted by monocular networks, they prevent the geometry from collapsing into flat or distorted planes.
Restricting Primitives' Degrees of Freedom
Unstructured 3D representations like Gaussian Splatting exacerbate ambiguity because individual primitives can move, scale, and rotate independently. To counter this, researchers bind the primitives to the underlying camera geometry.
In CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussians, Avinash Paliwal enforces spatial coherency by constraining the movement of Gaussians strictly along camera rays and regularizing their relationships using optical flow. By stripping away the primitives' freedom to drift independently in empty space, the optimization is forced to maintain surface continuity across the sparse views.
Hallucinating the Unseen with Generative Diffusion Priors
When viewpoints are so sparse that large regions of an object are completely occluded, geometric regularization alone is insufficient; there is simply no data available to regularize. Modern frameworks solve this by leveraging pretrained 2D or video diffusion models to "hallucinate" the missing viewpoints, effectively turning a sparse-view problem into a dense-view one.
Yan Xu et al. demonstrate this in Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion by treating sparse reconstruction as a video completion task. A video diffusion model generates smooth, physically plausible intermediate frames to bridge the wide gaps between the original camera positions, providing the dense multi-view constraints that algorithms need to resolve surface structures.
[Sparse View A] ────┐
▼
┌───────────┐
│ Diffusion │ ──► [Hallucinated Intermediate Views] ──► Dense Geometry
└───────────┘ Optimization
▲
[Sparse View B] ────┘
However, unconstrained generative models can introduce hallucinations that conflict with the true images. To balance imagination with fidelity, Junfeng Ni et al. developed Decompositional Neural Scene Reconstruction with Generative Diffusion Prior. They utilize a visibility-guided Score Distillation Sampling (SDS) mechanism.
This architecture ensures that the reconstruction pipeline remains strictly anchored to the ground-truth pixels in visible regions, while allowing the generative diffusion prior to sculpt and texture the hidden, unobserved zones. The result is a complete, artifact-free 3D asset generated from only a handful of glimpses.