Reconstructing a 3D scene from a collection of 2D images is fundamentally an inversion problem. Given a set of flat projections, optimization frameworks like Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) attempt to find a 3D spatial representation that perfectly reproduces the input images when rendered.
When the input data is dense and camera poses are precisely known, this optimization is highly effective. However, when views are sparse (few-shot), camera poses are unknown, or the environment lacks distinct visual textures, the problem becomes severely under-constrained. The optimization landscape collapses into a phenomenon known as the shape-radiance ambiguity: the model can satisfy the 2D pixel loss by creating physically impossible geometries, such as flattening the entire scene onto a single plane or generating disconnected "floating" artifacts right in front of the camera.
To resolve these ambiguities, a prominent research direction has emerged: bootstrapping and regularizing 3D reconstruction pipelines using monocular depth priors derived from data-driven foundation models or single-image estimators.
The Core Intuition: Relative Structure Over Absolute Scale
Why do monocular depth priors help if they are notoriously scale-ambiguous? A single-image depth estimator cannot know the absolute metric distance of an object without external context. However, what these models do possess—thanks to being trained on millions of diverse images—is a robust, data-driven understanding of relative geometry, object boundaries, and structural layouts.
Even if a predicted depth map is scaled or shifted incorrectly, its internal spatial relationships remain highly coherent. By injecting these relative depth maps into 3D optimization frameworks, researchers can provide a "global structural skeleton" that constrains the optimization space. Instead of allowing 3D primitives to settle anywhere in a vast vacuum, the prior forces them to conform to a plausible geometric arrangement from the very first iteration.
Bootstrapping Structure via Monocular Back-Projection
In scenarios where camera poses are entirely unknown or highly noisy, traditional Structure-from-Motion (SfM) pipelines like COLMAP often fail to find matching features across images. Without an initial point cloud, explicit representations like 3D Gaussian Splatting cannot begin optimization.
Monocular depth priors solve this "chicken-and-egg" problem through progressive back-projection. As demonstrated by Fu et al. in COLMAP-Free 3D Gaussian Splatting, pixels from a primary reference image can be unprojected into 3D space using their estimated monocular depth. This creates an immediate, dense cloud of 3D primitives that establishes the initial spatial structure of the scene.
[2D Input Image] ---> (Monocular Depth Model) ---> [Relative Depth Map]
|
v
[Initial 3D Scene] <--- (Back-Projection) <--- [Seeded 3D Primitives]
Once this initial geometric anchor is established, subsequent views can be incrementally aligned and integrated. Jiang et al. leverage this mechanism in A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose, showing that building a scene progressively from monocular back-projections prevents the model from converging on degenerate, flat solutions. Similarly, Jain et al. demonstrate in Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images that these single-image priors act as spatial anchors, allowing optimization algorithms to bypass traditional SfM preprocessing entirely and escape local minima during joint pose-and-scene optimization.
Overcoming Textureless and Degraded Environments
Traditional multi-view geometry relies heavily on photometric consistency—the assumption that a distinct physical point looks identical when viewed from different angles. When a scene contains large, textureless surfaces (like plain indoor walls) or suffers from degraded visibility (such as surgical endoscopies or underwater scattering), feature-matching algorithms fail because they cannot find distinct keypoints to track.
Monocular models do not rely on matching features across frames; instead, they infer shape from semantic context, shading, and perspective cues within a single frame.
In endoscopic surgery, where reflections and fluid surfaces distort visual tracking, Huang et al. show in Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting that initializing spatial distributions with foundation model depth maps provides a stable geometric starting point where traditional sparse point methods fail. This concept is extended to highly degraded environments by Wu et al. in Plenodium: Underwater 3D Scene Reconstruction with Plenoptic Medium Representation, where monocular pseudo-depth maps are used to densify regions obscured by underwater haze, ensuring complete structural coverage where multi-view consistency signals are entirely lost.
Hybrid Architectures: Merging SfM with Monocular Priors
While monocular priors offer dense, continuous geometry, they lack absolute metric scale. Conversely, sparse SfM point clouds provide highly accurate metric anchors but leave massive gaps in coverage. A powerful paradigm within this research direction is the hybridization of these two complementary signals.
[Sparse SfM Points] ----> (Accurate Metric Scale) -----\
+---> [Dense, Scale-Accurate Foundation]
[Monocular Depth] ----> (Dense Relative Geometry) ----/
In Multi-view Reconstruction via SfM-guided Monocular Depth Estimation, Guo et al. project sparse SfM point clouds into camera views to act as a geometric skeleton, which is then densified using a generative depth model. This anchors the generative model's predictions to a true metric scale.
For few-shot novel view synthesis, the sole author Pérez-Pellitero demonstrates in CoMapGS: Covisibility Map-based Gaussian Splatting for Sparse Novel View Synthesis that augmenting sparse SfM points with aligned monocular depth estimates creates a dense geometric foundation. This dense initialization prevents the optimization process from generating floating artifacts in unobserved regions, stabilizing the scene geometry even when views are extremely limited.
Rethinking Supervision: From Hard Constraints to Probabilistic Suggestions
Early attempts to use depth priors enforced them as "hard constraints"—minimizing a direct, point-wise distance loss between the optimized scene depth and the predicted monocular depth. However, because monocular models contain localized inaccuracies and lack perfect multi-view consistency, forcing a 3D model to match them exactly often degrades rendering quality and introduces blurriness.
Modern approaches treat monocular priors as probabilistic suggestions rather than absolute truths. Instead of enforcing pixel-perfect alignment, optimization frameworks regularize the distribution of geometry.
A clear realization of this concept is presented by Rau et al. in Depth-Guided NeRF Training via Earth Mover's Distance. Rather than penalizing point-wise discrepancies, the authors supervise the distribution of ray termination distances using the Earth Mover's Distance (EMD). This allows the NeRF optimization to shift and scale the depth profile locally to maximize rendering fidelity while still respecting the overall geometric arrangement suggested by the prior.
By shifting from rigid supervision to flexible, distribution-based regularization, the field has unlocked a way to exploit the structural intelligence of monocular foundation models without inheriting their localized errors.
Key Foundational Work
Learning Camera Viewpoint Using CNN to Improve 3D Body Pose Estimation (2016) — contributed 2 concept elements to this direction.
Superpixel meshes for fast edge-preserving surface reconstruction (2015) — contributed 1 concept elements to this direction.