Multi-View Stereo (MVS) and stereo matching have long served as the bedrock of 3D computer vision. Traditionally, these methods treat depth estimation as a purely geometric puzzle: given two or more images from different viewpoints, find corresponding pixels across views, compute their disparity along epipolar lines, and use triangulation to calculate exact metric depth.
However, this geometric formulation has a catastrophic blind spot. Triangulation fundamentally depends on finding distinct visual correspondences. In textureless regions (such as plain white walls or smooth floors), reflective surfaces, shadowed areas, or repetitive patterns, local pixel matching becomes highly ambiguous. The mathematical signal disappears, leaving the cost volume flat or noisy. Furthermore, in wide-baseline setups or occluded regions, features may only be visible to a single camera, making geometric matching mathematically impossible.
To resolve these ambiguities, a powerful paradigm has emerged: monocular-guided multi-view stereo. Instead of relying solely on geometric correspondences, this approach injects learned "common sense" from single-image depth estimators and vision foundation models into the multi-view pipeline. By combining the precise metric constraints of multi-view geometry with the robust structural priors of monocular vision, these systems can reliably infer depth where traditional triangulation fails.
The Intuition: Combining Local Matching with Global Context
The synergy between monocular and multi-view cues relies on a division of labor between two types of visual information:
- Multi-view matching provides high-frequency, localized geometric accuracy. When distinct textures are present, pixel-matching algorithms can pinpoint depth with pixel-level precision.
- Monocular priors provide low-frequency, global semantic context. Trained on massive datasets, monocular models understand object shapes, category-specific structures, and perspective cues (such as texture gradients and haze).
Early foundational works recognized this complementary relationship. Saxena et al. demonstrated that integrating learned monocular cues could fill in depth information where stereo correspondences fail, particularly in featureless regions. Later, Ummenhofer et al. showed that combining learned visual priors with geometric matching provides critical robustness in scenarios with minimal camera translation, where the geometric baseline is too weak for standard mathematical triangulation.
In modern architectures, this integration is achieved primarily through two core mechanisms: search space restriction and cost volume regularization.
Mechanism 1: Constraining the Search Space
In a standard plane-sweep MVS pipeline, the network must evaluate dozens of depth hypotheses across an expansive range for every pixel. This unconstrained search is not only computationally expensive but also highly prone to false matches in ambiguous regions.
Monocular guidance solves this by using a single-view depth prediction as a geometric anchor. Instead of searching blindly across the entire depth spectrum, the system restricts or centers its candidate depth planes around an initial monocular estimate.
Traditional MVS Search Space:
[------------------------------------------------------------------->] (Wide, prone to false matches)
Monocular-Guided Search Space:
Gaussian Distribution
.---.
/ \
[------------------/ X \---------------------------------------->] (Narrowed around monocular prior X)
As shown by Wang et al., centering the candidate depth planes around a monocular prediction significantly reduces matching ambiguity and computational overhead. To handle uncertainty, frameworks like the one proposed by Chen et al. define a explicit probability distribution (such as a Gaussian) around the monocular estimate. This distribution acts as a geometric heuristic that bounds the multi-view search space, stabilizing depth estimation in wide-baseline or sparse-view scenarios where traditional triangulation is unreliable due to low overlap.
Mechanism 2: Cost Volume Regularization via Semantic Priors
The second core mechanism operates directly within the feature space. Instead of using monocular depth merely as a post-processing filter or a strict boundary, modern networks fuse high-level semantic features from pre-trained foundation models directly into the multi-view cost volume.
When a cost volume is constructed solely from local matching features, textureless surfaces produce flat, ambiguous response curves. By concatenating or blending these local features with global features from a frozen monocular backbone—such as the Segment Anything Model (SAM) or Depth Anything—the network gains access to structural context.
Shvets et al. demonstrated that integrating high-level semantic features from foundation models allows the network to leverage category-specific priors (e.g., understanding that a region is a "flat wall" or a "smooth floor") to resolve geometric structures where visual matching is ambiguous. Similarly, Izquierdo et al. and Xu et al. showed that concatenating local geometric correspondences with high-level monocular features allows the model to maintain strict geometric accuracy in textured regions while using learned object-shape priors to smoothly regularize noisy or ambiguous correspondences in reflective or untextured areas.
Reconciling the Scale Mismatch
A significant technical challenge in this paradigm is the inherent discrepancy in scale. Multi-view stereo relies on rigid, metric geometric constraints derived from camera baselines. Conversely, monocular foundation models typically predict relative or up-to-scale depth, as a single image lacks the absolute scale information inherent in multi-view geometry.
To successfully marry these two inputs, architectures must dynamically align the monocular prior with the metric space. Researchers handle this mismatch in a few distinct ways:
- Scale-Agnostic Cost Volumes: Izquierdo et al. designed a flexible, scale-agnostic cost volume capable of natively blending scale-invariant monocular features with rigid geometric matching cues, allowing for zero-shot generalization across completely unseen environments.
- Recurrent Scale Alignment: Jiang et al. introduced a specialized Scale Update (SU) module within a recurrent stereo matching framework. This module continuously estimates and refines the scale alignment parameters, ensuring that the high-level semantic priors from the monocular model precisely match the metric dimensions dictated by the stereo baseline.
Application to Feed-Forward 3D Reconstruction
The fusion of monocular priors and multi-view geometry has proven particularly transformative for feed-forward 3D reconstruction and novel view synthesis, such as generalizable 3D Gaussian Splatting (3DGS).
When reconstructing a scene from a sparse set of wide-baseline images, traditional pipelines suffer from severe artifacts, creating "floaters" or completely missing geometry in non-overlapping or low-texture zones. By embedding pre-trained monocular features directly into the feature encoders, models like those presented by Liu et al. and Zhang et al. can generate plausible, structurally consistent geometry even in areas where stereo matching is mathematically impossible.
Ultimately, monocular-guided multi-view stereo shifts the problem of depth estimation from a brittle, purely mathematical calculation to a robust inference process. By leveraging the "collective wisdom" of vision foundation models, as Zhou notes, modern stereo pipelines no longer fail when the geometric signal disappears; instead, they gracefully transition to learned structural intuition to deliver clean, continuous, and physically accurate reconstructions.
Key Foundational Work
Make3D: Depth Perception from a Single Still Image (2008) — contributed 1 concept elements to this direction.
DeMoN: Depth and Motion Network for Learning Monocular Stereo (2017) — contributed 1 concept elements to this direction.