Multi-view stereo (MVS) recovers 3D shape by testing, for every pixel of a reference image, many candidate depths and checking which one makes the pixel look the same in the other calibrated views. The standard data structure for this test is the cost volume, a grid that stores a matching score for every pixel and every candidate depth. Its memory grows with image height × width × number of depth candidates × feature channels, so high-resolution MVS runs into GPU limits. The usual workaround is coarse-to-fine search: estimate depth roughly, then search only a narrow window around that estimate. When the rough estimate is wrong, for example on a blank wall, a mirror, or a repeated pattern, the narrow window no longer contains the true surface, and nothing later in the pipeline can recover it. Monocular depth networks, which predict depth from a single image, are good at exactly these regions. Their output has no absolute scale, though, and they sometimes produce confident but wrong geometry.
This note proposes to replace the rough-estimate stage with a per-pixel depth band. The band is built from a scale-aligned monocular prediction, and its width is calibrated statistically so that it contains the true depth with a chosen probability, say 95%. Each pixel then receives a number of depth candidates proportional to its band width. Confident planar regions get two or three candidates, and ambiguous boundaries get dozens. Memory therefore scales with the total uncertainty in the scene rather than with a fixed depth range. The contribution is not "use a monocular prior to narrow the search," which earlier work already does. The contribution is to make the narrowing come with a measurable guarantee against the failure that coarse-to-fine MVS cannot recover from. We call that failure lock-in: the true depth falls outside the final search window.
Core Claim
Coarse-to-fine MVS spends memory where it is least needed and saves memory where it is most dangerous. Two existing systems illustrate this. CVP-MVSNet uses the same 8 residual hypotheses at every pixel of every fine level. CasMVSNet narrows its depth range uniformly around the previous stage's prediction. DDL-MVS shows that depth error is far from uniform: it concentrates at depth discontinuities and is often bimodal there. Separately, How to Guide Adaptive Depth Sampling? shows that the RGB image alone can predict where a depth predictor will fail, with a reported 37% RMSE reduction over random spatial sampling on Synthia. That idea has not been applied to choosing which depth planes go into a cost volume.
Our claim has two parts. First, a band built from a locally rescaled monocular prediction can have calibrated coverage, meaning the fraction of pixels whose true depth lies inside the band matches the target on held-out scenes. Second, a variable-size cost volume built only inside those bands can match the accuracy of the best memory-efficient MVS methods at lower peak memory, while sharply reducing the lock-in rate. The lock-in rate is the diagnostic we introduce. No paper in our file reports it, even though it is the failure mode the cascade literature itself names.
Research Program
The pipeline has four stages.
Stage 1: monocular prediction on the reference view only. We run a frozen Depth Anything V2 model on the reference image alone. This follows MonoMVSNet, which found in ablation that extracting monocular features from all views was slower and actually less accurate than reference-only extraction. Source views go through a light CNN, as in the asymmetric design of RIAV-MVS. This keeps the large vision-transformer cost fixed regardless of how many views are used. That cost is the bottleneck that limited Multi-View SAM to 3–5 views.
Stage 2: sparse full-range probes for metric scale. We evaluate matching cost over the full depth range, but only at a sparse grid of pixels, in the spirit of Fast-MVSNet. Probes whose cost curve has a single sharp peak become metric anchors. We fit a spatially varying scale-and-shift field that maps the relative monocular prediction onto these anchors. The field is regularized to be piecewise smooth within monocular depth-gradient regions, so that different objects can take different scale factors. This is our N-view, homography-based counterpart to the dense scale map that DEFOM-Stereo learns for rectified two-view stereo. It also directly targets the regional scale inconsistency that MonoMVSNet lists as unresolved.
Stage 3: calibrated bands. A small head predicts a per-pixel residual scale $\sigma_p$ from four inputs: distance to the nearest reliable anchor, monocular depth gradient, probe cost entropy, and residual error of the scale fit. A calibration step then turns $\sigma_p$ into an interval with a coverage guarantee. On held-out training scenes we compute the normalized residuals $|z_p - \hat z_p|/\sigma_p$, where $z_p$ is ground-truth depth and $\hat z_p$ is the aligned monocular depth. We take their $(1-\alpha)$ quantile $q_{1-\alpha}$ separately within bins of predicted difficulty. The band and the number of candidates per pixel are then
$$I_p = [\hat z_p - q_{1-\alpha}\sigma_p,\ \hat z_p + q_{1-\alpha}\sigma_p], \qquad D_p = \min\left(D_{\max}, \lceil |I_p|/\delta \rceil\right),$$
where $\delta$ is the depth step size and $D_{\max}$ caps the number of candidates. Where the probe cost curve is clearly bimodal, as at depth edges, the band becomes a union of two intervals, following the two-mode depth model of DDL-MVS. Peak cost-volume memory becomes
$$M \propto C \sum_p D_p,$$
where $C$ is the feature channel count. This replaces the fixed $C \cdot H W D$. The memory saving is therefore a measurable consequence of how confident the calibrated bands are.
A caveat on the guarantee: split-conformal calibration assumes test scenes are exchangeable with calibration scenes. That assumption fails across datasets, so we report coverage separately in-domain and under transfer, and we treat coverage under transfer as an empirical question.
Stage 4: refinement inside the bands. The variable-length cost volume is stored as a packed sparse tensor. A 2D convolutional GRU (a recurrent unit that updates a 2D depth estimate step by step) indexes it, as in Effi-MVS and RIAV-MVS. This removes the coarse 3D CNN that Effi-MVS uses to produce its starting estimate. The GRU may only move depth within $I_p$. A pixel whose matching cost stays flat inside its band is flagged rather than forced, which leaves room for the uncertainty-weighted fusion idea of MOVEDepth.
Baselines
Naive reference. CasMVSNet with its standard cascade, retrained on our data split. The CasMVSNet paper reports 5.3 GB memory, 0.49 s runtime, and 0.355 mm overall error on DTU. We also include an aligned-monocular-only reference: the Stage 2 output with no multi-view refinement. It shows how much the multi-view step actually adds.
Strongest geometric competitor. Effi-MVS, which the file reports at 0.317 mm DTU overall error using 3.1–3.6 GB at 1600×1184 (Effi-MVS). It is the closest architecture to ours, since it also refines depth recurrently within a narrow window, but it starts from a learned coarse 3D volume instead of a monocular band.
Strongest monocular-guided competitor. MonoMVSNet. It reports 2.01 GB of memory and first place on the Tanks and Temples Intermediate and Advanced sets, but no reported result found on Lacuna for its DTU millimetre figures or its Tanks and Temples F-score, so we will rerun its released configuration.
Fair comparison. All methods are trained on DTU and fine-tuned on BlendedMVS with the same view count and input resolution. Each gets an equal hyperparameter-search budget. Peak memory is measured end-to-end, including the monocular network forward pass, on the same GPU.
What counts as beating them. DTU overall error at or below Effi-MVS's 0.317 mm with lower peak memory than both Effi-MVS and MonoMVSNet, together with a lock-in rate lower than every baseline's.
Ablations. Three ablations isolate the key ingredient. The first is calibrated bands versus uncalibrated bands of equal total candidate budget. The second is local versus global scale alignment. The third is bands driven by monocular cues versus bands driven by matching entropy alone.
Why not just…?
Why not just widen the cascade windows? Uniform widening does fix some lock-in, but it pays for it at every pixel, including confident ones. We will run CasMVSNet with its final-stage window widened until its total candidate count equals our $\sum_p D_p$, and compare lock-in rate and DTU error at identical memory. If uniform widening matches us at equal memory, the calibration idea adds nothing and the proposal fails.
Why not just use MonoMVSNet's monocular-guided sampling? This is the real threat. MonoMVSNet already adjusts sampling intervals using monocular depth, especially at boundaries, within a standard multi-scale cost volume. Three things differ in our design. We give no coverage guarantee to adjust toward in their method but do in ours; their candidate count per pixel is fixed while ours varies; and they align scale globally while we align it locally. The decisive test is lock-in rate and error on the hardest pixels: textureless and reflective regions of ScanNet, and discontinuity pixels on DTU. If MonoMVSNet matches us there, we would reframe the work as a calibration analysis of its sampler rather than a new system.
Why not just use matching entropy, as MOVEDepth does, without a monocular band? Entropy is uninformative exactly where it is needed. On a flat cost curve it says "uncertain" but gives no indication of where to look. The entropy-only ablation, which drives band widths from probe entropy around a geometric seed, tests this directly. We expect it to show adequate coverage only by producing very wide bands on textureless surfaces, and therefore high memory.
Why not just a fixed narrow band around globally aligned monocular depth? MOVEDepth's 16 planes work for forward-moving driving video. Intra-image scale inconsistency, documented by DEFOM-Stereo, means a single global fit leaves whole objects outside any fixed band. The global-alignment ablation measures exactly this as lost coverage.
Proposed Experiments
Coverage, our main diagnostic. For each method we report the fraction of ground-truth pixels whose depth lies outside the final search window, on DTU, ScanNet, and 7-Scenes. We plot it against peak memory as α varies, giving a memory–coverage curve. The method succeeds if it achieves 95% empirical coverage in-domain using fewer total candidates than CasMVSNet uses.
Accuracy. We report DTU accuracy, completeness, and overall error against the cited 0.317 mm (Effi-MVS) and 0.355 mm (CasMVSNet). We submit to Tanks and Temples for F-scores, where none are quoted in the file. On ScanNet we compare abs-rel against RIAV-MVS's reported 0.0734. For zero-shot transfer to 7-Scenes we compare δ<1.25 against RIAV-MVS's reported 0.8967, and report coverage under this distribution shift explicitly.
Memory scaling. We measure peak memory versus input resolution on ETH3D high-resolution scenes up to 2K and beyond. This tests whether the summed-band term stays below the cost of storing per-view feature maps, which the file identifies as the remaining memory bottleneck.
Hallucination check. We plant cases where the monocular prediction is confidently wrong, such as mirrors, and measure how often the sparse probes catch the mismatch and widen the band. If they do not, the guarantee does not hold on exactly the regions the method targets.
References
- Cost Volume Pyramid Based Depth Inference for Multi-View Stereo
- Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
- DDL-MVS: Depth Discontinuity Learning for MVS Networks
- How to Guide Adaptive Depth Sampling?
- MonoMVSNet: Monocular Priors Guided Multi-View Stereo Network
- RIAV-MVS: Recurrent-Indexing an Asymmetric Volume for Multi-View Stereo
- Joint Depth Prediction and Semantic Segmentation with Multi-View SAM
- Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement
- DEFOM-Stereo: Depth Foundation Model Based Stereo Matching
- Efficient Multi-view Stereo by Iterative Dynamic Cost Volume
- Crafting Monocular Cues and Velocity Guidance for Self-Supervised Multi-Frame Depth Learning