Multi-view stereo recovers 3D shape from photographs taken at several viewpoints. Learned versions build a "cost volume": for every pixel and every candidate depth, they record how well the views agree, then process this grid with 3D convolutions. The grid grows with image width times height times the number of candidate depths, so memory runs out quickly. The standard fix is a cascade: guess depth coarsely, then search finely only near the guess. When the coarse guess is wrong, the fine stages cannot recover. A separate problem appears with wide apertures, macro lenses, or views focused at different distances. Out-of-focus regions look different from one view to the next, so feature matching degrades exactly where blur is strongest.
This note argues that the two problems share a solution. The amount of blur at a pixel is itself a measurement of depth, and under the thin-lens model it is linear in inverse depth, the same quantity a stereo cost volume samples. Blur therefore tells the network where to place its few depth candidates. It also tells the network, at each candidate, how much to blur the sharper view before comparing it to the blurrier one. We propose a cost volume whose candidate depths come from a per-pixel defocus estimate rather than a coarse full-range stage, and whose matching cost is computed between blur-equalized features. Because the number of candidates per pixel is small and blur is applied from a fixed bank of precomputed levels, matching under the correct per-depth blur costs memory proportional to a handful of blur levels rather than to the number of depth candidates.
Core Claim
Write the blur-circle diameter from the thin-lens model in the research file as $c(d) = \kappa\,|1/s - 1/d|$. Here $d$ is object depth, $s$ is focus distance, and $\kappa = A f_0 s/(s-f_0)$ collects aperture diameter $A$ and focal length $f_0$. A measured blur $\hat c$ with noise $\sigma_c$ yields two candidate inverse depths:
$$\frac{1}{d} \in \left\{ \frac{1}{s} - \frac{\hat c}{\kappa},\; \frac{1}{s} + \frac{\hat c}{\kappa} \right\}, \qquad \sigma_{1/d} \approx \frac{\sigma_c}{\kappa}.$$
The two candidates correspond to points behind and in front of the focal plane. The uncertainty is constant in inverse depth because the relation is linear. Defocus thus behaves like a weak stereo pair whose baseline is the aperture, and its errors come from a mechanism (blur-scale estimation) unrelated to cross-view feature matching. We claim three things. First, a per-pixel two-component mixture over inverse depth, plus a small uniform "escape" floor, is a better hypothesis distribution than a coarse cascade stage, specifically on defocused captures. Second, matching blur-equalized features at those hypotheses fixes the matching failure that blur causes. Third, the combination gives better reconstructions than deblur-then-match or blur-augmented training at equal or lower peak GPU memory.
Relation to Prior Work
Prior work motivates the problem from both sides. Cascade designs such as Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching and Cost Volume Pyramid Based Depth Inference for Multi-View Stereo narrow the search around a coarse full-range estimate. That coarse estimate is the single point of failure the research file identifies. IS-MVSNet: Importance Sampling-Based MVSNet places candidates densely around a unimodal estimate. We borrow its idea that non-uniform sampling buys resolution for free. We differ because our density comes from an independent physical cue and is bimodal by construction. Efficient Multi-view Stereo by Iterative Dynamic Cost Volume samples only a few hypotheses per pixel with a recurrent update. It is our closest memory-efficient competitor, and it inherits the local-minimum problem the file notes. All of these assume sharp, photometrically consistent views.
The closest method for the blur side is Blur-Aware Disparity Estimation from Defocus Stereo Images. It synthetically blurs the sharper image by the relative blur predicted at each disparity before matching. It is hand-crafted, limited to two views, and evaluates the blur at every disparity. We reuse its blur-equalization principle inside a learned multi-view cost volume, and we evaluate it only at defocus-proposed hypotheses so memory stays bounded. Depth from Semi-Calibrated Stereo and Defocus fuses defocus and stereo in an offline energy over a dedicated camera rig. Deep Eyes: Binocular Depth-from-Focus on Focal Stack Pairs fuses focus depth and stereo depth after the fact, from 16-slice focal stacks, at about 9.7 s per image. In both, defocus is a parallel depth estimate. In ours, defocus shapes the cost volume itself, from single exposures per view. Joint Deblurring and 3D Reconstruction for Macrophotography handles macro blur by per-scene Gaussian splatting with a learned per-pixel kernel. It reports that larger kernels sharply raise memory and training time. This is a constraint our design must respect, and it motivates the precomputed blur bank instead of per-hypothesis convolution.
Two methods supply components. Aberration-Aware Depth-From-Focus shows that thin-lens training data creates a large gap on real lenses: its aberration-aware training cut depth-from-focus mean absolute error from 0.4706 to 0.2095. We take its small PSF-predicting network as an optional replacement for the Gaussian blur model. Single-image defocus estimators such as Accurate Single-Image Defocus Deblurring Based on Improved Integration with Defocus Map Estimation supply the per-pixel blur measurement $\hat c$. Where dual-pixel data exist, the sign of the dual-pixel disparity (Defocus Deblurring Using Dual-Pixel Data) resolves the front/back ambiguity and collapses the mixture to one mode.
Research Program
The system has three parts.
The first is a defocus head. It predicts, per view and per pixel, a blur estimate $\hat c$ and its uncertainty $\sigma_c$, trained with a likelihood loss against ground-truth blur rendered from ground-truth depth. With each view's focus distance $s_i$ and lens constant $\kappa_i$ (from metadata or calibration), the head's output becomes a two-mode inverse-depth proposal for the reference view. When source views are focused differently, their blur estimates add consistent constraints after reprojection. We fuse these by multiplying the per-view densities along the reference ray, which often breaks the front/back ambiguity without dual-pixel hardware.
The second is the hypothesis sampler. Each pixel draws $K$ inverse depths, with $K$ between 8 and 16. Most are drawn from the fused mixture by inverse-transform sampling, as IS-MVSNet does for its unimodal case. A fixed fraction is spread uniformly across the full range so that a defocus error remains recoverable. A recurrent refiner then updates the samples in the style of Effi-MVS.
The third is the blur-equalized cost. For each view we precompute a bank of $L$ Gaussian-blurred feature maps, with $L$ around 6, spaced geometrically in blur radius. At a hypothesis $d$, we predict the blur of the reference pixel and of its reprojection in view $i$. The sharper of the two features is read from the bank at relative blur $\sqrt{c_{\max}^2 - c_{\min}^2}$, using linear interpolation between levels. The matching cost is then a group-wise correlation.
Peak memory for the blur bank scales with $L \cdot C \cdot H \cdot W$ per view. The cost volume scales with $K \cdot C \cdot H \cdot W$. The naive alternative of correctly blurred matching over a uniform $D$-hypothesis volume scales with $D$ times kernel area. The research question is whether this bounded design keeps the accuracy of exhaustive blur-aware matching.
Baselines
The naive reference is CasMVSNet trained on sharp data and tested on our defocused renders. It shows how much blur alone hurts. On sharp DTU it reports a 35.6% overall error reduction over MVSNet, with memory falling from 10.8 GB to 5.3 GB and runtime from 1.21 s to 0.49 s.
The strongest memory-efficient competitor is Effi-MVS. It reports 0.317 mm overall error on sharp DTU using 3.1 to 3.6 GB at 1600×1184. It shares our few-hypotheses-per-pixel design, so it isolates whether the source of the hypotheses matters. We also run IS-MVSNet, which reports 62.82% F-score on Tanks and Temples Intermediate and 83.15% on ETH3D, because it is the direct non-defocus importance sampler. The research file also lists RA-MVSNet at 0.297 mm overall on DTU, which we include as the accuracy ceiling on sharp data. No reported result was found on Lacuna for any multi-view stereo method on defocus-blurred DTU, Tanks and Temples, or ETH3D. All defocused-setting numbers will therefore be produced by us.
For fairness, every baseline is retrained on the same defocused training renders with identical blur augmentation. Each gets the same tuning budget of 20 configurations and the same per-view focus metadata as an extra input channel. Each is held to the same peak-memory cap, which we set at Effi-MVS's 3.6 GB at 1600×1184.
The ablations remove one ingredient at a time. The first keeps defocus-driven sampling and drops blur equalization. The second keeps blur equalization and replaces defocus sampling with a coarse cascade stage. The third removes the uniform escape floor. The fourth replaces the bank with exact per-hypothesis convolution, which gives an uncapped upper bound on what the bank approximation costs. Beating the baselines means lower DTU overall error and higher Tanks and Temples F-score than every retrained baseline at blur radii above roughly 5 pixels, at no more peak memory, while staying within a small tolerance of each baseline on sharp inputs.
Why not just…?
Why not just deblur every view first and run a standard multi-view network? This is the most serious threat, because single-image deblurring is mature. INIKNet reports 26.055 dB PSNR on DPDD with 1.98M parameters, against Restormer's 25.980 dB. Our concern is that independent deblurring restores high frequencies differently in each view, which can create false matches, and that it throws away the blur cue that could have guided the search. Neither concern is established, so we test it directly. We run INIKNet followed by CasMVSNet and INIKNet followed by Effi-MVS, both fine-tuned end to end. If this pipeline matches our method on reconstruction error, the blur-equalization part of the proposal is unnecessary, and we would report only the sampling contribution.
Why not just train an existing network with blur augmentation? Learned features might become blur-invariant on their own. Blur invariance has a limit, though: two views of the same point carrying different blur share only the frequencies below the larger blur's cutoff. Augmentation cannot restore the discarded information, but explicit equalization at the right level uses it correctly. The retrained baselines described above answer this question. We will report the gap as a function of the cross-view blur difference, not only absolute blur. The prediction is that augmentation keeps pace when views share similar blur and falls behind as their blur differs.
Why not just feed a monocular depth or defocus map into the network as an extra input? This hands the network the cue without forcing it to sample correctly. Ablation 2 combined with an extra-channel variant of Effi-MVS isolates the question. If an extra input channel captures the benefit, the structured mixture sampler is not needed.
Why not just add more uniform hypotheses? This runs into the memory wall the research file documents: memory grows with the number of depth candidates. We compare against it at equal memory, which is the only comparison that matters here.
Proposed Experiments
Training and testing use DTU, Tanks and Temples, and ETH3D, re-rendered with depth-dependent blur from ground-truth depth. Blur is applied by layered compositing to approximate occlusion bleeding at depth edges. Apertures are swept, and each view gets an independently drawn focus distance. A second rendering set uses aberration-aware PSFs following Aberration-Aware Depth-From-Focus to test sensitivity to the thin-lens assumption. The FlyingThings3D dual focal stack data from Deep Eyes, using one slice per view, gives a comparison against BDfF-Net's reported 0.021 MAE. Note that this comparison deliberately gives our method much less input than BDfF-Net receives. A small real set of wide-aperture handheld captures with dual-pixel output, including macro scenes, tests transfer to real lenses.
Metrics are DTU accuracy, completeness, and overall error, Tanks and Temples and ETH3D F-scores, peak GPU memory, and runtime. Results are binned by blur radius and by cross-view blur difference.
Three checks decide the outcome. The first is a hypothesis-coverage curve: the fraction of pixels whose true depth lies within one sampling interval of some hypothesis, comparing defocus sampling against a cascade's first stage on textureless and repetitive regions. Defocus proposals should cover more pixels exactly where the cascade traps. The second is a memory sweep showing that the blur-bank method tracks the uncapped exact-convolution ablation within 5% of its error, while using a fraction of its memory. The third is a sharp-input check, where blur carries no depth information. Here the method must fall back gracefully through the uniform floor, with error within a small tolerance of Effi-MVS's 0.317 mm. If it falls back badly, the design is a defocus specialist rather than a general improvement, and we would say so.
References
- Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
- Cost Volume Pyramid Based Depth Inference for Multi-View Stereo
- IS-MVSNet: Importance Sampling-Based MVSNet
- Efficient Multi-view Stereo by Iterative Dynamic Cost Volume
- Blur-Aware Disparity Estimation from Defocus Stereo Images
- Depth from Semi-Calibrated Stereo and Defocus
- Deep Eyes: Binocular Depth-from-Focus on Focal Stack Pairs
- Joint Deblurring and 3D Reconstruction for Macrophotography
- Aberration-Aware Depth-From-Focus
- Accurate Single-Image Defocus Deblurring Based on Improved Integration with Defocus Map Estimation
- Defocus Deblurring Using Dual-Pixel Data
- Single Image Defocus Deblurring via Implicit Neural Inverse Kernels