Trust Monocular Depth Only Where Geometry Is Blind

Structure from motion (SfM), the process of recovering camera poses and 3D points from many overlapping photos, is accurate when cameras see a scene from well-separated angles. It becomes numerically fragile when they do not. Under small baselines, pure forward motion, or missing three-view tracks, a 1% error in image measurements can become a 100% error in recovered structure, as Barron, Jepson and Tsotsos showed. Monocular depth networks, which predict depth from a single image, have the opposite profile. They never break down numerically, but they are biased: they guess scale from object size, hallucinate depth from semantics, and collapse onto one surface when glass or reflections make depth ambiguous. Recent systems combine the two by adding monocular depth as an extra penalty inside bundle adjustment, the joint refinement of all poses and points. That penalty acts in every direction of the solution, including directions where the cameras already pin down the geometry. So a biased depth network can pull a well-measured reconstruction away from the truth.

We propose to apply the monocular prior only in the directions the multi-view measurements cannot determine. At each refinement step we compute how strongly the reprojection error alone constrains each point and each camera. We find the nearly unobserved directions: depth along a ray at low parallax, global scale, and the coupling between forward translation and rotation that appears in straight-line motion. The depth and normal priors are then projected onto that subspace. In two sentences: the sensitivity analysis from classical SfM tells us where geometry is blind, and the monocular network is allowed to fill in only those blind spots. Where cameras have good parallax, the network's bias should no longer be able to corrupt the result.

Core Claim

The closest work, MP-SfM, adds Metric3D-v2 depth and normal residuals to bundle adjustment and weights them by the network's own predicted pixel uncertainty. Its weighting answers the question "how unsure is the depth network?" It never asks "how much does the geometry already know here?" MP-SfM itself lists this as an open limitation: systematic bias in the prior propagates into the solve. Marginalized Bundle Adjustment takes another route. It makes the objective robust to monocular noise by maximizing the area under the cumulative distribution of residuals, but it still treats monocular depth as a direct observation everywhere.

Our claim is twofold. First, monocular depth error is dominated by spatially correlated bias rather than independent noise. Second, under bias, the right way to fuse the prior is to restrict it to the weakly observed subspace of the reprojection information matrix, not to scale it by a variance. Concretely, let $H$ be the information matrix (the Gauss-Newton approximation of the Hessian) of the reprojection-only objective, with eigenpairs $(\lambda_k, v_k)$. Let $\sigma_p^2$ be the monocular prior's variance. We replace the prior's information $\sigma_p^{-2} J_p^\top J_p$, where $J_p$ is the Jacobian of the prior residual, with

$$ H_{\text{prior}} = \sigma_p^{-2} \sum_k g\!\left(\lambda_k \sigma_p^2\right) \, (v_k^\top J_p^\top J_p v_k)\, v_k v_k^\top , $$

where $g$ is a gate that is 1 when the geometric information $\lambda_k$ is much smaller than the prior precision and falls to 0 when it is much larger. We will compare a hard threshold against a smooth gate $g(x) = 1/(1+x^2)$.

The gate is cheap where it matters most. For points, $H$ restricted to one point is a 3×3 block, so it can be eigen-decomposed in closed form. For cameras, we take the few smallest eigenvectors of the reduced camera system, the camera-only system left after the points are eliminated, using a Lanczos iteration. This is where the scale gauge and the forward-motion ambiguity live. If bias is the dominant monocular error, this projection should match MP-SfM in degenerate configurations and beat it wherever parallax exists.

Research Program

The work has three parts.

The first part is diagnosis. We need to test the premise that monocular depth error behaves like bias rather than noise. Using ground-truth depth on ETH3D and ScanNet, we will decompose Metric3D-v2 and Depth Anything v2 errors into a per-image affine component, a smooth low-frequency component, and a residual. We will then measure how correlated errors are across pixels. If the residual dominates and errors are nearly independent, our premise fails, and calibrated Bayesian fusion is the right answer instead. We report this first, before building anything.

The second part is the solver. We implement the gated prior inside MP-SfM's incremental pipeline, keeping its matcher, depth network, and dense consistency checks identical, so that the only change is how the prior enters bundle adjustment. We reuse MP-SfM's normal residuals, which carry orientation information that is insensitive to scale. We also study when the gate should be recomputed: every iteration, every registration, or only at initialization. A weakly observed subspace that changes during the solve could create the spurious local minima that Lawton warned about.

The third part is a controlled sensitivity study, which recasts Barron et al.'s analysis for the learned-prior era. On synthetic scenes and on real scenes with injected perturbations, we sweep three factors independently: baseline and parallax angle, keypoint noise, and structured monocular bias (a scale ramp with distance, and rescaling of individual objects to mimic semantic hallucination). The prediction is easy to state. Reprojection-only error grows as parallax shrinks, because triangulation information along a ray scales with the squared sine of the parallax angle. An always-on prior has an error floor that rises linearly with bias. The gated solver should follow the lower envelope of the two curves.

Baselines

The do-nothing reference is COLMAP with standard robust estimation and no monocular input. MP-SfM reports roughly 1% AUC@1° for it on ETH3D triplets with zero three-view overlap. It shows what geometry alone achieves.

The strongest existing method is MP-SfM. It is the only system in the file that injects monocular depth and normals into incremental bundle adjustment with per-pixel uncertainty, and it reports about 27% AUC@1° on the same zero-overlap ETH3D triplets. We will run it from its released configuration with the same Metric3D-v2 predictions, matches, and image subsets as our method. Our method changes only the prior term. Beating it means matching it within noise on zero-overlap triplets and on RealEstate10k forward motion, and exceeding it on ordinary ETH3D scenes and under injected bias. If we only match it everywhere, the bias premise is not borne out in practice.

Marginalized Bundle Adjustment is the strongest competitor that handles monocular error through a robust objective rather than weighting. It reports 97.3% relative rotation accuracy and 90.2% relative translation accuracy on ETH3D. We will run it on identical frames and depth inputs and report the same accuracy measures. As a stronger test, we will also add our gate to MBA's dense prior term, which checks whether the idea transfers across solvers.

Two ablations isolate the key ingredient. The first is calibrated Bayesian fusion: MP-SfM with its depth uncertainties recalibrated by temperature scaling on held-out ScanNet, so that variances are honest. The second is a per-point triangulation-angle gate, which switches the prior off above a parallax threshold but applies no camera-level gating. The gap between the second ablation and the full method measures what the camera-level subspace contributes. All methods get the same tuning budget: a grid over the prior weight and the gate threshold on a held-out ETH3D split, with no tuning on the test scenes.

Why not just…?

Why not just tune the prior weight λ in MP-SfM? A single global weight trades degenerate-scene robustness against well-conditioned accuracy. A conditioning-dependent weight is exactly what a constant cannot express. The tuned-λ curve on the bias sweep will show this directly: any fixed λ either has a bias floor or fails at low parallax. If one λ achieves both, our method is unnecessary, and the experiment will say so.

Why not just use well-calibrated uncertainties and standard Bayesian fusion? This is the real threat. With unbiased, independent errors, information-weighted fusion already lets geometry dominate where it is informative, and our gate would add nothing. The argument against it rests on bias. Xu et al. show that foundation depth models grow more confident while staying wrong on layered scenes, which they call converged bias. Poggi et al. show that predictive uncertainty is hard to separate from pose error even when learned carefully. A confidently biased prior, added in all directions, shifts well-constrained estimates no matter how its variance is calibrated. The diagnosis step and the temperature-calibrated ablation settle this. If calibrated fusion matches the gate under injected bias, we will report that the gate is redundant.

Why not just gate per point by triangulation angle? This handles low-parallax points. It misses camera-level ambiguities such as global scale and the forward translation–rotation confusion, where every point may have moderate parallax yet the camera solution is poorly determined. Barron et al. found worst-case amplification to be especially severe for rotation. The ablation measures this directly on RealEstate10k and on straight-driving KITTI odometry segments.

Why not let a network learn the gate, as in Depth from Motion? Depth from Motion learns a per-pixel weight between temporal stereo and single-frame semantics. Such a gate is trained on one distribution, and MAGneT-3D documents that this kind of model overfits training camera setups. Our gate is computed from the current solve, with nothing learned. We do not claim it is more accurate in-domain. The test is cross-dataset: a gate trained on KITTI should fall behind ours on ETH3D and RealEstate10k.

Proposed Experiments

Pose accuracy experiments use ETH3D zero-three-view-overlap triplets and full scenes (AUC@1°, 5°, 10°, plus relative rotation and translation accuracy), RealEstate10k forward walkthroughs, SMERF repetitive scenes, and KITTI odometry straight segments (trajectory error). The reported anchors are MP-SfM's roughly 27% and COLMAP's roughly 1% AUC@1° on ETH3D zero-overlap triplets, and MBA's 97.3% / 90.2% rotation / translation accuracy on ETH3D. No reported result was found in the Lacuna file for MP-SfM on RealEstate10k or SMERF, or for any method on Tanks and Temples, so for those benchmarks we will rerun all baselines ourselves.

The decisive check is the bias-injection sweep on well-conditioned ETH3D scenes. We add a distance-dependent scale ramp of 0–20% and per-object rescaling to the monocular depth. If MP-SfM's pose error rises with bias while the gated solver stays near reprojection-only accuracy, and the gated solver still keeps MP-SfM's gains on zero-overlap triplets, the claim holds. If calibrated Bayesian fusion closes that gap, the claim fails. A secondary check uses glass-heavy captures, where we expect converged bias to be largest. There, the gated solver should degrade more gracefully than MP-SfM wherever the cameras have parallax across the glass.