Regularize Where the Image Cannot Decide: Ambiguity-Weighted Manhattan Priors for Single-Image Depth

A single photograph does not determine 3D shape. Every pixel could lie anywhere along its viewing ray, so a depth network has to fill in what the image leaves open with learned guesses. Those guesses are good on average but geometrically sloppy. Networks trained with per-pixel losses often score low pixel error while producing walls that look bumpy and corners that are rounded off in the reconstructed point cloud, because no term ties pixels together into coherent planes. Indoor scenes make this worse. Large featureless walls and ceilings are where the image gives the least evidence, and they are also exactly the surfaces that the Manhattan world assumption describes well. That assumption says built environments are mostly planes facing a few mutually perpendicular directions.

This note proposes a test-time refinement for any pretrained single-image depth model. We solve a small optimization that keeps the network's depth where the network is confident. Where the network is uncertain, the optimization pulls surface orientations toward a small, automatically discovered set of perpendicular "frames," and it includes an explicit "none of the above" option for clutter. The new ingredient is what controls how strongly the structural prior acts at each pixel. Earlier methods weight smoothness by image gradients or photometric error. We weight it by a direct estimate of how ambiguous that pixel's depth is. The two-sentence version is this: apply the Manhattan prior in proportion to how unsure the depth model is, and let it override nothing the model actually knows. We expect this to flatten ambiguous walls and sharpen room corners without blocking up furniture or erasing real depth edges.

Core Claim

The weakness of spatially adaptive regularization is well documented. Weights computed from image gradients cannot tell a depth step from a painted edge or a shadow, so they switch smoothing off on flat but textured walls and produce ripples. Learned weight maps carry their own risk: they lack convergence guarantees inside iterative solvers. The Manhattan prior has the opposite profile. It is global and geometric rather than local, but enforced rigidly it creates blocky artifacts on clutter and spurious frames. Our claim is that the right quantity to adapt on is the depth estimator's own ambiguity, not image appearance. Coupling that ambiguity to a soft mixture-of-frames prior with an outlier class fixes the gradient method's photometric-versus-geometric confusion and the Manhattan method's clutter brittleness together. The weights come from a measurable uncertainty rather than a free network output, so the inner problem stays a fixed, well-behaved objective.

Concretely, let $\hat D$ be the network's depth map and $\sigma(x)$ an estimate of its log-depth uncertainty at pixel $x$. Let $n_x(D)$ be the surface normal implied by a candidate depth map $D$, and let $\{R_k\}$ be a small set of rotations whose axes define the allowed orientations. We minimize

$$E(D,\{R_k\},z) = \sum_x \frac{\rho\big(\log D(x) - \log \hat D(x)\big)}{\sigma(x)^2} + \beta \sum_x \Big[\, z_x \,\min_{k,a} \big(1 - |n_x(D)^\top R_k e_a|\big) + (1 - z_x)\, c_0 \Big].$$

In this objective, $\rho$ is a robust penalty and $e_a$ ranges over the three coordinate axes. The variable $z_x \in [0,1]$ is a soft assignment of pixel $x$ to the frame prior rather than to the outlier class, which costs a constant $c_0$. The scalar $\beta$ sets the overall prior strength. Dividing the data term by $\sigma(x)^2$ makes the effective regularization strength $\lambda(x) \propto \beta\,\sigma(x)^2$. Confident pixels are essentially pinned to the network's answer. Uncertain pixels are free to snap to a frame, but only when some frame explains them better than the outlier cost.

Positioning Against Prior Work

The closest bridge between adaptive regularization and single-image depth is Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Prediction. That work weights a second-order smoothness term per pixel by $\exp(-\gamma r(x))$, where $r$ is the stereo photometric reprojection residual. It motivates our design, since it shows per-pixel weighting helps, but it differs from us in two ways. Its weight is photometric, so it inherits the failure on textureless indoor surfaces where photometric matching collapses. Its prior is plain smoothness with no global orientation structure. We replace the photometric signal with depth uncertainty and the smoothness term with a shared set of frames.

Three works supply the structural machinery. A Mixture of Manhattan Frames: Beyond the Manhattan World generalizes one frame to $K$ frames on the sphere of normals, inferred by a Gibbs sampler with split and merge moves. We reuse its mixture model but replace sampling with expectation-maximization, because the file reports that sampling-based frame inference takes minutes per frame. Surface Normal Clustering for Implicit Representation of Manhattan Scenes discovers frames inside a multi-view radiance field by k-means on normals, adding clustering and orthogonality losses. Its weights are uniform, it needs multiple views, and it has no outlier class. AtlasGS enforces an Atlanta-world prior on Gaussian splatting. The Atlanta world keeps one vertical gravity direction and allows any number of wall directions. AtlasGS is also multi-view, and the file notes it breaks under uncalibrated pitch and roll or slanted ceilings. Our mixture includes tilted frames and so does not depend on gravity alignment.

The file also lists Virtual Normal, ASN, P2Net, NDDepth, Kim and Manduchi, Civera et al., and PAGCNet as connecting depth with planar or Manhattan structure. It gives no methodological details or identifiers for them. We therefore treat "add a geometric or planar consistency loss to monocular depth training" as already explored, and we do not claim the combination of Manhattan priors with monocular depth as new.

On the constraint side, Layout-Guided Novel View Synthesis From a Single Indoor Panorama documents how cuboid room assumptions hallucinate walls in L-shaped or slanted rooms. This is why we include both a mixture of frames and an outlier class. Deeply Aggregated Alternating Minimization for Image Restoration shows that predicting spatial hyperparameter maps beats scalar schedules in alternating minimization. That supports our alternating solver, but we derive the map from uncertainty instead of learning it, which avoids the instability that end-to-end weight maps are known for.

Research Program

The work has three components. The first is an ambiguity estimate that works for any off-the-shelf depth network. We compare three options. Test-time augmentation variance uses flips, small rescalings, and crops, and needs no retraining. A lightweight variance head trained with a Gaussian negative log-likelihood on held-out data is the second option. The third is a structural proxy combining distance to the nearest predicted occlusion boundary with local image entropy. The proxy exists mainly as a control, to test whether "uncertainty" is secretly just "texturelessness."

The second component is the solver. It alternates three steps. First, it re-estimates frames by expectation-maximization over normals under a von Mises–Fisher mixture, a Gaussian-like distribution on the sphere. Orthogonality is projected within each frame, and $K \le 4$ is chosen by an information criterion. Second, it updates the soft assignments $z_x$ in closed form. Third, it runs a few Gauss–Newton steps on log-depth. With assignments and frames fixed, the depth step is a smooth robust least-squares problem, so each round decreases $E$ monotonically. We target well under a second per image at standard indoor resolution.

The third component is an analysis of when the prior helps. We bin pixels by $\sigma(x)$ and by ground-truth surface type: planar-Manhattan, planar-oblique, or non-planar clutter. We then measure where error changes.

Baselines

The naive reference point is the pretrained depth network's raw output with no refinement. Every method below starts from that same output, uses the same networks, and runs on the same evaluation images. Hyperparameters ($\beta$, $c_0$, and the counterpart weights in the baselines) are tuned with equal search budgets on the same validation split.

The first structural baseline is per-image RANSAC plane fitting with snapping. Planes are detected on the back-projected point cloud and inlier points are projected onto them. It is the simplest strong post-processing a practitioner would try, and it directly attacks bumpy walls.

The second is a uniform-weight Manhattan refinement. It uses our exact energy with $\sigma(x)$ replaced by a constant, so it amounts to a single-image, depth-domain version of the clustering-and-orthogonality idea in Surface Normal Clustering for Implicit Representation of Manhattan Scenes.

The third is an edge-aware weighting baseline. It uses our energy but sets $\lambda(x)$ from inverse image gradient magnitude, the analytical strategy described for low-light enhancement in Joint Enhancement and Denoising Method via Sequential Decomposition. It is also a stand-in for the residual weighting in Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Prediction, since a single image has no stereo residual. Where code allows, we also train Wong and Soatto's model on its native stereo setting and apply our refinement on top.

The key ablation is a shuffled-uncertainty control. It keeps the histogram of $\sigma(x)$ but randomly permutes the values spatially. Any gain that survives this shuffle is not due to where the uncertainty is.

For the strongest existing method on our target benchmarks, we found no reported result on Lacuna for any of these baselines on NYU Depth v2, ScanNet, or a comparable indoor benchmark. The file contains no numeric results at all. We will therefore report every baseline ourselves under the shared protocol. Beating the best baseline means two things on held-out data. On Manhattan-planar regions, planarity and corner-angle error must improve with non-overlapping bootstrap confidence intervals. On clutter regions, depth error must not degrade beyond its interval.

Why not just…?

Why not just use a bigger pretrained depth model? Data-driven priors keep improving, as surveyed in Liu et al., and this is a real threat to the proposal. If the strongest available network already produces planar walls and square corners, our refinement will show no gain. Our method sits on top of any network, so the experiment settles the question directly: we run it on the strongest network we can obtain. We report the result as a negative finding if the improvement on planarity metrics vanishes. We expect a residual gap because pixel-wise training objectives contain no coplanarity term, but we treat that expectation as a hypothesis, not a premise.

Why not just fit planes with RANSAC afterwards? Plane fitting decides per region, with no shared orientations. It cannot make two disconnected walls exactly perpendicular, and it snaps confidently predicted geometry just as readily as ambiguous geometry. RANSAC is a baseline, and two experiments separate it from our method. The corner-angle metric tests whether shared frames matter. The clutter-region error tests whether hard snapping damages non-planar objects that our outlier class should leave alone.

Why not just weight smoothing by image edges? The file identifies the core failure of gradient-based weights: they mistake albedo and shadow edges for depth edges. The edge-aware baseline uses the identical solver and frame prior, so any difference is attributable to the weighting signal alone. A targeted subset of textured flat surfaces, such as patterned walls, posters, and tiled floors, is where we predict the gap is largest.

Why not just add a Manhattan normal loss during training? The file lists several works that do exactly this, which is why we do not claim it. Training-time losses fix one trade-off between fidelity and structure for every pixel and every scene. They also cannot adapt to a new building at test time. Where a released checkpoint of such a training-time method exists, we include it as an additional starting network, and we test whether our refinement still adds on top of it.

Proposed Experiments and Decisive Checks

We will evaluate on standard indoor RGB-D benchmarks with ground-truth depth, such as NYU Depth v2 and ScanNet, and on a synthetic structured-room set with exact plane labels. The file names no benchmarks or reported numbers, so all figures will come from our own runs. We report absolute relative error and the fraction of pixels within a 1.25 ratio of ground truth. We also report geometry metrics: plane-fit residual on ground-truth planar segments, angular error of normals, and angle error at wall–wall and wall–floor junctions. Every metric is split by surface type and by uncertainty bin.

Four checks decide whether the idea works. First, gains must concentrate in high-$\sigma$ bins and be near zero in low-$\sigma$ bins; otherwise the weighting is not doing what we claim. Second, ambiguity weighting must beat both uniform weighting and edge weighting, especially on textured flat surfaces, and the shuffled-uncertainty control must lose most of the gain. Third, on scenes that violate the strict Manhattan assumption, the mixture and outlier class must avoid the degradation that a single-frame cuboid-style prior shows, measured on slanted ceilings, L-shaped rooms, and heavy clutter. Fourth, the result must hold on at least two different base networks, including the strongest one available, so that it is not specific to one network's error pattern.

If the first or second check fails, the claim is wrong and the method reduces to ordinary Manhattan refinement. If only the strongest network shows no gain, we would conclude that large data-driven priors have absorbed the structure, and the useful contribution narrows to weaker networks and to robotics settings where the method's speed matters.