Monocular depth networks get most of an image right but fail at object outlines. Where a pole stands in front of a building, the predicted depth slides smoothly from near to far instead of jumping. When the depth map is lifted into 3D, the result is a smear of "flying pixels" hanging in mid-air between the two surfaces. These errors are confined to a thin ribbon of pixels, so they barely move whole-image error averages. They still corrupt point clouds and 3D detection. Medical image segmentation has already worked out how to spend high-resolution computation only where it is needed. A cheap global pass marks the hard regions, a learned sampler picks a handful of full-resolution patches there, and a local network refines only those patches. We propose bringing that machinery to depth, with one change that matters: inside each selected patch, the local network does not regress a single depth per pixel. It explains the patch as two depth layers, near and far, plus a mixing mask, and it computes that mask from depth evidence rather than taking it from an outside segmentation model.
The two-sentence version: a coarse depth model flags where depth probably jumps, and a learned sampler sends only those spots to a small high-resolution network. That network reconstructs each spot as a foreground sheet and a background sheet stitched by a predicted mask, so intermediate "ghost" depths cannot be produced.
Core Claim
Depth smearing comes from two separate causes, and existing fixes address only one of them at a time. The first cause is representational. A single continuous regression output is minimized by averaging two modes, which yields ghost depths. Depth Coefficients for Depth Completion addresses this by classifying depth into bins, so the network must commit to one mode. That change cut boundary error (thresholded mean absolute error on 16-row LiDAR KITTI) from 59.4 to 33.4. The second cause is resolution and compute. Boundaries need full-resolution reasoning, but applying that reasoning everywhere is wasteful.
Our claim is that a single local two-layer model, applied only inside learned boundary patches, handles both causes:
$$\hat d(p) = m(p)\, d_{\text{near}}(p) + \big(1-m(p)\big)\, d_{\text{far}}(p).$$
Here $p$ is a pixel inside a sampled patch, $d_{\text{near}}$ and $d_{\text{far}}$ are complete depth sheets predicted for the patch, and $m(p)\in[0,1]$ is a sharpened mask predicted by the same local network. The intended result is boundary accuracy that matches or beats global edge-aware losses, with markedly fewer flying pixels and a fraction of the full-resolution compute.
Relation to Prior Work
Two papers motivate the gap. Mind The Edge shows that sparse LiDAR rarely samples true silhouettes, so standard training ignores boundaries. It adds a perpendicular-gradient edge loss over the whole image and raises edge-localization AUC on KITTI-DE from 47.56% to 61.87%. ROIFormer takes the opposite stance. It restricts attention to semantic regions of interest and down-weights boundary pixels in the photometric loss because those pixels are unreliable. We read this tension as a division of labor rather than a contradiction. Photometric reconstruction is indeed untrustworthy at occlusion boundaries, so the global model should not be asked to resolve them. A dedicated stage with its own supervision should resolve them instead. ROIFormer suppresses boundary pixels. We route them to that dedicated stage.
Three papers supply the method.
- No More Sliding Window contributes the global-pass, Gumbel-Softmax top-K patch sampler, and learned aggregation. It cut computation by 91% on TotalSegmentator, but it was built only for discrete medical labels. We change what the sampler scores and what the local network outputs.
- Layered Depth Refinement with Mask Guidance contributes the near/far layer composition. It reports a relative refinement ratio of 16.569 on TartanAir. It works on the whole image, requires an external high-quality mask, and inherits every error in that mask. It also showed that simply concatenating a mask as an input channel fails. We predict the mask locally from coarse depth and image features and train it end to end.
- Polyper splits a coarse mask into an interior and a boundary band using fixed-width morphology. It shows that this split matters: removing it cost 4.40 mIoU. Our patches play the role of an adaptive band whose position and extent the sampler learns, which answers Polyper's fixed-kernel limitation.
The coarse-to-fine stereo methods Cascade Cost Volume and Cost Volume Pyramid also narrow the depth search hierarchically. They refine uniformly over the whole spatial grid and rely on multi-view matching, so they do not address single-view boundary bleeding.
Research Program
The system has three parts.
Coarse stage. Any monocular depth network run at reduced resolution. We add a small head that outputs a step map: the per-pixel probability that a depth discontinuity lies within a few pixels. On synthetic data this head is supervised with discontinuities taken from dense ground-truth depth. On real data it is supervised by consistency with the refiner's own selected masks.
Sampler. A differentiable top-K sampler, following No More Sliding Window, draws K full-resolution patches with probability proportional to the step map. A soft non-maximum suppression term penalizes overlapping patches, which addresses the redundancy problem that paper reports.
Local refiner. For each patch, a small network receives the full-resolution RGB crop, the upsampled coarse depth, and coarse features. It outputs $d_{\text{near}}$, $d_{\text{far}}$, and $m$. Both depth sheets are trained with the depth-coefficient cross-entropy over bins. The composite is trained with an $L_1$ loss and the perpendicular edge loss of Mind The Edge. A mild entropy penalty pushes $m$ toward 0 or 1 except where the evidence is genuinely sub-pixel. Refined patches are blended back into the coarse map with learned weights.
Training. The refiner is trained on dense synthetic depth from TartanAir, Hypersim, and GTA-PreSIL, where true discontinuities are labeled everywhere. On KITTI and NYUv2 the whole system is then fine-tuned with each base model's native supervision. That means sparse LiDAR for the sparsely supervised setting and photometric loss for the self-supervised setting, with ROIFormer-style down-weighting applied only to pixels that a selected patch covers. A secondary track applies the same module to depth completion on 16-row KITTI LiDAR, where the depth-coefficient result gives a direct comparison point.
The analysis asks three questions. Where does the sampler place patches compared with true discontinuities? Does the predicted mask ever disagree usefully with appearance edges, for example at same-colored objects at different depths? How do boundary metrics trade off against K, which sets the compute budget?
Baselines
Unmodified base network (do-nothing reference). We use the same monocular backbone that we attach the refiner to, trained identically without the refiner. Every gain we claim is measured against it at matched training data, including the synthetic pretraining. This isolates whether the module adds anything beyond extra data.
Global edge-aware training: Mind The Edge. This is the strongest competitor on boundary localization. It attacks the same failure with an edge loss and synthetic edge pseudo-labels, at no extra inference cost. Its reported result is KITTI-DE AUC of 61.87% (from 47.56%) in Mind The Edge. We retrain it on our backbone with the same synthetic data and the same tuning budget. Beating it means higher KITTI-DE AUC at equal or lower Abs Rel. DDAD-DE serves as a held-out check, and no reported result was found on Lacuna for DDAD-DE.
Indoor boundary error: BoRe-Depth. This is the strongest indoor reference. BoRe-Depth reports a Depth Boundary Error (DBE) of 2.083 and Abs Rel of 0.101 on NYUv2, and 0.124 Abs Rel zero-shot on iBims-1 (BoRe-Depth). Beating it means a lower DBE without losing Abs Rel, measured with a comparably small backbone.
Mask-guided layering: Layered Depth Refinement. This method supplies our composition idea, so the comparison tests whether learned local masks beat external ones. We run it with masks from a strong off-the-shelf instance segmenter on the same base predictions. We report the relative refinement ratio on TartanAir against its stated 16.569, together with flying-pixel counts.
Self-supervised outdoor context. We quote KITTI Eigen numbers to show that the module does not hurt bulk accuracy, not as a target we must beat:
- TriDepth reports Abs Rel 0.093 and RMSE 4.272 (Edge-Fattening).
- ROIFormer reports Abs Rel 0.096 and RMSE 4.148 at high resolution, running at 45 FPS on a V100 GPU.
Three ablations isolate the key ingredient:
- The same two-layer head applied densely over the whole full-resolution image, which removes selection.
- The single-output refiner inside the same patches, which removes layering.
- Patch selection by a fixed threshold on coarse depth gradient with Polyper-style fixed dilation, and separately random patches, which removes the learned sampler.
Why not just…?
Why not just train globally with the depth-coefficient loss or TriDepth's triplet loss and skip the hierarchy? This is the most serious threat, and we treat it as one. The depth-coefficient loss already cuts boundary error by nearly half on KITTI completion. Our dense two-layer ablation plus a global depth-coefficient baseline will settle it. If either matches our patch-based boundary metrics, the remaining contribution is efficiency only, and we will say so. Our reason to expect a gap is that the global versions run at whatever resolution the whole image affords. They also have no mechanism forcing a binary choice per pixel, since bins still permit two-mode outputs blended by the decoder. TriDepth additionally needs a hand-tuned margin.
Why not just concatenate a segmentation mask as an input channel, or run Layered Depth Refinement with good external masks? Layered Depth Refinement itself reports that concatenation fails because the network blurs across the mask edge. Its external-mask pipeline inherits every mask error, and appearance edges do not always coincide with depth edges. The Layered baseline above, run with a strong segmenter, tests this directly. We will report results separately for patches where appearance and depth edges disagree.
Why not just refine uniformly at full resolution, as cascade cost volumes do? Uniform refinement spends most of its compute on interiors that are already correct. Whether it also loses accuracy is an empirical question, answered by the dense-head ablation at matched parameter count. The claim we can defend regardless of that outcome is boundary metrics as a function of computation, plotted across values of K.
Why not just pick patches with a gradient threshold? Coarse depth is smeared exactly where we need it to be sharp, so its gradient is weak at thin structures such as poles and cables. The heuristic-selection and random-patch ablations measure what fraction of true discontinuities each selector covers at a fixed K.
Proposed Experiments
Main outdoor evaluation is on KITTI-DE edge AUC, with DDAD-DE as a held-out check, alongside KITTI Eigen Abs Rel and RMSE. Indoors we use NYUv2 DBE and Abs Rel, and iBims-1 zero-shot. TartanAir and Hypersim provide dense-ground-truth measurement of flying pixels. We count a reconstructed point as flying if it lies more than a fixed relative distance from both adjacent true surfaces along its viewing ray.
The depth-completion track reports tMAE on 16-row KITTI against the 33.4 reported by the depth-coefficient paper. It also reports 3D detection average precision on the Easy category, where that paper reported a rise from 36.7% to 64.9%.
We report computation (multiply-accumulates) and latency against K for every method.
Three checks are decisive:
- The patch-based two-layer refiner must beat the dense two-layer ablation on boundary-metric-per-computation and match it on raw boundary metrics.
- It must beat Mind The Edge on KITTI-DE AUC without raising Abs Rel.
- It must cut flying-pixel rate relative to Layered Depth Refinement on patches where appearance and depth edges disagree.
If the first check fails on accuracy but holds on compute, the paper becomes an efficiency result. If the third fails, depth-derived masks offer no advantage over appearance masks and the layering claim is withdrawn.
References
- Depth Coefficients for Depth Completion
- Mind The Edge: Refining Depth Edges in Sparsely-Supervised Monocular Depth Estimation
- ROIFormer: Semantic-Aware Region of Interest Transformer for Efficient Self-Supervised Monocular Depth Estimation
- No More Sliding Window: Efficient 3D Medical Image Segmentation with Differentiable Top-K Patch Sampling
- Layered Depth Refinement with Mask Guidance
- Polyper: Boundary Sensitive Polyp Segmentation
- Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
- Cost Volume Pyramid Based Depth Inference for Multi-View Stereo
- BoRe-Depth: Self-Supervised Monocular Depth Estimation with Boundary Refinement for Embedded Systems
- Self-Supervised Monocular Depth Estimation: Solving the Edge-Fattening Problem