Placing Geometric Edges: A Discontinuity-Location Loss for Joint Depth and Normal Estimation

Networks that predict depth (distance from the camera per pixel) and surface normals (the 3D direction each surface faces) from a single image still fail where it matters most for 3D use: at edges. Where one object occludes another, they produce "flying pixels" smeared between foreground and background. Where two walls meet, they produce rounded corners. The usual fix is to give boundary pixels more weight in the loss. Our argument is that this cannot work by itself. Giving a pixel more weight does not change what the best prediction for that pixel is. It only changes how much training effort the pixel attracts. A pixel that is equally likely to be foreground or background keeps the same compromise answer however heavily it is weighted.

We propose a loss on where the predicted geometric edges are, rather than on how wrong individual edge pixels are. The method extracts soft edge maps from the predicted depth and normals, separating two physically different kinds of edge. A depth step is an occlusion, where depth jumps. A crease is a fold, where depth is continuous but the normal turns. The loss then pushes each kind of predicted edge toward the matching ground-truth edge, in the style of active contours. The same learned distance-to-edge signal decides, per pixel, how much wide-context information the network mixes in. This gives large receptive fields on flat interiors and small ones next to edges. The outcome would be a supervised training objective, with no external segmentation step, that targets edge placement in continuous geometry, together with a direct test of whether simpler reweighting already achieves the same thing.

Core Claim

Existing joint depth-normal methods couple the two outputs through geometry but supervise them with pixel-wise losses. GeoNet does this with least-squares depth-to-normal and kernel-regression normal-to-depth modules. ASN does it with normals from sampled point triplets, weighted by learned feature similarity so that triplets avoid crossing edges. MIDNet uses cross-attention between depth and normal streams on a high-resolution backbone (HRNet, which keeps full-resolution features throughout). NDDepth uses plane-distance parameterization with Felzenszwalb segments. In all four, edges are at most avoided or implied. None of them is supervised for location.

On the other side, the Active Boundary Loss (ABL) moves predicted segmentation contours toward ground-truth contours using distance transforms, and beats post-hoc edge refinement (SegFix) in boundary F-score. However, it is defined only for discrete class probabilities.

Our claim is that an ABL-style placement loss, rebuilt for continuous geometry and split by edge type, sharpens both occlusion and crease edges in a way that matched-budget boundary reweighting does not. The pointwise argument makes this concrete. For a pixel whose true depth is $d_1$ or $d_2$ with some probabilities, the minimizer of $w\,\mathbb{E}|\hat d - d|$ is the same median for every weight $w > 0$. Any per-pixel weighting therefore leaves the ambiguous pixel's optimal answer unchanged. A loss over the arrangement of neighboring pixels is different: it rewards a single step at the most likely location and does not reward a smear.

Research Program

Edge extraction from predictions. For neighboring pixels $p$ and $q$, we define two soft edge strengths. The occlusion strength is $o_{pq} = \sigma\big((|\log \hat d_p - \log \hat d_q| - \tau_d)/s\big)$. The crease strength is $c_{pq} = (1 - o_{pq})\,\sigma\big((1 - \hat n_p^\top \hat n_q - \tau_n)/s\big)$. Here $\hat d$ is predicted depth, $\hat n$ is the predicted unit normal, $\sigma$ is the logistic function, $\tau_d$ and $\tau_n$ are thresholds, and $s$ is a temperature. The same formulas applied to ground-truth depth and normals give target edges.

Using the log-depth ratio makes the occlusion test scale-aware, which matters for distant outdoor geometry. The factor $(1 - o_{pq})$ stops a depth jump from also being counted as a crease. Because both edge maps come from geometry, a high-contrast poster on a flat wall produces no target edge. This is the failure mode that ASN documents for detectors driven by image intensity.

Placement loss. We follow ABL. At pixels where the predicted edge strength is high, we compute the ground-truth distance transform for that edge type. We then apply a cross-entropy loss that moves edge strength toward the 8-neighbor closest to the true edge, using ABL's gradient-detachment trick to suppress conflicting updates. ABL needs a warm-up before contours can move, so the placement loss is switched on only after a standard pixel-loss phase. We report sensitivity to this schedule.

Type-aware consistency. The depth-normal consistency term follows ASN or GeoNet, so geometry is still coupled. The difference is that this term is masked where predicted occlusion strength is high, because normals are undefined across an occlusion. Across creases it is kept, but evaluated separately on each side.

This also addresses a disagreement in the literature. Occlusion-aware multi-view depth with CNM constraints down-weights occlusion pixels to avoid flying pixels, whereas ABL up-weights boundaries. In our scheme, the per-pixel depth error at an occlusion edge is down-weighted, while the edge's position is actively supervised.

Edge-gated context. We add a small head that predicts a distance-to-geometric-edge field. It is trained through the placement loss by supervising it with the ground-truth distance transform. This field produces per-pixel softmax weights over the branches of an atrous spatial pyramid pooling (ASPP) module, meaning parallel convolutions with different dilation rates. Non-monotonic dilation rates are used to avoid the gridding artifacts reported for augmented ASPP.

This multi-scale component is deliberately secondary. Its hypothesis can be falsified: gating should help only when the placement loss trains the distance field. Without that loss, it should collapse to ordinary ASPP.

Host networks. We implement the method on ASN's HRNet-48 pipeline and on a lighter ResNet-50 GeoNet-style two-stream model. The question is whether the gains hold across capacity.

Baselines

The naive reference point is the host network trained with its own published losses: ASN's depth loss plus adaptive normal constraint, and GeoNet's pixel-wise depth and normal losses. These models are retrained by us on identical data splits, resolution, and schedule, with the same hyperparameter-search budget given to every variant. For context we also quote published NYUv2 results.

ASN reports Rel 0.101 and normal mean angle error 20.0°, and cites VNL at 0.108 and 23.7°. GeoNet reports normal mean angle error 19.0° and RMSE 0.569 m with ResNet-50. MIDNet reports RMSE 0.427. NDDepth reports Abs Rel 0.087 on NYUv2 and SILog 9.62 on KITTI.

NDDepth is the strongest published depth competitor, and on its own terms it is a monocular depth method with a normal-distance head. We retrain it under our budget where feasible. Beating it on whole-image metrics is not our claim. Our claim is improved edge metrics with Abs Rel no worse than the matched host baseline by more than its seed-to-seed spread.

For edge-specific metrics we found no reported result on Lacuna for any of these methods. All edge numbers will therefore come from our own reruns.

The ablation that isolates the key ingredient is a matched-mass boundary reweighting baseline. It uses the same geometry-derived, type-split ground-truth edges and the same narrow trimap band. The band pixels are weighted so that the total extra loss mass equals the placement loss's mean contribution, but the loss stays pixel-wise. If this control matches our method, then location supervision adds nothing and the core claim fails.

The remaining ablations run on the same budget:

  • an untyped variant that merges occlusions and creases;
  • removing consistency masking;
  • the CNM paper's occlusion down-weighting alone;
  • edge-gated ASPP with and without the placement loss, against plain ASPP and plain HRNet.

Why not just…?

Why not just weight the pixel loss by ground-truth depth and normal discontinuities? This is the most serious threat. It uses the same clean, geometry-derived edges and costs almost nothing. Our pointwise argument predicts that it will improve average error near edges, because more capacity goes there, but that it will leave flying pixels at occlusions where the side is ambiguous. The matched-mass control described above tests this directly. If flying-pixel counts and edge-location error are statistically indistinguishable between the two, we would report that reweighting suffices and drop the placement loss.

Why not just weight by image gradients, as edge-aware consistency does? Edge-aware depth-normal consistency weights smoothness by RGB gradients. ASN shows that texture and shadows create false geometric edges on flat surfaces under such weighting, and that edges between surfaces of the same color go unseen. We include this weighting as a baseline. We expect it to hurt normals on textured planes, and we will measure normal error on a texture-rich subset of the NYUv2 test set.

Why not just use ASN's feature-similarity sampling, or a bigger high-resolution backbone? ASN already keeps triplets from crossing edges, and MIDNet shows that HRNet plus cross-task attention lowers RMSE. Both, however, act on how features are aggregated, not on where the predicted step sits. Our method is built on top of ASN for exactly this reason: any gain is then measured beyond that mechanism. The HRNet-48 versus ResNet-50 comparison shows whether extra capacity alone closes the edge gap.

Why not just refine edges after prediction? SURGE argues that color-driven conditional random fields over-smooth objects of the same color, and ABL beats SegFix without extra inference cost. We still run joint bilateral filtering guided by RGB on the baseline's outputs as a post-hoc reference, so the comparison is made on our own metrics.

Proposed Experiments

The main benchmark is NYUv2, which has paired depth and normals. Models are trained on the standard raw frames and tested on the 654-image split.

Edge metrics need care. Kinect ground truth is noisy at depth edges, and the filled depth contains inpainting artifacts. We therefore extract ground-truth edges only where valid raw depth exists on both sides, and we report results with and without filled-depth pixels. ScanNet serves as a cross-dataset test, following ASN. KITTI tests outdoor transfer, where the planar priors of earlier methods are known to struggle. Because LiDAR ground truth is sparse, on KITTI we report SILog and Abs Rel only, plus flying-pixel counts at occlusion edges detected from projected LiDAR.

Whole-image metrics are Abs Rel, RMSE, and normal mean and median angle error. Edge metrics are:

  • precision and recall of predicted occlusion and crease edges within a 1–3 pixel tolerance, reported separately by type;
  • mean distance from predicted to true edges;
  • the fraction of point-cloud points lying between foreground and background depth within a band around occlusions, which measures flying pixels;
  • normal angle error inside a crease band.

Every variant is trained with three seeds.

The decisive checks are as follows. First, the placement loss must beat the matched-mass reweighting control on edge-location error and flying pixels by more than the spread across seeds, while staying within seed spread on Abs Rel. Second, the typed variant must beat the untyped one on crease-band normal error. Third, edge-gated ASPP must help only when the placement loss is present. A failure of the first check refutes the core claim. A failure of the third removes the multi-scale component, leaving the loss as a standalone contribution.