Tangent-Plane Deformable Convolution for Depth Refinement

Depth completion turns a sparse set of laser (LiDAR) depth measurements plus a color image into a dense depth map. A strong recent recipe first predicts a rough dense map, then refines it with a deformable convolution, a convolution whose sampling points are not fixed to a square grid but are shifted by learned 2D offsets for each pixel. The refinement layer then fills each pixel from well-chosen neighbors. The shifts are learned in the image plane with no knowledge of 3D shape, and they create a problem that has not been named. When a kernel moves its sampling points to one side of an object edge to avoid the background, it averages depths from neighbors that all lie on one side of the pixel. On a tilted surface such as a road, a wall seen at an angle, or a table top, that one-sided average is biased toward whichever side the kernel moved to. So the mechanism that avoids depth bleeding across edges also produces a systematic depth error on slanted surfaces near those edges.

We propose a deformable convolution that places its sampling points in 3D on a predicted local tangent plane and projects them into the image through the camera. Each neighbor's depth is then moved back to the center pixel along that plane before averaging, instead of being averaged raw. The change is small and runs in a single pass. It makes one-sided sampling unbiased on planar patches. It also gives a natural rule for rejecting neighbors: a neighbor whose measured depth disagrees with the plane probably sits across a 3D break and gets its weight reduced. The core claim is that plane-based transport, and not the gating alone or an added geometric loss, is what removes the remaining error of deformable depth refinement near slanted boundaries.

Where this sits

The method comes from Revisiting Deformable Convolution for Depth Completion (ReDC). ReDC showed that deformable convolution fails on raw sparse depth but works well when applied once to a coarse dense map, reaching 728.31 mm RMSE on the KITTI test set. Its offsets are unconstrained 2D displacements, and its output combines raw sampled values. We keep its placement, a single refinement pass on a dense coarse map, and change what the offsets mean and how the sampled depths are combined.

The closest geometric relative is NTPP-MVSNet. It pairs deformable neighbor sampling with local tangent planes, but it does so in multi-view stereo. There, tangent planes projected by homography produce depth hypotheses for a cost volume, and the deformable offsets themselves stay unconstrained. We work from a single view with sparse LiDAR and no cost volume, and the plane directly sets where taps land and how their values are carried back to the center.

Three lines of work motivate the proposal by showing that 3D-aware neighborhoods matter:

  • GraphCSPN rebuilds spatial propagation as a graph over 3D nearest neighbors. It needs only 3 iterations, where earlier propagation networks needed 18–24.
  • DeCoTR lifts features to a point cloud and applies point transformers.
  • Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints diffuses a plane-origin distance, the distance from the camera center to a pixel's tangent plane, which stays constant along a planar surface.

We reuse that last paper's quantity as our transport rule. Unlike all three, we build no graph, run no point transformer and do no iterative diffusion. The 3D reasoning is folded into the sampling pattern of one convolution.

Virtual Normal supervises depth networks by comparing normals of planes spanned by distant point triplets. It constrains this proposal, because it is the obvious "just add a loss" alternative that we must beat. DCNv2 adds a learned per-tap weight in $[0,1]$, the modulation, and shows that unguided offsets drift onto background. We tie that weight to plane consistency.

Core mechanism

The layer works as follows at a pixel $p_0$ with coarse depth $d_0$, camera intrinsics $K$ and viewing ray $r_0 = K^{-1}\tilde p_0$ (where $\tilde p_0$ is the pixel in homogeneous coordinates):

  1. A light head predicts a unit normal $n$ and, for each of the $N$ taps, two in-plane coefficients $(a_n, b_n)$.
  2. Each tap is placed in 3D at $X_n = d_0 r_0 + a_n e_1 + b_n e_2$, where $e_1, e_2$ span the plane orthogonal to $n$.
  3. The tap is projected to an image location $p_n = \pi(K X_n)$, with $\pi$ the perspective division. The 2D offset of an ordinary deformable convolution is therefore produced by the plane rather than predicted freely.
  4. At $p_n$ the layer reads the coarse depth $D(p_n)$ and forms the observed point $\hat X_n = D(p_n) K^{-1}\tilde p_n$.
  5. Because the plane-origin distance $n\cdot X$ is constant on a plane, each tap gives an estimate of the center depth. The refined depth combines these estimates:

$$\hat d_0 = \sum_{n=1}^{N} \omega_n \frac{n\cdot \hat X_n}{n\cdot r_0}, \qquad \omega_n \propto w_n\, m_n\, \exp\!\left(-\frac{|D(p_n) - z^{\text{plane}}_n|}{\tau}\right).$$

In this formula, $w_n$ are the learned kernel weights, and $m_n$ is the DCNv2-style learned modulation. The term $z^{\text{plane}}_n$ is the depth the plane predicts at $p_n$, and $\tau$ is a learned temperature. The weights $\omega_n$ are normalized to sum to one. The exponential factor down-weights taps whose measured depth leaves the plane, which is a soft 3D break detector.

The same structure is applied to features in intermediate refinement layers. The normals are supervised with normals computed from ground truth where it is dense enough (NYU-Depth-v2). On KITTI, where ground truth is semi-dense, we use a Virtual Normal loss instead. A residual correction head can add high-frequency detail that planes cannot represent, such as thin poles.

Research Program

The work has three parts.

The first is a controlled diagnosis. On the same coarse backbone, we measure how ReDC-style free offsets distribute their taps near depth edges. We then check the prediction that error is concentrated where two conditions meet: the local surface is slanted, and the taps are asymmetric about the center.

The second part implements the tangent-plane layer as a drop-in replacement for the refinement module. We use a custom sampling kernel so that its latency can be compared honestly with ReDC's single pass.

The third part tests generalization in the two settings where the field reports weaknesses. One is indoor scenes with many tilted planes (NYU-Depth-v2), where ReDC was never evaluated. The other is long range (DDAD, up to 250 m), where DeCoTR reports degradation for models trained within KITTI's 80 m. In both settings, transporting depth along a plane should help more than averaging raw values, because perspective foreshortening makes image-plane averages more biased at a distance.

Baselines

The naive reference is the coarse backbone output with no refinement, together with an ordinary rigid 3×3 refinement layer of matched parameter count. This baseline shows how much any refinement buys.

The strongest directly comparable method is ReDC, which reports 728.31 mm RMSE and 204.60 mm MAE on the KITTI test set. We will reproduce its deformable refinement module on our shared backbone, with the same training data, schedule, number of epochs and hyperparameter search budget. That way the only difference is the refinement layer. We will also quote its published number.

The strongest 3D-aware competitors are GraphCSPN, with 738.41 RMSE on KITTI and 0.090 m on NYU-Depth-v2 using 3 iterations, and DeCoTR, with 0.086 m RMSE on NYU-Depth-v2 and 0.188 zero-shot on ScanNet. We will run them from their released settings where possible and report their published figures otherwise. On NYU we also list NLSPN at 0.092 with 18 iterations, the number reported in GraphCSPN.

Three ablations isolate the key ingredient. The first is ReDC plus a Virtual Normal loss. The second is ReDC plus our 3D-consistency gate but with raw-value averaging, which isolates transport from gating. The third is our layer with transport but without the gate.

Beating the field would mean several things. On the same backbone we should lower KITTI RMSE below our reproduction of ReDC. On the slanted-near-edge subset, the gain should be clearly larger than the overall gain. On NYU-Depth-v2 we should match or beat GraphCSPN's 0.090 m in a single pass. On NYU we aim to approach DeCoTR's 0.086 m at lower latency, without claiming to exceed it.

Why not just…?

Why not just add a Virtual Normal loss to ReDC? A loss changes what the network is rewarded for, but it does not change how the refinement layer combines values. With one-sided taps, a raw weighted average of slanted-surface depths stays biased unless the offsets happen to stay symmetric. Staying symmetric is exactly what boundary avoidance prevents. Virtual Normal itself leaves sharp boundaries and thin structures as open problems. The ReDC+VN ablation tests this directly. If it closes the gap on the slanted-near-edge subset, our transport claim is wrong.

Why not just gate ReDC's taps by 3D distance in the coarse map, like a bilateral filter? This is the most serious threat. It is cheap, and it may capture most of the benefit at genuine breaks. What it cannot do is remove the bias on continuous slanted surfaces, because the taps it keeps are still averaged raw. The gate-only ablation is designed to settle this. If gate-only matches the full layer on the slanted-near-edge subset and on DDAD, we will report that the gate is the contribution and drop the tangent-plane parameterization.

Why not just use 3D graphs or point transformers, as GraphCSPN and DeCoTR do? They are well supported, but they require nearest-neighbor searches, downsampling into patches or point sets, and sometimes iterations. GraphCSPN needs patch approximations, and DeCoTR needs downsampling. The comparison is accuracy at matched latency and memory at full resolution. If they win at equal cost, the case for our layer rests only on simplicity.

Why not just start from a monocular foundation model, as LDCM does? LDCM regresses full 3D points and uses a DepthAnythingV2 prior, which improves zero-shot robustness. That is a complementary choice of backbone, not a refinement layer. We will test our layer on top of a foundation-model coarse map as well, to check that its gain does not disappear when the coarse map is already good.

Proposed Experiments

Training and testing use KITTI Depth Completion and NYU-Depth-v2, with zero-shot transfer to DDAD and ScanNet. Beyond RMSE, MAE and relative error, we report four diagnostics:

  • RMSE restricted to pixels within a few pixels of ground-truth depth edges, binned by the angle between the surface normal and the viewing ray;
  • the fraction of "flying pixels" whose depth falls between the foreground and background at a break;
  • mean normal-angle error on NYU;
  • latency per frame.

The decisive check is the slanted-near-edge bin. The theory predicts that the advantage of transport over raw averaging grows with surface slant and with tap asymmetry, and vanishes on fronto-parallel surfaces, which face the camera squarely. If the measured gain does not follow this pattern, the mechanism is not what we claim, whatever the headline RMSE shows. A second check is range. The DDAD gain should grow with distance, as foreshortening grows. A third check is robustness when the coarse map is deliberately degraded with noise and holes. The 3D gate should prevent the degradation from spreading through the refinement, which answers the dependence on backbone quality that the research file notes for ReDC and the lifting methods.