SDA-SNE: Spatial Discontinuity-Aware Surface Normal Estimation via Multi-Directional Dynamic Programming

Nan Ming, Yi Feng, Rui Fan

Abstract

SDA-SNE is a spatial discontinuity-aware surface normal estimator that uses multi-directional dynamic programming to adaptively select "inlier" points on a surface. By refining depth gradients through a novel recursive interpolation algorithm, it reduces angular errors by up to 60% near edges and ridges compared to previous state-of-the-art methods.

Introduction

In computer vision and robotics, understanding the "tilt" of a surface—its surface normal—is crucial for tasks like object recognition and scene understanding. While depth-based estimation has improved, most algorithms struggle at edges, often "blurring" boundaries between surfaces. This occurs because standard Surface Normal Estimators (SNEs) fit a plane to a fixed neighborhood, which may contain points from multiple surfaces at a discontinuity. SDA-SNE addresses this by treating surface estimation as a dynamic programming problem, adaptively selecting "inlier" points to preserve sharp features.

Methodology

The SDA-SNE framework introduces a three-pronged strategy to achieve spatial discontinuity awareness:

  1. Multi-Directional Dynamic Programming (DP): Instead of using a fixed box of neighbors, the algorithm adaptively determines which nearby pixels are "inliers" (on the same plane) by minimizing a smoothness energy along different paths.
  2. Path Discontinuity (PD) Norm: This metric evaluates the discontinuity of a path based on second-order derivatives, acting as a mathematical "guardrail" that prevents the estimator from crossing over ridges or edges.
  3. Recursive Polynomial Interpolation (RPI): Once the best neighbors are found, the depth gradient is calculated using RPI, an ultrafast iterative algorithm that achieves the accuracy of high-order polynomials with real-time performance.
The SDA-SNE framework uses multi-directional dynamic programming for pixel-wise state transfer and iterative refinement of smoothness energy and depth gradients.

Crucially, this SDA strategy is a "plug-and-play" module that can be added to existing depth-to-normal estimators to boost their performance.

Key Insights

A major realization of this work is that normal quality depends more on selecting the right neighbors than on increasing the neighborhood size. By using DP to select neighbors, the algorithm effectively "sees" the geometry before calculating the normal. Furthermore, the process converges extremely fast; the authors found that accuracy gains typically saturate after just three iterations. This efficiency, combined with a linear computational complexity of $O(MN)$, makes it suitable for high-frame-rate robotics tasks.

Experimental Results

SDA-SNE was evaluated on standard benchmarks including the 3F2N and IRS datasets. The results demonstrate a massive reduction in Average Angular Error (AAE), ranging from 20% to 60% across various levels of geometric complexity.

Qualitative comparison on a teapot model showing that SDA-SNE significantly reduces angular error at edges and ridges compared to the 3F2N baseline.

On the challenging "Hard" 3F2N dataset, SDA-SNE achieved an AAE of approximately 8.1°, nearly halving the error of the previous state-of-the-art (~15.3°). The method also demonstrates superior robustness in high-noise environments, maintaining high precision even when input depth maps are grainy.

Conclusion

SDA-SNE provides a robust, real-time solution for high-fidelity surface normal estimation. By focusing on the selection of the "right" neighbors, it preserves sharp geometric features essential for advanced robotic perception. Future work may explore integrating this geometric awareness into end-to-end deep learning architectures.

Summary updated

Abstract

The state-of-the-art (SoTA) surface normal estimators (SNEs) generally translate depth images into surface normal maps in an end-to-end fashion. Although such SNEs have greatly minimized the trade-off between efficiency and accuracy, their performance on spatial discontinuities, e.g., edges and ridges, is still unsatisfactory. To address this issue, this paper first introduces a novel multi-directional dynamic programming strategy to adaptively determine inliers (co-planar 3D points) by minimizing a (path) smoothness energy. The depth gradients can then be refined iteratively using a novel recursive polynomial interpolation algorithm, which helps yield more reasonable surface normals. Our introduced spatial discontinuity-aware (SDA) depth gradient refinement strategy is compatible with any depth-to-normal SNEs. Our proposed SDA-SNE achieves much greater performance than all other SoTA approaches, especially near/on spatial discontinuities. We further evaluate the performance of SDA-SNE with respect to different iterations, and the results suggest that it converges fast after only a few iterations. This ensures its high efficiency in various robotics and computer vision applications requiring real-time performance. Additional experiments on the datasets with different extents of random noise further validate our SDA-SNE's robustness and environmental adaptability. Our source code, demo video, and supplementary material are publicly available at mias.group/SDA-SNE.
BibTeX
@inproceedings{ming2022sda,
  author = {Nan Ming and Yi Feng and Rui Fan},
  title = {{SDA-SNE: Spatial Discontinuity-Aware Surface Normal Estimation via Multi-Directional Dynamic Programming}},
  booktitle = {3DV},
  year = {2022},
  url = {https://openreview.net/forum?id=9HjGtaSO1Oe},
}

Loading related content...

Figure