D2NT: A High-Performing Depth-to-Normal Translator

Yi Feng, Bohuan Xue, Ming Liu, Qijun Chen, Rui Fan

Introduction

Surface normal estimation is a fundamental task in computer vision and robotics, providing critical geometric information for scene understanding, obstacle detection, and visual odometry. Traditional Surface Normal Estimators (SNEs) typically follow a two-step process: converting 2D depth images into 3D point clouds and then fitting planes to estimate local normals. While accurate, these geometry-based methods are often computationally expensive. Conversely, faster methods frequently suffer from accuracy degradation at geometric discontinuities—sharp edges and ridges—where simple filtering kernels inadvertently aggregate points from distinct surfaces.

D2NT addresses this trade-off by introducing a direct "translation" from depth to surface normals, bypassing the 3D point cloud generation phase while maintaining high precision even in complex scenes.

Methodology

The proposed system utilizes a three-stage pipeline designed for both speed and robustness.

1. D2NT: The Translator

The core innovation is the D2NT translator, which derives surface normals directly from depth values ($z$) and their partial derivatives using the pinhole camera model. By mathematically linking depth gradients to surface orientation, the method eliminates the need to compute or store intermediate 3D coordinates, significantly reducing the computational footprint.

2. Discontinuity-Aware Gradient (DAG) Filter

To mitigate edge-blurring artifacts, the authors developed the DAG filter. This component employs a Laplacian operator to assess local surface smoothness. By utilizing a "truncated softmin" function, the filter adaptively adjusts weights to exclude neighboring pixels that likely belong to a different surface, thereby preserving sharp geometric boundaries.

3. Markov Random Field-based Normal Refinement (MNR)

The pipeline concludes with the MNR module, a post-processing step that treats the normal map as a graph. It employs energy minimization to enforce global consistency among neighboring normals while respecting the boundaries identified by the DAG filter. This module is modular and can be integrated with other SNEs to enhance their performance.

The D2NT pipeline, featuring the Discontinuity-Aware Gradient (DAG) filter, the core D2NT translator, and the Markov Random Field-based Normal Refinement (MNR) module.

Experimental Evaluation

The authors evaluated D2NT on the 3F2N Dataset, a synthetic benchmark categorized into "Easy," "Medium," and "Hard" scenes. Synthetic data was chosen to ensure mathematically perfect ground truth normals, avoiding the interpolation errors common in real-world datasets like NYUv2.

Key Results

D2NT demonstrates state-of-the-art performance in both speed and accuracy:

  • Efficiency: The base D2NT algorithm is exceptionally fast, with a runtime of 1.82 ms. The comprehensive D2NT V3 (including DAG and MNR) operates at 7.99 ms, well below the 33 ms threshold required for real-time 30 FPS applications.
  • Precision: D2NT V3 achieved the lowest average angular error across all test sets. In "Easy" scenes, it recorded an error of 0.89°, a significant improvement over the 1.66° achieved by the previous leading method, 3F2N.
Runtime vs. Angular Error comparison. D2NT variants (V2, V3) occupy the bottom-left quadrant, indicating superior speed and accuracy compared to existing real-time algorithms.

Discussion and Limitations

The primary strength of D2NT lies in its ability to provide high-fidelity normal maps at high speeds by treating the problem as a direct translation. However, the method's reliance on camera intrinsics ($f_x, f_y, u_0, v_0$) means that calibration errors can directly impact accuracy. Furthermore, while robust to Gaussian noise, the performance of the DAG filter under the complex noise patterns of real-world sensors (e.g., LiDAR mixed pixels) remains an area for further investigation. Future work will focus on integrating D2NT into end-to-end robotic systems for SLAM and autonomous navigation.

Summary updated

Abstract

Surface normal holds significant importance in visual environmental perception, serving as a source of rich geometric information. However, the state-of-the-art (SoTA) surface normal estimators (SNEs) generally suffer from an unsatisfactory trade-off between efficiency and accuracy. To resolve this dilemma, this paper first presents a superfast depth-to-normal translator (D2NT), which can directly translate depth images into surface normal maps without calculating 3D coordinates. We then propose a discontinuity-aware gradient (DAG) filter, which adaptively generates gradient convolution kernels to improve depth gradient estimation. Finally, we propose a surface normal refinement module that can easily be integrated into any depth-to-normal SNEs, substantially improving the surface normal estimation accuracy. Our proposed algorithm demonstrates the best accuracy among all other existing real-time SNEs and achieves the SoTA trade-off between efficiency and accuracy.
BibTeX
@inproceedings{feng2023d,
  author = {Yi Feng and Bohuan Xue and Ming Liu and Qijun Chen and Rui Fan},
  title = {{D2NT: A High-Performing Depth-to-Normal Translator}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10161000},
}
Figure