Three-Filters-to-Normal+: Revisiting Discontinuity Discrimination in Depth-to-Normal Translation

Jingwei Yang, Bohuan Xue, Yi Feng, Deming Wang, Rui Fan, Qijun Chen

TL;DR

This paper introduces 3F2N+, an enhanced surface normal estimator that significantly improves accuracy in regions with depth discontinuities by incorporating novel discontinuity discrimination modules. By leveraging depth curvature and correlation coefficients, 3F2N+ achieves state-of-the-art performance on challenging datasets and demonstrates its utility in downstream robotics tasks.

What Problem Does This Paper Address?

Estimating surface normals from depth data is crucial for many computer vision and robotics applications, such as object recognition, scene understanding, and navigation. Existing methods, particularly "depth-to-normal translators," often struggle with accuracy in areas where depth changes abruptly (discontinuities), such as edges of objects or changes in surface orientation. These discontinuities can lead to noisy or incorrect normal estimations, which propagate errors into subsequent tasks. Furthermore, real-world depth data often lacks ground truth surface normals, making robust evaluation difficult. This paper aims to address these limitations by developing a more robust surface normal estimator that specifically handles discontinuities and by providing a more realistic synthetic dataset for evaluation.

Methods

The core contribution of this paper is the Discontinuity Discrimination Module (DDM), which is integrated into their previous work, 3F2N, to create 3F2N+. The DDM combines two key strategies:

  • Depth Curvature Minimization: This approach quantifies the local "smoothness" of a surface. Lower depth curvature indicates a smoother surface, while higher curvature suggests a steeper change, potentially indicating a discontinuity. The paper explores four types of curvatures (mean, maximum, normal, and Gaussian) and finds that mean curvature offers the best performance. The surface normal is then estimated by selecting the neighboring point with the smallest depth curvature.
  • Correlation Coefficient Maximization: This method uses statistical measures (Kendall's and Pearson's correlation coefficients) to assess the consistency of depth values between a target point and its neighbors. Higher correlation suggests that the points likely belong to the same continuous plane. The surface normal is chosen from the neighbor with the maximum correlation.

These two strategies are combined within a Conditional Random Field (CRF) framework. The CRF models the relationships between pixels and their surface normals, using the discontinuity measures from the DDM to penalize incorrect normal estimations near discontinuities. This allows for a more optimized and accurate surface normal estimation, especially in challenging regions.

Datasets

The paper introduces a new, large-scale synthetic dataset called the Synthetic Surface Normal (SSN) dataset. This dataset comprises 20,000 depth images generated from 20 3D mesh models, covering both indoor and outdoor scenarios. Crucially, the SSN dataset includes versions with and without added random Gaussian noise, providing a more realistic evaluation platform for surface normal estimators compared to existing noise-free datasets. The authors also utilize their previously introduced 3F2N datasets (Easy, Medium, Hard) for evaluation.

Key Insights and Ideas

  • Discontinuities are the Achilles' heel: The paper strongly emphasizes that the performance of surface normal estimators is heavily dictated by their ability to handle depth discontinuities. This is a critical insight for practical robotics applications where such discontinuities are ubiquitous.
  • Geometry and Statistics are a powerful duo: Combining geometric cues (depth curvature) with statistical measures (correlation coefficients) provides a more comprehensive understanding of surface properties and discontinuity likelihood.
  • Mean curvature reigns supreme: Among the explored geometric measures, mean curvature proved to be the most effective for discontinuity discrimination, highlighting its robustness.
  • DDM as a plug-and-play module: The proposed DDM is designed to be a modular component that can be integrated into various surface normal estimation methods, offering a generalizable solution for improving robustness.
  • Realistic datasets are essential: The creation of the noisy SSN dataset underscores the need for more challenging and realistic benchmarks to truly assess the performance of algorithms in real-world conditions.

Key Results

  • State-of-the-art accuracy: 3F2N+ achieves the lowest average angular errors across various datasets, including the new SSN dataset. For instance, on the SSN dataset, it achieves errors of 7.85° (clean-indoor), 8.95° (clean-outdoor), 9.25° (noisy-indoor), and 11.98° (noisy-outdoor).
  • Significant improvement over 3F2N: 3F2N+ reduces the average angular error (eA) by up to 44.17% on the 3F2N datasets compared to the original 3F2N.
  • Robustness to noise: The SSN dataset results demonstrate that 3F2N+ maintains high accuracy even with noisy depth inputs.
  • Competitive inference speed: While achieving superior accuracy, 3F2N+ maintains a respectable inference speed of 44 FPS, making it suitable for real-time applications.
  • Enhanced downstream task performance: Integrating 3F2N+ into freespace detection, 6D object pose estimation, and point cloud completion tasks leads to noticeable improvements in performance metrics like IoU, F1-score, and accuracy.

Limitations and Future Work

  • Computational cost: While 3F2N+ is efficient, the paper notes that correlation coefficient-based methods are slightly slower than depth curvature-based ones. Future work could focus on further optimizing these correlation calculations.
  • Generalizability of DDM: The paper suggests that the DDM can serve as a universal solution for enhancing SNEs. Future research could explore its application to a wider range of SNE architectures and real-world sensor data.
  • Further exploration of CRF parameters: The paper mentions that the CRF's performance depends on the weights (σ and λ). Further fine-tuning and adaptive methods for these weights could potentially yield even better results.

Summary updated

Abstract

This article introduces three-filters-to-normal+ (3F2N+), an extension of our previous work three-filters-to-normal (3F2N), with a specific focus on incorporating discontinuity discrimination capability into surface normal estimators (SNEs). 3F2N+ achieves this capability by utilizing a novel discontinuity discrimination module (DDM), which combines depth curvature minimization and correlation coefficient maximization through conditional random fields (CRFs). To evaluate the robustness of SNEs on noisy data, we create a large-scale synthetic surface normal (SSN) dataset containing 20 scenarios (ten indoor scenarios and ten outdoor scenarios with and without random Gaussian noise added to depth images). Extensive experiments demonstrate that 3F2N+ achieves greater performance than all other geometry-based surface normal estimators, with average angular errors of 7.85$^\circ$, 8.95$^\circ$, 9.25$^\circ$, and 11.98$^\circ$ on the clean-indoor, clean-outdoor, noisy-indoor, and noisy-outdoor datasets, respectively. We conduct three additional experiments to demonstrate the effectiveness of incorporating our proposed 3F2N+ into downstream robot perception tasks, including freespace detection, 6D object pose estimation, and point cloud completion. Our source code and datasets are publicly available at https://mias.group/3F2Nplus.
BibTeX
@inproceedings{yang2023three,
  author = {Jingwei Yang and Bohuan Xue and Yi Feng and Deming Wang and Rui Fan and Qijun Chen},
  title = {{Three-Filters-to-Normal+: Revisiting Discontinuity Discrimination in Depth-to-Normal Translation}},
  booktitle = {IEEE Trans Autom. Sci. Eng},
  year = {2023},
}
Figure