Multi-Sample Consensus Driven Unsupervised Normal Estimation for 3D Point Clouds

Jie Zhang, Minghui Nie, Junjie Cao, Jian Liu, Ligang Liu

Abstract

Deep normal estimators have made great strides on synthetic benchmarks. Unfortunately, their performance dramatically drops on real scan data since they are typically supervised only on synthetic datasets. The point-wise annotation of ground truth normals is vulnerable to inefficiency and inaccuracies, making it nearly impossible to build perfect real datasets for supervised deep learning. To overcome this challenge, we propose a multi-sample consensus paradigm for unsupervised normal estimation. This paradigm consists of multi-candidate sampling, candidate rejection, and mode determination. We introduce PointNorm-Net (MSUNE-Net), the first unsupervised deep normal estimator. It transfers online consensus stages to offline training, resulting in inference times 100 times faster than optimization-based methods while achieving superior accuracy. Comprehensive experiments demonstrate that our unsupervised methods outperform state-of-the-art supervised estimators on both synthetic and real-world datasets like NYUV2 and KITTI.

Introduction

Surface normals—vectors perpendicular to a surface at a given point—are fundamental for 3D tasks such as robot grasping, autonomous driving, and scene reconstruction. While deep learning has revolutionized this field, most existing models are supervised, requiring massive amounts of labeled data. This approach faces two major hurdles: the annotation gap, where manual labeling of millions of points is impractical, and the domain gap, where real-world sensor noise (LiDAR, Kinect) differs significantly from synthetic training noise. PointNorm-Net addresses these by learning directly from raw, unlabeled point clouds, ensuring robustness to specific sensor characteristics.

Methodology

The core of PointNorm-Net is a three-stage multi-modal distribution estimation paradigm. It leverages a mathematical property established by the authors: on a smooth surface with zero-mean noise, the ground-truth normal is the expectation of randomly sampled candidate normals.

The three-stage pipeline of PointNorm-Net: Candidate Normal Initialization, Feasible Candidate Filtering, and Major Mode Estimation.

The pipeline consists of:

  1. Candidate Normal Initialization (CNI): Randomly sampling neighboring point groups to fit local planes and generate candidate normals. The model uses adaptive neighborhoods to adjust to varying noise levels.
  2. Feasible Candidate Filtering (FCF): Using kernel density estimation to filter out "blurry" normals that lack strong consensus among neighboring points.
  3. Major Mode Estimation: Employing a novel Candidate Consensus Loss ($L_{ccn}$) to train a neural network to identify the most representative normal in multi-modal distributions (e.g., near sharp edges). A secondary feature-preserving loss ($L_w$) ensures that points on different sides of an edge do not interfere with each other, preventing over-smoothing.

Experimental Results

PointNorm-Net was evaluated across a diverse range of data sources, including real-world Kinect (PCV, NYUV2), LiDAR (KITTI), and TLS (Semantic3D) datasets.

Comparison of RMSE and inference time across various normal estimation methods. PointNorm-Net achieves the lowest RMSE with high efficiency.

Key findings include:

  • Superior Generalization: On the PCV Kinect dataset, PointNorm-Net achieved an RMS error of 10.89, significantly lower than the supervised DeepFit (13.68) and traditional optimization methods like HF-cubes (11.72).
  • Efficiency: The model requires only a single forward pass (0.36ms per point), matching the speed of supervised models while being significantly faster than traditional optimization-based estimators.
  • Robustness: High accuracy was maintained across different sensor types without per-sensor retraining, demonstrating its ability to bridge the domain gap effectively.

Conclusion

PointNorm-Net represents a significant step forward as the first self-supervised deep learning framework for normal estimation. By framing the problem as finding the major mode of a multi-modal distribution, it preserves sharp geometric features while remaining robust to real-world noise. While challenges remain regarding extremely thin structures and non-uniform point densities, the multi-sample consensus framework offers a promising path for unsupervised 3D scene understanding.

Summary updated

Abstract

Deep normal estimators have made great strides on synthetic benchmarks. Unfortunately, their performance dramatically drops on the real scan data since they are supervised only on synthetic datasets. The point-wise annotation of ground truth normals is vulnerable to inefficiency and inaccuracies, which totally makes it impossible to build perfect real datasets for supervised deep learning. To overcome the challenge, we propose a multi-sample consensus paradigm for unsupervised normal estimation. The paradigm consists of multi-candidate sampling, candidate rejection, and mode determination. The latter two are driven by neighbor point consensus and candidate consensus respectively. Two primary implementations of the paradigm, MSUNE and MSUNE-Net, are proposed. MSUNE minimizes a candidate consensus loss in mode determination. As a robust optimization method, it outperforms the cutting-edge supervised deep learning methods on real data at the cost of longer runtime for sampling enough candidate normals for each query point. MSUNE-Net, the first unsupervised deep normal estimator as far as we know, significantly promotes the multi-sample consensus further. It transfers the three online stages of MSUNE to offline training. Thereby its inference time is 100 times faster. Besides that, more accurate inference is achieved, since the candidates of query points from similar patches can form a sufficiently large candidate set implicitly in MSUNE-Net. Comprehensive experiments demonstrate that the two proposed unsupervised methods are noticeably superior to some supervised deep normal estimators on the most common synthetic dataset. More importantly, they show better generalization ability and outperform all the SOTA conventional and deep methods on three real datasets: NYUV2, KITTI, and a dataset from PCV [1].
BibTeX
@article{zhang2023multi,
  author = {Jie Zhang and Minghui Nie and Junjie Cao and Jian Liu and Ligang Liu},
  title = {{Multi-Sample Consensus Driven Unsupervised Normal Estimation for 3D Point Clouds}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.48550/arxiv.2304.04884},
}
Figure