3D Point Cloud Enhancement Using Graph-Modelled Multiview Depth Measurements

Xue Zhang, Gene Cheung, Jiahao Pang, Dong Tian

Abstract

This paper introduces a novel "a priori" framework for 3D point cloud enhancement. By denoising and dequantizing raw depth measurements from multiple cameras before they are projected into 3D space, the proposed method leverages geometric correlations and graph-based signal processing. Experimental results demonstrate significant improvements in point cloud quality and error reduction compared to traditional post-processing techniques that operate on synthesized 3D models.

Introduction

3D point clouds are essential for various applications, from autonomous driving to virtual reality. Typically, these clouds are generated by combining depth images from multiple sensors, such as the Microsoft Kinect or Intel RealSense. However, these sensors are prone to electronic noise and quantization errors—where continuous depth values are rounded to discrete bits—resulting in "blocky" or "staircase" artifacts in the final 3D model.

Most existing solutions address these issues a posteriori, attempting to smooth the 3D surface after the point cloud has already been synthesized. This paper argues that such an approach is sub-optimal. Much like processing a raw image file yields better results than editing a compressed JPEG, enhancing depth maps at the source allows for more effective noise removal before the complex 3D synthesis pipeline obscures the original noise patterns.

Methodology

The authors formulate the enhancement as a Maximum A Posteriori (MAP) optimization problem, focusing on three core pillars:

1. Image Formation and Multiview Geometry

The process begins by explicitly modeling the depth sensing process, accounting for both additive Gaussian noise and bit-depth quantization. To exploit the redundancy of multiple cameras viewing the same scene, the method uses a non-linear warping function based on 3D geometry.

Multiview depth sensing geometry and corresponding depth maps.

To make this mathematically tractable, a first-order Taylor series expansion is employed to create a linear approximation of the relationship between pixels in different camera views. This allows the algorithm to optimize both depth maps simultaneously, ensuring geometric consistency.

2. Feature Graph Learning

The depth map is treated as a signal on a graph. The structure of this graph (the Laplacian) is learned from previously processed rows of pixels using metric learning. This enables the algorithm to understand local geometry, such as surface normals, allowing it to preserve sharp edges while smoothing out noise on flat surfaces. The resulting objective function is convex and differentiable, solved efficiently using the Fast Gradient Method (FGM).

Experimental Evaluation

The researchers evaluated their method using five high-quality depth image pairs from the Middlebury datasets (Adirondack, Recycle, Playtable, Teddy, and ArtL). These datasets provide ground truth data, allowing for the simulation of various sensor noise levels ($\sigma^2_n = 50, 70, 90$) and quantization effects to test robustness.

Results and Discussion

The proposed method was compared against three state-of-the-art 3D denoising algorithms: APSS, RIMLS, and MRPCA.

Quantitative Performance

The method consistently outperformed competitors across all metrics. On average, it reduced Point-to-Point (C2C) error by 0.68 to 1.13 units and Point-to-Plane (C2P) error by 2.68 to 5.93 units compared to the second-best algorithm.

Qualitative Visual Quality

Visualizations using CloudCompare reveal that the proposed method results in significantly "bluer" error maps, indicating smaller absolute distances from the ground truth.

Comparison of Point-to-Point (C2C) absolute distances across different denoising methods.

As seen in the comparison, the proposed method effectively removes the staircase artifacts caused by quantization, producing a much smoother and more accurate surface representation than traditional methods like MRPCA or RIMLS.

Conclusion

The "a priori" approach to point cloud enhancement proves superior because it preserves the statistical properties of the noise during the denoising process. By combining joint optimization of multiview depth maps with Graph-Signal Processing (GSP), the method achieves high-fidelity results that respect the underlying geometry of the scene.

Future work will focus on addressing current limitations, such as handling occlusions where cameras do not share a complete field of view, and optimizing the framework for real-time video depth enhancement.

Summary updated

BibTeX
@inproceedings{zhang2020d,
  author = {Xue Zhang and Gene Cheung and Jiahao Pang and Dong Tian},
  title = {{3D Point Cloud Enhancement Using Graph-Modelled Multiview Depth Measurements}},
  booktitle = {ICIP},
  year = {2020},
  url = {https://openreview.net/forum?id=Qimf6qtSFO},
}
Figure