Data-Driven Point Cloud Objects Completion

Yang Zhang, Zhen Liu, Xiang Li, Yu Zang

TL;DR

This paper introduces PCCNet, a novel deep learning framework that uses 2D images to complete incomplete 3D point clouds, a common problem in laser scanning. By fusing image and point cloud features with an attention mechanism and adding a projection loss for multi-view consistency, PCCNet significantly improves object completion, especially for objects with large missing parts.

What Problem Does This Paper Address?

Laser scanning technologies, like LiDAR, are increasingly used to capture 3D environments for applications ranging from autonomous driving to virtual reality. However, the resulting 3D point clouds often suffer from incompleteness due to scanning angles and occlusions. This is particularly problematic for objects that are partially hidden or have complex geometries. Traditional point cloud completion methods struggle with large missing regions, often failing to reconstruct the object's overall structure. This paper aims to address this challenge by developing a data-driven approach that leverages readily available 2D images to infer and complete missing 3D information.

Methods

The core of the proposed solution is the Point Cloud Completion Network (PCCNet), an image-guided deep learning framework. PCCNet takes an incomplete 3D point cloud and its corresponding 2D image as input. The network comprises:

  • 2D Encoder: Extracts features from the input image.
  • 3D Encoder: Extracts features from the incomplete point cloud, using a PointNet++ architecture to capture hierarchical geometric information.
  • Attention-based 2D-3D Fusion Module: This novel module adaptively fuses the 2D and 3D features. It learns weights to balance the contribution of each modality based on their information integrity, allowing the network to "understand" the object from both perspectives.
  • Decoder: An encoder-decoder structure that reconstructs the complete 3D point cloud from the fused features.
  • Projection Loss: An additional loss function that enforces multi-view consistency. It projects both the generated and ground truth point clouds onto 2D images and penalizes discrepancies in these projections, ensuring a more coherent spatial distribution.
  • Merging and Alignment: The generated complete point cloud is aligned with the original incomplete point cloud using the Iterative Closest Point (ICP) algorithm, and then merged to form the final output.

Datasets

  • ShapeNetCore55: Used for training the network. The authors synthesized training data by sampling points from CAD models and rendering images from specific viewpoints to mimic real-world scenarios.
  • Mobile Laser Scanning (MLS) data: Used for testing. This includes real street images and corresponding point clouds acquired by a RIEGL VMX-450 MLS system. The authors specifically mention using data where objects are significantly incomplete.

Key Insights and Ideas

  • Leveraging 2D for 3D Completion: The paper demonstrates that 2D images can provide crucial contextual information to infer missing 3D structures, especially when traditional 3D-only methods fail.
  • Adaptive Feature Fusion: The attention-based fusion module is key. It allows the network to dynamically weigh the importance of image versus point cloud features, which is critical when one modality might be more informative than the other (e.g., an image might show a complete object silhouette, while the point cloud is heavily occluded).
  • Multi-view Consistency is Crucial: The projection loss is not just an add-on; it significantly improves the spatial coherence and accuracy of the generated point clouds by ensuring consistency across different viewpoints.
  • Data-Driven Approach for Extreme Cases: Unlike rule-based or geometric methods that struggle with large missing parts, a deep learning approach trained on diverse data can learn to generalize and complete even severely incomplete objects.

Key Results

  • Improved Reconstruction: PCCNet demonstrates superior 3D reconstruction abilities compared to existing generative networks like OGN and PSGN, particularly on images with complex backgrounds.
  • Effective Completion of Large Missing Parts: The method successfully completes objects with significant occlusions, outperforming traditional point cloud completion methods (e.g., method [8]) which fail in such scenarios.
  • Impact of Projection Loss: Adding the projection loss leads to faster training and higher accuracy, as evidenced by faster CD loss decrease and improved metrics.
  • Quantitative Improvements: Tables 1 and 2 show that PCCNet (PCCNet_P) achieves better CD scores and IoU scores compared to its variants (PCCNet_WP, PCCNet_WF) and other state-of-the-art methods. For example, for cars, PCCNet_P achieves a CD score of 0.00123, outperforming PSGN (0.00128) and PCCNet_WF (0.00127).

Limitations and Future Work

  • Limited Categories: The authors acknowledge that their training and testing were primarily focused on car objects due to the limitations of the ShapeNet dataset categories. They express confidence that the method is suitable for other categories as well.
  • Future Work: The paper suggests that the unified deep-learning architecture for combining 2D and 3D features is a promising area for future research in 3D modeling and processing. The proposed network offers an efficient way to integrate these modalities for point cloud completion.

Summary updated

Abstract

With the development of the laser scanning technique, it is easier to obtain 3D large-scale scene rapidly. However, many scanned objects may suffer serious incompletion caused by the scanning angles or occlusion, which has severely impacted their future usage for the 3D perception and modeling, while traditional point cloud completion methods often fails to provide satisfactory results due to the large missing parts. In this paper, by utilising 2D single-view images to infer 3D structures, we propose a data-driven Point Cloud Completion Network ( P C C N e t ), which is an image-guided deep-learning-based object completion framework. With the input of incomplete point clouds and the corresponding scanned image, the network can acquire enough completion rules through an encoder-decoder architecture. Based on an attention-based 2D-3D fusion module, the network is able to integrate 2D and 3D features adaptively according to their information integrity. We also propose a projection loss as an additional supervisor to have a consistent spatial distribution from multi-view observations. To demonstrate the effectiveness, first, the proposed P C C N e t is compared to recent generative networks and has shown more powerful 3D reconstruction abilities. Then, P C C N e t is compared to a recent point cloud completion methods, which has demonstrate that the proposed P C C N e t is able to provide satisfied completion results for objects with large missing parts.
BibTeX
@inproceedings{zhang2019data,
  author = {Yang Zhang and Zhen Liu and Xiang Li and Yu Zang},
  title = {{Data-Driven Point Cloud Objects Completion}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=BRB3j1lB48},
}
Figure