3D Point Cloud Recognition Based on a Multi-View Convolutional Neural Network

Le Zhang, Jian Sun, Qiang ZhengSensors 2018 · DOI

TL;DR

This paper tackles the challenge of recognizing 3D objects from LiDAR point clouds by proposing a novel multi-view 2.5D representation. They introduce a multi-view convolutional neural network that effectively fuses features from these 2.5D views to achieve robust 3D object recognition without requiring full 3D reconstruction.

What Problem Does This Paper Address?

Recognizing 3D objects from LiDAR point clouds is a critical task in fields like autonomous driving and robotics. However, traditional methods often struggle with the inherent characteristics of point cloud data: it's irregular (non-grid), unordered, and often lacks rich texture information. LiDAR sensors, in particular, capture data as a series of 2.5D scans from different viewpoints, rather than a complete 3D representation. This paper addresses the problem of effectively utilizing these multi-view 2.5D LiDAR scans for accurate 3D object recognition, aiming to overcome the limitations of processing raw 3D point clouds directly and avoiding computationally expensive 3D reconstruction.

Methods

The core of this work lies in their proposed multi-view convolutional neural network (CNN) architecture. The key components are:

  • Multi-View 2.5D Representation: Instead of using raw 3D point clouds, the authors generate multiple 2.5D point clouds from different viewpoints of a 3D model. These 2.5D views are then colored based on their spatial coordinates (x, y, z) to provide some visual cues.
  • Feature Extraction Structure: For each 2.5D view, a dedicated feature extraction branch is employed. This branch includes:
  • Transformation Correction Network: This network learns to correct for affine transformations (translation, rotation, scaling) in the input point clouds or their features, making the recognition invariant to these geometric changes.
  • Convolutional Layers: Standard convolutional layers are used to extract local features from the 2.5D point cloud data.
  • Symmetry Module: This module, composed of convolution and max-pooling layers, ensures that the extracted features are invariant to the order of points within a view.
  • View Fusion Network: This novel component is crucial. It takes the features extracted from all individual 2.5D views and fuses them into a single, global feature descriptor. This allows the model to leverage information from multiple perspectives.
  • Classifier: Finally, fully connected layers are used to classify the object based on the fused global feature descriptor.

The entire model is designed to process raw 2.5D point clouds directly, avoiding the need for explicit 3D reconstruction or complex pre-processing.

Datasets

The authors evaluate their method on two publicly available datasets:

  • ModelNet10: Contains 10 categories of 3D objects.
  • ModelNet40: Contains 40 categories of 3D objects.

They generate multi-view 2.5D point cloud data from the CAD models in these datasets for their experiments.

Key Insights and Ideas

  • 2.5D is Sufficient: The paper demonstrates that by effectively combining information from multiple 2.5D views, it's possible to achieve high recognition accuracy without needing a full 3D representation. This is particularly relevant for LiDAR data, which is naturally captured in this format.
  • View Fusion is Key: The proposed view fusion network significantly outperforms simpler methods like view-pooling, highlighting the importance of a more sophisticated approach to integrating information across different viewpoints.
  • Robustness to Transformations: The inclusion of transformation correction networks makes the model inherently robust to common geometric variations like translation, rotation, and scaling, which is a significant advantage for real-world applications.
  • Low-Resolution Performance: The model shows good performance even with a reduced number of points, suggesting its applicability to scenarios where data density might be limited.

Key Results

  • The proposed multi-view CNN achieved state-of-the-art or competitive accuracy on both ModelNet10 and ModelNet40 datasets.
  • On ModelNet10, their method with 12 views reached 93.5% accuracy, and with 20 views, it achieved 94.1%. On ModelNet40, they reported 90.5% and 90.8% accuracy for 12 and 20 views, respectively.
  • The view fusion network was shown to be critical, with its removal leading to a significant drop in accuracy (e.g., 3.0% on ModelNet10 when replaced by view-pooling).
  • The model demonstrated good performance across varying numbers of points, with accuracy increasing slightly as the number of points increased from 64 to 512.
  • Increasing the number of views from 1 to 6 and then to 12 also consistently improved accuracy, indicating the value of multi-view information.

Limitations and Future Work

  • Symmetry Module Improvement: The authors acknowledge that while the symmetry module helps with point disorder, they aim to find new methods to avoid potential information loss caused by it.
  • View Selection: Future work could focus on identifying the most informative views to reduce processing time and achieve effective real-time 3D recognition and tracking.
  • Computational Cost: While the method avoids 3D reconstruction, using a large number of views and points can increase computational cost and training time. The authors suggest balancing accuracy needs with efficiency requirements.

Summary updated

Abstract

The recognition of three-dimensional (3D) lidar (light detection and ranging) point clouds remains a significant issue in point cloud processing. Traditional point cloud recognition employs the 3D point clouds from the whole object. Nevertheless, the lidar data is a collection of two-and-a-half-dimensional (2.5D) point clouds (each 2.5D point cloud comes from a single view) obtained by scanning the object within a certain field angle by lidar. To deal with this problem, we initially propose a novel representation which expresses 3D point clouds using 2.5D point clouds from multiple views and then we generate multi-view 2.5D point cloud data based on the Point Cloud Library (PCL). Subsequently, we design an effective recognition model based on a multi-view convolutional neural network. The model directly acts on the raw 2.5D point clouds from all views and learns to get a global feature descriptor by fusing the features from all views by the view fusion network. It has been proved that our approach can achieve an excellent recognition performance without any requirement for three-dimensional reconstruction and the preprocessing of point clouds. In conclusion, this paper can effectively solve the recognition problem of lidar point clouds and provide vital practical value.
BibTeX
@inproceedings{zhang2018d,
  author = {Le Zhang and Jian Sun and Qiang Zheng},
  title = {{3D Point Cloud Recognition Based on a Multi-View Convolutional Neural Network}},
  booktitle = {Sensors},
  year = {2018},
  doi = {10.3390/s18113681},
}
Figure