Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review

Ying Li, Lingfei Ma, Zilong Zhong, Fei Liu, Michael A. Chapman, Dongpu Cao, Jonathan Li

This review paper, published in 2020, provides a comprehensive roadmap of the rapidly evolving landscape of deep learning (DL) applied to LiDAR point clouds, specifically within the context of autonomous driving (AD).

TL;DR

The paper systematizes five years of breakthroughs in 3D deep learning, categorizing over 140 key contributions into segmentation, detection, and classification tasks. It serves as a foundational guide for understanding how researchers have moved from 2D image-based architectures to native 3D processing of irregular, sparse, and massive LiDAR datasets.

What Problem Does This Paper Address?

While LiDAR is indispensable for autonomous vehicles (AVs) due to its ability to provide accurate 3D geo-referenced information regardless of lighting conditions, the data it produces is "messy." Unlike the neat, regular grids of pixels in 2D images, LiDAR point clouds are:

  • Unstructured and Irregular: Points are scattered in 3D space with no inherent order.
  • Sparse and Uneven: Point density drops significantly with distance.
  • Massive: A single frame can contain hundreds of thousands of points, making real-time processing a computational nightmare.

The authors address the lack of a general survey that specifically bridges the gap between general 3D computer vision and the unique, real-time requirements of autonomous driving.

Methods

The paper categorizes DL frameworks based on how they represent and "digest" 3D data:

  1. Voxel-based Models: These convert points into regular 3D grids (voxels). While this allows the use of 3D convolutions (e.g., VoxNet), it often suffers from high memory consumption and information loss due to discretization.
  2. Point-based Models: These process raw point coordinates directly. PointNet was the pioneer here, using symmetric functions to achieve permutation invariance (meaning the model doesn't care about the order in which points are listed).
  3. Graph-based Models: These treat points as nodes in a graph and use edges to represent spatial relationships. This is excellent for capturing local geometric structures (e.g., DGCNN).
  4. View-based Models: These project 3D points onto 2D planes, such as Bird’s Eye View (BEV) or front-view images, allowing the use of highly optimized 2D CNNs (e.g., MV3D).

Datasets

The authors summarize the "battlegrounds" where these algorithms are tested:

  • Segmentation: Semantic3D (massive outdoor scenes) and Paris-Lille-3D.
  • Detection: KITTI is the gold standard for AV benchmarks, providing real-world driving data with 3D bounding boxes.
  • Classification: ModelNet (mostly CAD models) and Sydney Urban Objects (real-world LiDAR scans of vehicles, pedestrians, etc.).
  • Long-term Autonomy: The Oxford RobotCar dataset, which covers 1,000 km of driving over a year to test performance across different seasons and weather.

Key Insights and Ideas

  • The Permutation Challenge: Because point clouds are sets, a model must produce the same result regardless of the input order. This led to the widespread adoption of "max-pooling" as a global feature aggregator.
  • Surface vs. Volume: LiDAR only captures the surfaces of objects. This makes regressing the "center" of a 3D object difficult because there are no points there. Models like VoteNet solve this by having points "vote" for the likely center of an object.
  • Efficiency vs. Accuracy: Voxel-based methods are often more accurate but slower; point-based methods are efficient but can struggle with very large-scale scenes. The trend is moving toward hybrid architectures.
  • The Importance of Intensity: Beyond XYZ coordinates, the "reflectance" or intensity of a point is a crucial feature for distinguishing between materials (e.g., a lane marking vs. asphalt).

Key Results

The paper provides a snapshot of the state-of-the-art (circa 2020):

  • Segmentation: SPGraph (Superpoint Graph) led the pack on the Semantic3D benchmark with a 94.0% overall accuracy, demonstrating the power of grouping points into meaningful "superpoints" before processing.
  • Detection: On the KITTI benchmark, MV3D and VoxelNet showed that combining different views (BEV + Front View) or using voxel feature encoding significantly improved 3D car localization.
  • Classification: RotationNet achieved a staggering 97.37% accuracy on ModelNet, highlighting that treating viewpoints as latent variables is highly effective for 3D object recognition.

Limitations and Future Work

The authors identify several "frontiers" that remain unsolved:

  • Multi-source Data Fusion: Effectively combining LiDAR with Radar and Cameras in an end-to-end trainable way remains difficult due to different data formats and temporal synchronization.
  • Context Knowledge: Current models are good at geometry but poor at "semantics." For example, understanding the meaning of a specific traffic sign from sparse points is still a major challenge.
  • Weakly/Unsupervised Learning: Labeling 3D point clouds is incredibly expensive and slow. Developing models that can learn from unlabeled data is critical for scaling AV technology.
  • Real-time Constraints: Many high-performing models are still too slow for the millisecond-level decision-making required in high-speed autonomous driving.

Summary updated

Abstract

Recently, the advancement of deep learning (DL) in discriminative feature learning from 3-D LiDAR data has led to rapid development in the field of autonomous driving. However, automated processing uneven, unstructured, noisy, and massive 3-D point clouds are a challenging and tedious task. In this article, we provide a systematic review of existing compelling DL architectures applied in LiDAR point clouds, detailing for specific tasks in autonomous driving, such as segmentation, detection, and classification. Although several published research articles focus on specific topics in computer vision for autonomous vehicles, to date, no general survey on DL applied in LiDAR point clouds for autonomous vehicles exists. Thus, the goal of this article is to narrow the gap in this topic. More than 140 key contributions in the recent five years are summarized in this survey, including the milestone 3-D deep architectures, the remarkable DL applications in 3-D semantic segmentation, object detection, and classification; specific data sets, evaluation metrics, and the state-of-the-art performance. Finally, we conclude the remaining challenges and future researches.
BibTeX
@article{li2020deep,
  author = {Ying Li and Lingfei Ma and Zilong Zhong and Fei Liu and Michael A. Chapman and Dongpu Cao and Jonathan Li},
  title = {{Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review}},
  journal = {IEEE Transactions on Neural Networks and Learning Systems},
  year = {2020},
  doi = {10.1109/tnnls.2020.3015992},
}

Loading related content...

Figure