Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review

Ying Li, Lingfei Ma, Zilong Zhong, Fei Liu, Dongpu Cao, Jonathan Li, Michael A. Chapman

Abstract: Recently, the advancement of deep learning in discriminative feature learning from 3D LiDAR data has led to rapid development in the field of autonomous driving. However, automated processing of uneven, unstructured, noisy, and massive 3D point clouds is a challenging and tedious task. In this paper, we provide a systematic review of existing compelling deep learning architectures applied in LiDAR point clouds, detailing specific tasks in autonomous driving such as segmentation, detection, and classification. Although several published research papers focus on specific topics in computer vision for autonomous vehicles, to date, no general survey on deep learning applied in LiDAR point clouds for autonomous vehicles exists. Thus, the goal of this paper is to narrow the gap in this topic. More than 140 key contributions in the recent five years are summarized in this survey, including the milestone 3D deep architectures, the remarkable deep learning applications in 3D semantic segmentation, object detection, and classification; specific datasets, evaluation metrics, and the state of the art performance. Finally, we conclude the remaining challenges and future researches.

Introduction

LiDAR (Light Detection and Ranging) has become an indispensable sensor for autonomous vehicles (AVs), providing precise 3D geo-referenced information that remains reliable across varying lighting conditions. However, the data produced—3D point clouds—presents significant computational challenges. Unlike 2D images, which are structured as regular grids of pixels, LiDAR point clouds are inherently unstructured, irregular, and sparse. Point density decreases significantly with distance, and a single frame can contain hundreds of thousands of points, necessitating highly efficient real-time processing.

Taxonomy of the review and the gap it addresses in existing literature.

This review systematizes five years of breakthroughs in 3D deep learning (DL), bridging the gap between general computer vision and the specialized requirements of autonomous driving. It categorizes over 140 key contributions into three primary tasks: segmentation, detection, and classification.

Data Representation and Methods

The core of the research involves how deep learning models "digest" irregular 3D data. The paper identifies four primary architectural approaches:

  1. Voxel-based Models: These convert points into regular 3D grids (voxels), enabling the use of 3D convolutions (e.g., VoxNet). However, this often leads to high memory usage and information loss during discretization.
  2. Point-based Models: Pioneered by PointNet, these models process raw coordinates directly using symmetric functions to ensure the model is invariant to the order of input points.
  3. Graph-based Models: These treat points as nodes in a graph, using edges to capture local geometric structures (e.g., DGCNN).
  4. View-based Models: These project 3D points onto 2D planes, such as Bird’s Eye View (BEV), allowing the application of optimized 2D CNNs (e.g., MV3D).

Core Tasks in Autonomous Driving

The review focuses on three critical applications that enable a vehicle to perceive its environment:

Visual representation of the three core tasks: (a) Point Cloud Segmentation, (b) 3D Object Detection, and (c) 3D Object Classification.
  • Segmentation: Assigning a semantic label to every point in a scene. SPGraph (Superpoint Graph) has shown remarkable performance here by grouping points into meaningful clusters before processing.
  • Detection: Identifying and localizing objects with 3D bounding boxes. Models like VoxelNet and VoteNet address the difficulty of finding object centers in sparse surface-only data.
  • Classification: Identifying the category of a specific point cluster. RotationNet achieved high accuracy on benchmarks like ModelNet by treating viewpoints as latent variables.

Key Insights and Future Frontiers

A major takeaway from the survey is the shift toward hybrid architectures that balance efficiency and accuracy. While voxel methods are precise, point-based methods are often more computationally efficient for large-scale scenes. Furthermore, the "reflectance" or intensity of LiDAR points is highlighted as a crucial feature for distinguishing materials like asphalt from lane markings.

Looking forward, the authors identify several remaining challenges:

  • Multi-source Fusion: Integrating LiDAR with Radar and cameras remains difficult due to temporal and spatial synchronization issues.
  • Weakly Supervised Learning: Labeling 3D data is prohibitively expensive; the field must move toward learning from unlabeled data.
  • Real-time Constraints: Many top-performing models still struggle to meet the millisecond-level latency required for high-speed driving.

By providing a roadmap of these developments, this paper serves as a foundational guide for researchers navigating the complex landscape of 3D perception for autonomous systems.

Summary updated

Abstract

Recently, the advancement of deep learning in discriminative feature learning from 3D LiDAR data has led to rapid development in the field of autonomous driving. However, automated processing uneven, unstructured, noisy, and massive 3D point clouds is a challenging and tedious task. In this paper, we provide a systematic review of existing compelling deep learning architectures applied in LiDAR point clouds, detailing for specific tasks in autonomous driving such as segmentation, detection, and classification. Although several published research papers focus on specific topics in computer vision for autonomous vehicles, to date, no general survey on deep learning applied in LiDAR point clouds for autonomous vehicles exists. Thus, the goal of this paper is to narrow the gap in this topic. More than 140 key contributions in the recent five years are summarized in this survey, including the milestone 3D deep architectures, the remarkable deep learning applications in 3D semantic segmentation, object detection, and classification; specific datasets, evaluation metrics, and the state of the art performance. Finally, we conclude the remaining challenges and future researches.
BibTeX
@article{li2020deep,
  author = {Ying Li and Lingfei Ma and Zilong Zhong and Fei Liu and Dongpu Cao and Jonathan Li and Michael A. Chapman},
  title = {{Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review}},
  journal = {IEEE Transactions on Neural Networks and Learning Systems},
  year = {2020},
  doi = {10.1109/tnnls.2020.3015992},
}
Figure