PointNet on FPGA for Real-Time LiDAR Point Cloud Processing

Lin Bai, Yecheng Lyu, Xin Xu, Xinming Huang

Abstract

LiDAR sensors have been widely used in many autonomous vehicle modalities, such as perception, mapping, and localization. This paper presents an FPGA-based deep learning platform for real-time point cloud processing targeted on autonomous vehicles. The software driver for the Velodyne LiDAR sensor is modified and moved into the on-chip processor system, while the programmable logic is designed as a customized hardware accelerator. As the state-of-art deep learning algorithm for point cloud processing, PointNet is successfully implemented on the proposed FPGA platform. Targeted on a Xilinx Zynq UltraScale+ MPSoC ZCU104 development board, the FPGA implementations of PointNet achieve the computing performance of 182.1 GOPS and 280.0 GOPS for classification and segmentation respectively. The proposed design can support an input up to 4096 points per frame. The processing time is 19.8 ms for classification and 34.6 ms for segmentation, which meets the real-time requirement for most of the existing LiDAR sensors.

Introduction

LiDAR sensors are critical for autonomous vehicles, providing high-fidelity 3D spatial data. However, processing this "point cloud" data in real-time presents a significant computational challenge. Traditional methods often project 3D data into 2D images or 3D grids to leverage standard hardware, but this leads to information loss. Point-based methods like PointNet process raw $(X, Y, Z, I)$ data directly, offering higher accuracy at the cost of increased computational intensity. While GPUs can handle these loads, their power consumption is often prohibitive for embedded automotive applications. This paper addresses the need for a low-power, high-throughput hardware accelerator capable of processing LiDAR data with minimal latency directly on the vehicle.

Methods

The proposed system is implemented on a Xilinx Zynq UltraScale+ MPSoC, utilizing a heterogeneous architecture that combines an ARM Processing System (PS) with FPGA Programmable Logic (PL).

PointNet Architecture for Classification and Segmentation

The ARM processor manages software-intensive tasks, including the LiDAR driver, coordinate transformations, and time-offset compensation. The FPGA logic houses a customized hardware accelerator specifically designed for PointNet's operations.

Hardware Optimizations

Since PointNet relies heavily on shared Multi-Layer Perceptrons (MLPs), which are mathematically equivalent to $1 \times 1$ convolutions or matrix multiplications, the architecture centers on a scalable SIMD (Single Instruction, Multiple Data) engine.

Matrix Multiplication Loop Structure for Hardware Acceleration

Key optimizations include:

  • Loop Unrolling: Nested loops are optimized to balance on-chip DSP slice utilization against communication bandwidth.
  • Quantization: Network parameters are converted from floating-point to 8-bit and 16-bit integers (INT8/INT16) to reduce memory footprint and power consumption.
  • Double Buffering: This technique enables concurrent data reading and processing, maximizing accelerator throughput.

Datasets and Evaluation

The study evaluates the hardware implementation using the original PointNet architecture as a benchmark. Real-world testing was conducted using a Velodyne VLP-16 LiDAR sensor. The system was tested with frames containing 4096 points, a standard density for autonomous driving regions of interest.

Key Insights

  • Edge Integration: By connecting the LiDAR directly to the SoC via Ethernet, the design eliminates the need for an intermediate PC, creating a standalone edge device.
  • Data Layout Flexibility: The accelerator supports both row-major and column-major output, accommodating the different data requirements of classification and segmentation tasks.
  • Architectural Efficiency: Recognizing PointNet as a series of shared MLPs allowed the team to simplify the hardware into a highly optimized matrix multiplication engine.

Results

Implementation on the ZCU104 board demonstrated significant performance, particularly with INT8 quantization:

  • Throughput: Reached 182.1 GOPS for classification and 280.0 GOPS for segmentation.
  • Latency: Classification took 19.8 ms (~50.5 FPS), while segmentation took 34.6 ms (~28.9 FPS).
  • Real-Time Performance: Standard LiDAR sensors typically operate at 10Hz (100ms per frame). This accelerator processes frames in roughly one-third of that time, ensuring ample overhead for other critical vehicle systems.

Limitations and Future Work

The current design is optimized for 4096 points; higher-resolution sensors would require sub-sampling or partitioning. While the scalable architecture can theoretically support more complex models like PointNet++, this implementation focuses on the foundational PointNet. Future research could also explore the specific impact of INT8/INT16 quantization on classification accuracy compared to full-precision models.

Summary updated

Abstract

LiDAR sensors have been widely used in many autonomous vehicle modalities, such as perception, mapping, and localization. This paper presents an FPGA-based deep learning platform for real-time point cloud processing targeted on autonomous vehicles. The software driver for the Velodyne LiDAR sensor is modified and moved into the on-chip processor system, while the programmable logic is designed as a customized hardware accelerator. As the state-of-art deep learning algorithm for point cloud processing, PointNet is successfully implemented on the proposed FPGA platform. Targeted on a Xilinx Zynq UltraScale+ MPSoC ZCU104 development board, the FPGA implementations of PointNet achieve the computing performance of 182.1 GOPS and 280.0 GOPS for classification and segmentation respectively. The proposed design can support an input up to 4096 points per frame. The processing time is 19.8 ms for classification and 34.6 ms for segmentation, which meets the real-time requirement for most of the existing LiDAR sensors.
BibTeX
@misc{bai2020pointnet,
  author = {Lin Bai and Yecheng Lyu and Xin Xu and Xinming Huang},
  title = {{PointNet on FPGA for Real-Time LiDAR Point Cloud Processing}},
  year = {2020},
  eprint = {2006.00049},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.00049},
}
Figure