To the Point: Efficient 3D Object Detection in the Range Image with Graph Convolution Kernels

Yuning Chai, Pei Sun, Jiquan Ngiam, Weiyue Wang, Benjamin Caine, Vijay Vasudevan, Xiao Zhang, Dragomir Anguelov

Abstract

3D object detection is vital for many robotics applications. For tasks where a 2D perspective range image exists, we propose to learn a 3D representation directly from this range image view. To this end, we designed a 2D convolutional network architecture that carries the 3D spherical coordinates of each pixel throughout the network. Its layers can consume any arbitrary convolution kernel in place of the default inner product kernel and exploit the underlying local geometry around each pixel. We outline four such kernels: a dense kernel according to the bag-of-words paradigm, and three graph kernels inspired by recent graph neural network advances: the Transformer, the PointNet, and the Edge Convolution. We also explore cross-modality fusion with the camera image, facilitated by operating in the perspective range image view. Our method performs competitively on the Waymo Open Dataset and improves the state-of-the-art AP for pedestrian detection from 69.7% to 75.5%. It is also efficient in that our smallest model, which still outperforms the popular PointPillars in quality, requires 180 times fewer FLOPS and model parameters.

The Challenge of 3D Object Detection

In autonomous driving and robotics, LiDAR sensors produce a dense 2D scan pattern known as a "range image." To detect 3D objects, most state-of-the-art methods project these 2D pixels into a 3D space and segment them into a 3D voxel grid. However, 3D voxelization has major drawbacks: computational complexity scales quadratically with range, and the limited resolution of voxels makes it difficult to detect thin objects like pedestrians.

Alternatively, applying standard 2D Convolutional Neural Networks (CNNs) directly to the 2D range image is fast and memory-efficient but historically suffers from a massive drop in accuracy. Standard 2D convolutions treat the input as a flat grid and cannot easily exploit the underlying 3D geometric structure of the scene. This paper bridges that gap, aiming to achieve the accuracy of 3D voxel methods with the efficiency of 2D range image methods.

Perspective Point Cloud (PPC) Architecture

The proposed Perspective Point Cloud (PPC) model processes the 2D range image using specialized "Point-Set Aggregation" layers. Instead of standard 2D convolutions, these layers carry the 3D spherical coordinates (azimuth, inclination, and depth) of each pixel throughout the network.

Comparison of 3D detection paradigms and the proposed Perspective Point-Set Aggregation Layer.

To leverage this 3D data, the authors replace the standard inner-product kernel of a 2D CNN with four alternative 3D-aware kernels:

  1. Range-Quantized (RQ) 2D Kernel: Applies different weights based on the relative depth difference between neighboring pixels.
  2. Self-Attention Kernel: A Transformer-inspired graph operator.
  3. PointNet Kernel: Uses a Multi-Layer Perceptron (MLP) to encode neighboring features and their relative 3D coordinates, pooling them via max-pooling.
  4. EdgeConv Kernel: Similar to PointNet, but explicitly includes the center pixel's feature alongside the relative positional encodings.

Additionally, the model employs a Smart Down-Sampling strategy. Because LiDAR images often have missing returns, standard grid down-sampling might sample empty pixels. This strategy selects the closest valid pixel to the pooling window's centroid, providing a 1.4% boost in accuracy.

Key Results and Efficiency

Evaluated on the Waymo Open Dataset, the PPC model achieves significant performance gains, particularly in pedestrian detection. The PPC + EdgeConv model achieved a new state-of-the-art (SOTA) for pedestrians, jumping from previous bests of ~69.7% to 75.5% AP. This boost is attributed to the perspective view capturing the full shape of tall, thin pedestrians without losing fidelity to large voxels. For vehicle detection, the model achieved 65.2% AP, outperforming most recent single-stage methods.

Complexity vs. accuracy comparison showing APH L2 performance against GFLOPS.

The model's primary advantage is its extreme efficiency. By using graph neural network concepts on a dense 2D grid, the model avoids the expensive nearest-neighbor searches required by standard 3D point cloud networks. The smallest version of the model (using a PointNet kernel) achieved higher accuracy than the widely used PointPillars baseline while requiring 180 times fewer FLOPS and containing only 24k parameters.

Insights and Limitations

Experimental results indicate that an asymmetric spherical polar coordinate system significantly outperforms standard world Cartesian coordinates (59.6% vs 54.2% APH) for encoding relative positions. Furthermore, EdgeConv and PointNet kernels outperformed the Transformer-inspired self-attention kernel, likely due to the small $3 \times 3$ kernel size used.

Despite its strengths, the model assumes the LiDAR sensor is mostly upright. Significant pitch or roll can distort bounding boxes in the 2D range image. Additionally, the reliance on a structured 2D grid makes applying standard 3D data augmentations, like random point dropping, difficult without destroying the network's input structure.

Summary updated

Abstract

3D object detection is vital for many robotics applications. For tasks where a 2D perspective range image exists, we propose to learn a 3D representation directly from this range image view. To this end, we designed a 2D convolutional network architecture that carries the 3D spherical coordinates of each pixel throughout the network. Its layers can consume any arbitrary convolution kernel in place of the default inner product kernel and exploit the underlying local geometry around each pixel. We outline four such kernels: a dense kernel according to the bag-of-words paradigm, and three graph kernels inspired by recent graph neural network advances: the Transformer, the PointNet, and the Edge Convolution. We also explore cross-modality fusion with the camera image, facilitated by operating in the perspective range image view. Our method performs competitively on the Waymo Open Dataset and improves the state-of-the-art AP for pedestrian detection from 69.7% to 75.5%. It is also efficient in that our smallest model, which still outperforms the popular PointPillars in quality, requires 180 times fewer FLOPS and model parameters
BibTeX
@inproceedings{chai2021point,
  author = {Yuning Chai and Pei Sun and Jiquan Ngiam and Weiyue Wang and Benjamin Caine and Vijay Vasudevan and Xiao Zhang and Dragomir Anguelov},
  title = {{To the Point: Efficient 3D Object Detection in the Range Image with Graph Convolution Kernels}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=gR7qOBL3vt1Q},
}
Figure