Abstract
3D object detection is vital for many robotics applications. For tasks where a 2D perspective range image exists, we propose to learn a 3D representation directly from this range image view. To this end, we designed a 2D convolutional network architecture that carries the 3D spherical coordinates of each pixel throughout the network. Its layers can consume any arbitrary convolution kernel in place of the default inner product kernel and exploit the underlying local geometry around each pixel. We outline four such kernels: a dense kernel according to the bag-of-words paradigm, and three graph kernels inspired by recent graph neural network advances: the Transformer, the PointNet, and the Edge Convolution. Our method performs competitively on the Waymo Open Dataset and improves the state-of-the-art AP for pedestrian detection from 69.7% to 75.5%. It is also efficient in that our smallest model, which still outperforms the popular PointPillars in quality, requires 180 times fewer FLOPS and model parameters.
Introduction
LiDAR sensors are the primary perception modality for autonomous vehicles, but processing their data is computationally intensive. Traditional methods typically rely on voxelization—converting 3D point clouds into a grid of cubes—or graph-based representations. Voxelization is accurate but scales poorly as sensor range increases due to the quadratic growth of empty space. Graph-based methods offer better scaling but require expensive nearest-neighbor searches to construct the graph.
An alternative is the Range Image. LiDARs naturally produce a 2D image where each pixel records distance (range) and intensity. While processing 2D images is highly efficient, standard 2D convolutions are "3D-blind," treating pixels as neighbors regardless of their actual distance in 3D space. This research bridges that gap, maintaining the efficiency of 2D processing while regaining the geometric precision of 3D methods.
Methods
We introduce the Perspective Point Cloud (PPC) representation, which keeps data in a 2D grid while carrying the 3D spherical coordinates of every pixel throughout the entire neural network.
To enable 3D awareness, standard 2D convolution kernels are replaced with four novel alternatives within a Perspective Point-Set Aggregation Layer:
- Range-quantized (RQ) Kernel: Applies weights based on the 3D distance between neighbors.
- Self-Attention Kernel: A Transformer-inspired approach using asymmetric positional encoding for 3D geometric relationships.
- PointNet & EdgeConv Kernels: Adapt popular 3D graph operations into the 2D grid using MLPs to learn local geometric features.
Furthermore, Smart Down-sampling is used to prevent data loss in sparse scans. Instead of standard pooling, the network selects the valid pixel closest to the local 3D centroid, ensuring small objects like pedestrians remain detectable.
Results
The model was evaluated on the Waymo Open Dataset (WOD), which consists of 1,000 sequences of high-resolution LiDAR and camera data.
Key performance metrics include:
- State-of-the-Art Accuracy: The model improved the APH (Average Precision with Heading) for pedestrians from 69.7% to 75.5%.
- Extreme Efficiency: The smallest model variant outperformed the industry-standard PointPillars while requiring 180 times fewer FLOPS and parameters.
- Compact Footprint: One high-performance version uses only 24k parameters.
- Vehicle Detection: The model matched most SOTA vehicle detectors, though it trailed slightly behind complex two-stage models.
Discussion and Limitations
By operating in the range image view, the model avoids wasting computation on empty space. This perspective view also facilitates seamless multi-sensor fusion, as projecting camera pixels onto LiDAR pixels is mathematically straightforward. The method is particularly effective for pedestrians, as the range image preserves their full vertical silhouette better than voxel grids.
However, some limitations remain. The model assumes a relatively level sensor; significant vehicle pitch or roll could disrupt the 2D range image geometry. Additionally, standard 3D data augmentations like rotation are difficult to implement in a fixed 2D grid. While ideal for autonomous driving, these factors might challenge applications in indoor robotics with more varied sensor orientations.
Summary updated