Li3DeTr: A LiDAR Based 3D Detection Transformer

Gopi Krishna Erabati, Helder AraujoWACV 2023

TL;DR

This paper introduces Li3DeTr, an end-to-end LiDAR-based 3D object detection transformer for autonomous driving. By leveraging a novel cross-attention mechanism in the decoder, Li3DeTr effectively links global LiDAR features to 3D predictions, achieving state-of-the-art performance on nuScenes and competitive results on KITTI without relying on Non-Maximum Suppression (NMS).

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurate 3D object detection to understand their surroundings. LiDAR sensors provide precise 3D information, but processing their sparse, unordered point clouds is challenging. Traditional methods often struggle to capture rich semantic information or require computationally expensive post-processing steps like NMS to refine detections. While vision transformers have shown promise in 2D object detection, adapting them effectively to 3D LiDAR data, especially for end-to-end detection without NMS, remains an open problem.

Methods

Li3DeTr is an end-to-end, single-stage transformer architecture. It comprises three main modules:

  • Backbone: This module extracts local voxel features from the LiDAR point cloud. The authors explore two pipelines: one using 3D sparse convolutions (like SECOND) and another using BEV pillars (like PointPillars). A Feature Pyramid Network (FPN) is used to obtain multi-scale local features.
  • Encoder: To capture global context and long-range dependencies, the local features are fed into a multi-scale deformable attention encoder. This encoder alternates between deformable attention blocks and MLPs, repeated Lenc times.
  • Decoder: This is where the core innovation lies. The decoder takes object queries and the global features from the encoder. It features a novel Li3DeTr cross-attention block that links the global LiDAR features to the 3D object predictions by leveraging learned object queries. Object queries interact with each other via multi-head self-attention. The decoder is repeated Ldec times, with each layer refining the object queries and ultimately predicting 3D bounding box parameters. Crucially, the entire process is formulated as a direct set prediction problem, eliminating the need for NMS, inspired by DETR.

Datasets

The paper evaluates Li3DeTr on two widely used autonomous driving datasets:

  • nuScenes: A large-scale, multi-modal dataset with 32-beam LiDAR point clouds.
  • KITTI: A benchmark dataset for autonomous driving, focusing on cars, pedestrians, and cyclists.

Key Insights and Ideas

  • Transformer for LiDAR: Demonstrates the effectiveness of transformer architectures for 3D LiDAR object detection, moving beyond traditional CNN-based approaches.
  • NMS-Free Detection: By formulating the problem as a direct set prediction task, Li3DeTr avoids the need for NMS, simplifying the pipeline and potentially improving performance.
  • Li3DeTr Cross-Attention: The novel cross-attention block is key to linking global LiDAR features with object predictions, enabling better detection of objects across various scales and distances.
  • Data Efficiency: The paper highlights that transformers are data-hungry, and Li3DeTr's performance on KITTI is competitive but not state-of-the-art compared to nuScenes, likely due to KITTI's smaller training set and sparser point clouds.

Key Results

  • nuScenes: Li3DeTr achieves 61.3% mAP and 67.6% NDS, surpassing state-of-the-art methods like CenterPoint and Object-DGCNN, even though those methods use NMS.
  • KITTI: The network achieves competitive performance on the KITTI dataset for car detection.
  • Knowledge Distillation: Using knowledge distillation with a teacher-student model slightly improves performance, achieving 62.2% mAP and 68.0% NDS on nuScenes.
  • Qualitative Results: Visualizations show Li3DeTr effectively detecting small objects (traffic cones) and large objects (trucks, buses), as well as some unannotated cars.

Limitations and Future Work

  • Information Loss: The paper acknowledges that downsampling feature maps in the backbone can lead to information loss, making it harder to detect smaller objects like pedestrians and barriers. Future work could focus on backbone designs that maintain original resolution while increasing receptive fields.
  • Data Hunger: Transformer models, including Li3DeTr, are data-hungry. The performance on KITTI, with its smaller dataset, is competitive but not leading, suggesting that larger datasets are beneficial for transformer-based approaches.
  • Backbone Flexibility: While the architecture is flexible regarding backbones, the performance varies. Further exploration of optimal backbone choices for different applications is suggested.

Summary updated

BibTeX
@inproceedings{erabati2023li,
  author = {Gopi Krishna Erabati and Helder Araujo},
  title = {{Li3DeTr: A LiDAR Based 3D Detection Transformer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure