BEV-MAE: Bird's Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios

Zhiwei Lin, Yongtao Wang, Shengxiang Qi, Nan Dong, Ming-Hsuan Yang

TL;DR

This paper introduces BEV-MAE, a novel self-supervised pre-training framework for LiDAR-based 3D object detection in autonomous driving. By employing a bird's eye view (BEV) guided masking strategy and a learnable point token, BEV-MAE efficiently learns robust feature representations from unlabeled point cloud data, leading to state-of-the-art performance on downstream detection tasks.

What Problem Does This Paper Address?

Autonomous driving systems heavily rely on accurate 3D object detection to perceive their surroundings. Current state-of-the-art LiDAR-based methods often train from scratch, which demands massive amounts of meticulously labeled data. Acquiring and annotating this data is a costly and time-consuming bottleneck. While self-supervised learning offers a promising alternative by leveraging abundant unlabeled data, existing methods often struggle with the unique characteristics of outdoor point clouds (e.g., sparsity at a distance) and may not align well with the BEV representations commonly used in autonomous driving detectors. This paper aims to bridge this gap by developing an efficient self-supervised pre-training method tailored for LiDAR-based 3D object detection in autonomous driving scenarios.

Methods

BEV-MAE introduces several key innovations:

  • BEV-Guided Masking Strategy: Instead of masking random points or voxels, BEV-MAE projects point clouds onto a Bird's Eye View (BEV) grid and masks entire non-empty BEV grids. This approach encourages the 3D encoder to learn features from a BEV perspective, directly aligning with the representation used in many downstream detectors and simplifying the decoder design.
  • Learnable Point Token: To address the potential reduction in the receptive field of sparse convolutional encoders when processing masked inputs, BEV-MAE replaces masked points with a shared, learnable point token. This token facilitates information flow between visible areas without introducing explicit positional information, maintaining a consistent receptive field size during pre-training and fine-tuning.
  • Point Density Prediction: Recognizing that outdoor point clouds become sparser with increasing distance from the LiDAR sensor, BEV-MAE incorporates point density prediction as an auxiliary reconstruction task. This helps the encoder learn crucial location information, which is vital for accurate object detection.
  • Lightweight Decoder: The BEV-guided masking strategy allows for a simple, one-layer 3x3 convolutional decoder, significantly reducing computational overhead and pre-training time compared to more complex decoder designs.

Datasets

The paper evaluates BEV-MAE on two prominent large-scale autonomous driving datasets:

  • nuScenes: Used for evaluating the performance of 3D object detection after pre-training.
  • Waymo Open Dataset: Also used for evaluating 3D object detection performance and for analyzing data efficiency.

The authors also mention using a combined dataset for pre-training and fine-tuning to assess cross-dataset transferability.

Key Insights and Ideas

  • BEV-centric pre-training is key: Directly learning features from a BEV perspective during pre-training significantly benefits downstream BEV-based object detectors, avoiding the need for complex feature transformations later.
  • Masking strategy matters: The proposed BEV-guided masking is more efficient and requires less GPU memory than traditional voxel masking, demonstrating the importance of tailoring the masking to the target domain's representation.
  • Density prediction is more than just counting: Predicting point density, rather than just the number of points, provides a more stable and effective signal for learning location information, crucial for object detection.
  • Simplicity wins: A lightweight decoder is sufficient for effective reconstruction and density prediction, highlighting that complex decoders are not always necessary for successful masked autoencoder pre-training.

Key Results

  • State-of-the-art performance: BEV-MAE achieves new state-of-the-art results on the nuScenes benchmark, reaching 73.6 NDS and 69.6 mAP when combined with the TransFusion-L detector.
  • Improved pre-training efficiency: BEV-MAE significantly outperforms other self-supervised methods on the Waymo dataset, achieving better performance with less pre-training time and data fraction. For instance, it surpasses ProposalContrast with 100% data by using only 20% of the data.
  • Data efficiency: BEV-MAE demonstrates strong data efficiency, providing substantial performance gains even with as little as 5% of labeled data for fine-tuning.
  • Reduced computational cost: The BEV-guided masking strategy leads to lower GPU memory consumption and faster pre-training compared to voxel masking.

Limitations and Future Work

  • Domain Gap in Transfer Learning: While BEV-MAE shows promise in transfer learning, the paper notes that the domain gap between datasets (e.g., Waymo vs. nuScenes) can negatively affect transfer performance. Combining datasets for pre-training shows potential for improvement.
  • Detector Capacity Bottleneck: On the Waymo dataset, when fine-tuning with the full dataset, BEV-MAE's improvements are marginal, suggesting that the detector's capacity might become the limiting factor rather than the pre-trained features.
  • Future Work: The authors suggest exploring more realistic pre-training settings, such as pre-training on combined datasets, to further enhance transferability and performance. Investigating how BEV-MAE can exploit existing large-scale datasets for better pre-training is also a promising direction.

Summary updated

Abstract

Existing LiDAR-based 3D object detection methods for autonomous driving scenarios mainly adopt the training-from-scratch paradigm. Unfortunately, this paradigm heavily relies on large-scale labeled data, whose collection can be expensive and time-consuming. Self-supervised pre-training is an effective and desirable way to alleviate this dependence on extensive annotated data. In this work, we present BEV-MAE, an efficient masked autoencoder pre-training framework for LiDAR-based 3D object detection in autonomous driving. Specifically, we propose a bird's eye view (BEV) guided masking strategy to guide the 3D encoder learning feature representation in a BEV perspective and avoid complex decoder design during pre-training. Furthermore, we introduce a learnable point token to maintain a consistent receptive field size of the 3D encoder with fine-tuning for masked point cloud inputs. Based on the property of outdoor point clouds in autonomous driving scenarios, i.e., the point clouds of distant objects are more sparse, we propose point density prediction to enable the 3D encoder to learn location information, which is essential for object detection. Experimental results show that BEV-MAE surpasses prior state-of-the-art self-supervised methods and achieves a favorably pre-training efficiency. Furthermore, based on TransFusion-L, BEV-MAE achieves new state-of-the-art LiDAR-based 3D object detection results, with 73.6 NDS and 69.6 mAP on the nuScenes benchmark. The source code will be released at https://github.com/VDIGPKU/BEV-MAE.
BibTeX
@inproceedings{lin2024bev,
  author = {Zhiwei Lin and Yongtao Wang and Shengxiang Qi and Nan Dong and Ming-Hsuan Yang},
  title = {{BEV-MAE: Bird's Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28141},
}
Figure