GeoMIM: Towards Better 3D Knowledge Transfer via Masked Image Modeling for Multi-view 3D Understanding

Jihao Liu, Tai Wang, Boxiao Liu, Qihang Zhang, Yu Liu, Hongsheng Li

TL;DR

GeoMIM introduces a pretrain-finetune paradigm that transfers geometry-rich knowledge from a pretrained LiDAR model to a multi-view camera network via Masked Image Modeling (MIM). By reconstructing LiDAR Bird's Eye View (BEV) features during pretraining instead of using direct distillation during finetuning, it successfully bridges the modality gap and achieves state-of-the-art multi-view 3D object detection and segmentation performance.

What Problem Does This Paper Address?

Multi-view camera-based 3D object detection is crucial for autonomous driving, but inferring 3D geometry from dense 2D images is an inherently ill-posed problem. To help camera networks learn spatial awareness, recent state-of-the-art methods distill knowledge from a pretrained LiDAR model (the "teacher") into the camera model (the "student") in the Bird's Eye View (BEV) space.

However, the authors identify a critical bottleneck: there is a massive domain gap between LiDAR and camera BEV features. LiDAR features are sparse, precise, and grounded in true 3D points, whereas camera features are dense and based on estimated depth. The authors show that while direct feature distillation helps weaker, ImageNet-pretrained models, it actually harms stronger camera models initialized with self-supervised pretraining. GeoMIM addresses this by shifting the teacher's guidance from the finetuning stage to a self-supervised pretraining stage, allowing the camera network to absorb geometric priors without being constrained by the LiDAR-camera domain gap during final task deployment.

Methods

GeoMIM adapts the Masked Autoencoder (MAE) framework for multi-view 3D understanding. The architecture consists of three main components:

  1. Masking and Encoder: Multi-view input images are split into patches, and 50% of them are randomly masked. A Swin Transformer encoder processes only the remaining visible tokens.
  2. Decoupled GeoMIM Decoder: Instead of a single decoder, GeoMIM splits the reconstruction task into two branches to separate semantic and geometric learning:
  3. Semantic Decoder: Reconstructs dense, perspective-view semantic features.
  4. Geometry Decoder: Predicts dense perspective-view depth maps. This branch is uniquely camera-aware, taking camera intrinsic and extrinsic parameters as inputs to scale features via a Squeeze-and-Excitation module.
  5. BEV Projection and Target Reconstruction: The reconstructed semantic features and depth maps are projected into the 3D BEV space using a Lift-Splat-Shoot (LSS) operation. The training target is to match the high-level BEV features generated by a frozen, pretrained LiDAR model.
Figure 1: Overview of GeoMIM. For pretraining, the multi-view images are randomly masked for a proportion of image tokens, and only the visible tokens are processed by the encoder. Right before decoding, the token embeddings are filled with mask tokens for separately decoding dense camera-view semantic features and depth maps, which are then projected to BEV space for reconstructing the LiDAR BEV features. After pretraining, only the encoder is finetuned on downstream tasks.

To handle multi-camera inputs efficiently, the authors introduce a Cross-View Attention (CVA) block within the decoder layers. Rather than computing global self-attention across all views (which is computationally prohibitive), CVA groups tokens by their row indices across adjacent, overlapping camera views. This allows the network to cross-reference overlapping visual fields with linear computational complexity.

Figure 1: Overview of GeoMIM. For pretraining, the multi-view images are randomly masked for a proportion of image tokens, and only the visible tokens are processed by the encoder. Right before decoding, the token embeddings are filled with mask tokens for separately decoding dense camera-view semantic features and depth maps, which are then projected to BEV space for reconstructing the LiDAR BEV features. After pretraining, only the encoder is finetuned on downstream tasks.

Datasets

The model was trained and evaluated using the following benchmarks:

  • nuScenes: The primary dataset used for pretraining, 3D object detection, and 3D semantic segmentation evaluation (consisting of 1,000 driving scenes with 6-camera setups and 360° LiDAR).
  • Open Waymo: Used to evaluate the cross-dataset transferability of the pretrained camera backbone.
  • nuImages: Used to test the backbone's transferability to downstream 2D object detection and instance segmentation tasks.

Key Insights and Ideas

  • Pretraining vs. Distillation: Directly forcing a strong camera network to mimic a LiDAR network during downstream finetuning introduces conflicting objectives due to modality differences. Moving this alignment to the pretraining phase acts as a powerful geometric initialization, which is discarded during finetuning to avoid the domain gap.
  • The Failure of Standard 2D MAE in 3D: Standard 2D MAE reconstructs raw pixels, which works well on massive, diverse datasets like ImageNet. However, autonomous driving datasets are less visually diverse, making pixel reconstruction prone to overfitting. Reconstructing high-level LiDAR BEV features forces the network to learn structural and spatial representations instead of low-level pixel textures.
  • Decoupled Decoders Maximize Capacity: Separating depth estimation from semantic feature completion forces the geometry branch to focus purely on spatial cues, significantly boosting localization accuracy (mATE).

Key Results

  • nuScenes 3D Detection: On the nuScenes test set, GeoMIM achieved state-of-the-art performance with 64.4% NDS and 56.1% mAP using a Swin-Large backbone, outperforming previous methods without requiring extra data or test-time augmentation.
  • nuScenes 3D Segmentation: Achieved 70.5% mIoU on the test set, a 1.1% improvement over the previous best camera-based method (TPVFormer).
  • Transferability: When transferred to the Open Waymo dataset, the GeoMIM-pretrained backbone improved the LET-3D-APL metric by 3.0% compared to a standard self-supervised baseline (MixMAE), proving that the camera-aware pretraining generalizes well to entirely different camera rigs.

Limitations and Future Work

The authors note two primary limitations:

  1. Data Requirements: GeoMIM still requires a substantial amount of multi-view data during the pretraining phase to learn high-quality representations.
  2. Teacher Dependence: The quality of the learned camera representations is fundamentally bounded by the quality and completeness of the pretrained LiDAR teacher's BEV features. If the LiDAR model misses objects or contains noise, those errors can propagate into the camera model's pretraining targets.

Summary updated

Abstract

Multi-view camera-based 3D detection is a challenging problem in computer vision. Recent works leverage a pretrained LiDAR detection model to transfer knowledge to a camera-based student network. However, we argue that there is a major domain gap between the LiDAR BEV features and the camera-based BEV features, as they have different characteristics and are derived from different sources. In this paper, we propose Geometry Enhanced Masked Image Modeling (GeoMIM) to transfer the knowledge of the LiDAR model in a pretrain-finetune paradigm for improving the multi-view camera-based 3D detection. GeoMIM is a multi-camera vision transformer with Cross-View Attention (CVA) blocks that uses LiDAR BEV features encoded by the pretrained BEV model as learning targets. During pretraining, GeoMIM's decoder has a semantic branch completing dense perspective-view features and the other geometry branch reconstructing dense perspective-view depth maps. The depth branch is designed to be camera-aware by inputting the camera's parameters for better transfer capability. Extensive results demonstrate that GeoMIM outperforms existing methods on nuScenes benchmark, achieving state-of-the-art performance for camera-based 3D object detection and 3D segmentation. Code and pretrained models are available at https://github.com/Sense-X/GeoMIM.
BibTeX
@inproceedings{liu2023geomim,
  author = {Jihao Liu and Tai Wang and Boxiao Liu and Qihang Zhang and Yu Liu and Hongsheng Li},
  title = {{GeoMIM: Towards Better 3D Knowledge Transfer via Masked Image Modeling for Multi-view 3D Understanding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01635},
}
Figure