TL;DR
GeoMIM introduces a pretrain-finetune paradigm that transfers geometry-rich knowledge from a pretrained LiDAR model to a multi-view camera network via Masked Image Modeling (MIM). By reconstructing LiDAR Bird's Eye View (BEV) features during pretraining instead of using direct distillation during finetuning, it successfully bridges the modality gap and achieves state-of-the-art multi-view 3D object detection and segmentation performance.
What Problem Does This Paper Address?
Multi-view camera-based 3D object detection is crucial for autonomous driving, but inferring 3D geometry from dense 2D images is an inherently ill-posed problem. To help camera networks learn spatial awareness, recent state-of-the-art methods distill knowledge from a pretrained LiDAR model (the "teacher") into the camera model (the "student") in the Bird's Eye View (BEV) space.
However, the authors identify a critical bottleneck: there is a massive domain gap between LiDAR and camera BEV features. LiDAR features are sparse, precise, and grounded in true 3D points, whereas camera features are dense and based on estimated depth. The authors show that while direct feature distillation helps weaker, ImageNet-pretrained models, it actually harms stronger camera models initialized with self-supervised pretraining. GeoMIM addresses this by shifting the teacher's guidance from the finetuning stage to a self-supervised pretraining stage, allowing the camera network to absorb geometric priors without being constrained by the LiDAR-camera domain gap during final task deployment.
Methods
GeoMIM adapts the Masked Autoencoder (MAE) framework for multi-view 3D understanding. The architecture consists of three main components:
- Masking and Encoder: Multi-view input images are split into patches, and 50% of them are randomly masked. A Swin Transformer encoder processes only the remaining visible tokens.
- Decoupled GeoMIM Decoder: Instead of a single decoder, GeoMIM splits the reconstruction task into two branches to separate semantic and geometric learning:
- Semantic Decoder: Reconstructs dense, perspective-view semantic features.
- Geometry Decoder: Predicts dense perspective-view depth maps. This branch is uniquely camera-aware, taking camera intrinsic and extrinsic parameters as inputs to scale features via a Squeeze-and-Excitation module.
- BEV Projection and Target Reconstruction: The reconstructed semantic features and depth maps are projected into the 3D BEV space using a Lift-Splat-Shoot (LSS) operation. The training target is to match the high-level BEV features generated by a frozen, pretrained LiDAR model.
To handle multi-camera inputs efficiently, the authors introduce a Cross-View Attention (CVA) block within the decoder layers. Rather than computing global self-attention across all views (which is computationally prohibitive), CVA groups tokens by their row indices across adjacent, overlapping camera views. This allows the network to cross-reference overlapping visual fields with linear computational complexity.
Datasets
The model was trained and evaluated using the following benchmarks:
- nuScenes: The primary dataset used for pretraining, 3D object detection, and 3D semantic segmentation evaluation (consisting of 1,000 driving scenes with 6-camera setups and 360° LiDAR).
- Open Waymo: Used to evaluate the cross-dataset transferability of the pretrained camera backbone.
- nuImages: Used to test the backbone's transferability to downstream 2D object detection and instance segmentation tasks.
Key Insights and Ideas
- Pretraining vs. Distillation: Directly forcing a strong camera network to mimic a LiDAR network during downstream finetuning introduces conflicting objectives due to modality differences. Moving this alignment to the pretraining phase acts as a powerful geometric initialization, which is discarded during finetuning to avoid the domain gap.
- The Failure of Standard 2D MAE in 3D: Standard 2D MAE reconstructs raw pixels, which works well on massive, diverse datasets like ImageNet. However, autonomous driving datasets are less visually diverse, making pixel reconstruction prone to overfitting. Reconstructing high-level LiDAR BEV features forces the network to learn structural and spatial representations instead of low-level pixel textures.
- Decoupled Decoders Maximize Capacity: Separating depth estimation from semantic feature completion forces the geometry branch to focus purely on spatial cues, significantly boosting localization accuracy (mATE).
Key Results
- nuScenes 3D Detection: On the nuScenes test set, GeoMIM achieved state-of-the-art performance with 64.4% NDS and 56.1% mAP using a Swin-Large backbone, outperforming previous methods without requiring extra data or test-time augmentation.
- nuScenes 3D Segmentation: Achieved 70.5% mIoU on the test set, a 1.1% improvement over the previous best camera-based method (TPVFormer).
- Transferability: When transferred to the Open Waymo dataset, the GeoMIM-pretrained backbone improved the LET-3D-APL metric by 3.0% compared to a standard self-supervised baseline (MixMAE), proving that the camera-aware pretraining generalizes well to entirely different camera rigs.
Limitations and Future Work
The authors note two primary limitations:
- Data Requirements: GeoMIM still requires a substantial amount of multi-view data during the pretraining phase to learn high-quality representations.
- Teacher Dependence: The quality of the learned camera representations is fundamentally bounded by the quality and completeness of the pretrained LiDAR teacher's BEV features. If the LiDAR model misses objects or contains noise, those errors can propagate into the camera model's pretraining targets.
Summary updated