Multimodal Object Query Initialization for 3D Object Detection

Mathijs R. van Geerenstein, Felicia Ruppel, Klaus Dietmayer, Dariu M. GavrilaICRA 2024 · DOI

TL;DR

This paper introduces EfficientQ3M, a novel and efficient method for initializing object queries in transformer-based 3D object detection. By leveraging both LiDAR and camera data in a "modality-balanced" way, EfficientQ3M achieves state-of-the-art performance on the nuScenes benchmark with fewer queries and decoder layers, while also being significantly faster than previous methods.

What Problem Does This Paper Address?

3D object detection is crucial for autonomous driving, enabling vehicles to understand their surroundings. Transformer-based models have shown great promise in this area, using "object queries" as initial guesses for potential objects. Current methods often initialize these queries using only LiDAR data, which is sparse, or rely on complex, input-agnostic initialization schemes that can be slow and computationally expensive. This limits the full potential of multimodal sensor fusion (LiDAR and camera) and introduces latency. The paper aims to bridge this gap by proposing an efficient, multimodal initialization strategy that effectively utilizes both LiDAR and camera information.

Methods

The core contribution is EfficientQ3M, a multimodal input-dependent object query initialization method. Instead of relying on fixed, input-agnostic query distributions, EfficientQ3M predicts initial query locations based on the current sensor inputs.

Here's a breakdown of the key steps:

  1. Dense Grid of Proposals: A large, fixed grid of "dense query location proposals" is created across the detection area.
  2. Feature Sampling and Fusion: For each proposal, LiDAR and camera features are sampled at the corresponding location. These features are then fused using a lightweight network to create initial query feature vectors.
  3. Initial Bounding Box Prediction: These dense queries are used to predict a set of bounding boxes.
  4. Top-M Selection: The top-M bounding boxes with the highest confidence scores are selected.
  5. Query Refinement: The 3D locations of these top-M proposals are refined with predicted 3D offsets.
  6. Final Query Initialization: New query feature vectors are generated by re-sampling features at these refined locations. These M queries, along with their refined 3D locations, are then fed into the transformer decoder.

A key aspect is the modality-balanced transformer decoder, where object queries can access both LiDAR and camera features throughout the decoder layers, unlike sequential fusion methods. This approach is modular and can be applied to various sensor combinations.

Datasets

The primary dataset used for training and evaluation is nuScenes. This is a large-scale autonomous driving dataset featuring 1000 scenes with 3D bounding box annotations for 10 object classes, collected at 2 Hz.

Key Insights and Ideas

  • Input-Dependent Initialization is Key: Initializing object queries based on actual sensor data, rather than a fixed distribution, significantly improves performance and efficiency.
  • Modality-Balanced Fusion Matters: Allowing queries to access both LiDAR and camera features throughout the decoder, rather than sequentially, unlocks the full potential of multimodal sensing.
  • Efficiency Through Lightweight Initialization: The proposed initialization is computationally lightweight, enabling faster inference and reducing the need for many decoder layers.
  • Fewer Queries, Better Performance: EfficientQ3M demonstrates that with effective input-dependent initialization, a smaller number of queries (e.g., 200) can achieve superior results compared to methods using many more queries (e.g., 900).
  • 3D Location Prediction is Advantageous: Predicting initial 3D query locations, as opposed to 2D, allows for more accurate sampling of image features.

Key Results

  • State-of-the-Art Performance: EfficientQ3M achieves state-of-the-art results on the nuScenes benchmark for both LiDAR-only and LiDAR-camera fusion settings.
  • Improved Accuracy: For LiDAR-camera fusion, EfficientQ3M outperforms FUTR3D and TransFusion.
  • Significant Efficiency Gains: The proposed method adds only +6.9 ms of latency for LiDAR-camera fusion, compared to TransFusion's +105.7 ms overhead.
  • Reduced Query/Layer Requirements: With 200 queries, EfficientQ3M outperforms FUTR3D with 900 queries, and requires fewer decoder layers for comparable performance.
  • Ablation Studies:
    • Initializing queries with sensor features (Refs.
  • Feats.) shows a benefit, especially with fewer decoder layers (1 layer), compared to just positional embeddings.
  • The input-agnostic baseline degrades significantly when the number of object queries is reduced, highlighting the advantage of input-dependent methods.

Limitations and Future Work

  • Limitations: The paper notes that while 200 queries are sufficient, using 900 queries with the proposed method still shows a marginal benefit, potentially due to larger self-attention operations or compensating for imperfect initialization. The authors also mention that their method is not specific to any sensor suite, but the current experiments focus on LiDAR and camera.
  • Future Work: The authors suggest extending the scope of experiments to include more sensor setups, such as RADAR-camera and camera-only configurations. They also propose exploring the application of their modular method to different transformer decoders.

Summary updated

BibTeX
@inproceedings{geerenstein2024multimodal,
  author = {Mathijs R. van Geerenstein and Felicia Ruppel and Klaus Dietmayer and Dariu M. Gavrila},
  title = {{Multimodal Object Query Initialization for 3D Object Detection}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10610905},
}
Figure