Influence of Camera-LiDAR Configuration on 3D Object Detection for Autonomous Driving

Ye Li, Hanjiang Hu, Zuxin Liu, Xiaohao Xu, Xiaonan Huang, Ding Zhao

TL;DR

This paper demonstrates that the physical configuration of sensors—where you mount them and their field of view—can impact 3D object detection performance by as much as 30% in average precision. To avoid the costly "collect-train-evaluate" cycle for every new hardware setup, the authors propose a unified, information-theoretic surrogate metric (S-MS) that accurately predicts detection performance based solely on sensor ray-casting and ground-truth bounding boxes.

What Problem Does This Paper Address?

In the world of autonomous driving, researchers have spent years perfecting sensor fusion algorithms (how to combine camera images and LiDAR point clouds). However, very little attention has been paid to the physical configuration of these sensors. Should LiDARs be stacked in the center or spread out in a line? How does camera field-of-view (FOV) interact with LiDAR placement?

Currently, answering these questions requires a "brute force" approach: mount the sensors, collect a massive dataset, train a deep learning model, and test it. This cycle is incredibly slow and expensive. There is a clear need for a mathematical way to evaluate sensor setups before committing to data collection and training.

Methods

The authors introduce a systematic framework to evaluate sensor configurations using a novel surrogate metric.

  1. Sensor Perception Modeling: They treat both cameras and LiDARs as ray-casting systems. LiDAR is modeled as rotating vertical rays, while the camera is modeled using a pin-hole projection where each pixel represents a ray traversing 3D space. LiDAR and Camera sensing models Camera sensing model
LiDAR and Camera sensing models
Camera sensing model
  1. Probabilistic Occupancy Grid (POG): The region of interest (ROI) is divided into voxels. Using ground-truth data, they calculate the probability that any given voxel is occupied by a target object (Car, Pedestrian, or Bicycle).
  2. Unified Surrogate Metric (S-MS): Drawing from information theory, they define a metric based on Information Gain (IG). Essentially, they measure how much a specific sensor configuration reduces the uncertainty of the voxel occupancy distribution. The final metric, S-MS, is a weighted sum of the information gain from the camera and the LiDAR.
  3. Accelerated Evaluation Framework: They built a pipeline in the CARLA simulator to automate data acquisition, model training, and performance testing across various configurations. Evaluation framework
Evaluation framework

Datasets

The study utilized the CARLA simulator to generate eight "nuScenes-like" datasets.

  • Scale: Each dataset contains approximately 38,000 frames.
  • Content: Six camera images and one LiDAR point cloud per frame, with 3D bounding boxes for Cars, Bicycles, and Pedestrians.
  • Diversity: Data was collected across four different virtual towns (Town 1, 3, 4, and 6) with manually recorded routes to ensure consistency across sensor setups.

Key Insights and Ideas

  • Hardware is as important as Software: The choice of sensor placement can lead to a 30% discrepancy in detection accuracy, even when using the exact same fusion algorithm.
  • The "Trapezoid" Advantage: Among the tested LiDAR configurations (Center, Pyramid, Line, Trapezoid), the Trapezoid layout (four LiDARs placed in parallel front and back) generally yielded the best performance.
  • Cross-Modal Compensation: A superior LiDAR configuration can sometimes compensate for a narrower camera FOV, and vice versa. For example, the "Narrow Camera + Trapezoid LiDAR" setup often outperformed "Wide Camera + Center LiDAR."
  • Algorithmic Sensitivity: Certain sensor layouts can act like "adversarial attacks" on specific algorithms. For instance, the "Pyramid" LiDAR layout significantly degraded the performance of the Transfusion algorithm for car detection, despite working well for others.

Key Results

The authors validated their surrogate metric against three state-of-the-art fusion algorithms: Transfusion, Bevfusion, and PointAugmenting.

  • Metric Correlation: The proposed S-MS metric showed a clear rising tendency correlated with actual mean Average Precision (mAP). As the information gain predicted by the metric increased, so did the real-world detection performance. Correlation between S-MS and mAP
  • Camera FOV Impact: "Wide" camera configurations (70°-110° FOV) consistently outperformed "Narrow" configurations (55° FOV) by up to 10% mAP.
  • Object Specificity: The "Line" LiDAR configuration proved excellent for small objects like Bicycles and Pedestrians because it increases point cloud density in the lateral direction, where these objects are most frequently encountered.
Correlation between S-MS and mAP

Limitations and Future Work

  • Hyperparameter Sensitivity: The weight $\lambda$ used to balance camera vs. LiDAR information gain was kept constant, but the authors suggest it might need to vary based on the specific detection algorithm or target object type.
  • Sensor Variety: The current study focuses on mechanical LiDAR and standard cameras. Future work could extend the metric to include solid-state LiDAR, radar, and more diverse industrial sensor placements.
  • Real-World Gap: While the CARLA results are promising, validating the surrogate metric on physical vehicles in real-world environments remains a critical next step.

Summary updated

Abstract

Cameras and LiDARs are both important sensors for autonomous driving, playing critical roles in 3D object detection. Camera-LiDAR Fusion has been a prevalent solution for robust and accurate driving perception. In contrast to the vast majority of existing arts that focus on how to improve the performance of 3D target detection through cross-modal schemes, deep learning algorithms, and training tricks, we devote attention to the impact of sensor configurations on the performance of learning-based methods. To achieve this, we propose a unified information-theoretic surrogate metric for camera and LiDAR evaluation based on the proposed sensor perception model. We also design an accelerated high-quality framework for data acquisition, model training, and performance evaluation that functions with the CARLA simulator. To show the correlation between detection performance and our surrogate metrics, We conduct experiments using several camera-LiDAR placements and parameters inspired by self-driving companies and research institutions. Extensive experimental results of representative algorithms on nuScenes dataset validate the effectiveness of our surrogate metric, demonstrating that sensor configurations significantly impact point-cloud-image fusion based detection models, which contribute up to 30% discrepancy in terms of the average precision.
BibTeX
@inproceedings{li2024influence,
  author = {Ye Li and Hanjiang Hu and Zuxin Liu and Xiaohao Xu and Xiaonan Huang and Ding Zhao},
  title = {{Influence of Camera-LiDAR Configuration on 3D Object Detection for Autonomous Driving}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10610896},
}
Figure