Experimental Evaluation of 3D-LIDAR Camera Extrinsic Calibration

Subodh Mishra, Philip R. Osteen, Gaurav Pandey, Srikanth Saripalli

TL;DR

This paper experimentally evaluates three popular 3D-LIDAR to camera extrinsic calibration algorithms (PPC-Cal, PBPC-Cal, and MSG-Cal) on a real-world robotic platform. They find that PBPC-Cal offers the best robustness, especially with noisy sensors, while MSG-Cal shows promise for multi-sensor systems but is sensitive to sensor noise.

What Problem Does This Paper Address?

Autonomous robots rely on fusing data from multiple sensors, like cameras and LIDARs, to understand their environment. For this fusion to be accurate, the spatial relationship (extrinsic calibration) between these sensors must be precisely known. While many algorithms exist for this, there's a lack of comprehensive experimental comparisons that evaluate their practical performance, robustness to noise, and ease of use on real-world robotic systems. This paper aims to fill that gap by providing a thorough experimental evaluation of three prominent 3D-LIDAR to camera calibration methods.

Methods

The authors compare three algorithms:

  • PPC-Cal (Point to Plane Constraint Calibration): This method uses correspondences between points detected on a planar target in the LIDAR point cloud and the plane's parameters estimated in the camera image. It relies on a checkerboard pattern for easy plane detection in the camera.
  • PBPC-Cal (Point to Back-projected Plane Constraint Calibration): An extension of PPC-Cal, this algorithm adds a "point to back-projected plane" constraint. This requires detecting not only the planar target but also its edges in both sensor modalities, providing more geometric constraints.
  • MSG-Cal (Multi-Sensor Graph-based Calibration): This is a graph-based optimization approach that jointly calibrates multiple sensors simultaneously. It starts with pair-wise calibrations (using PPC-Cal or similar constraints) and then refines them through a global optimization process. It utilizes AprilTags for camera-based target detection.

The evaluation focuses on robustness to random initialization, accuracy using Mean Line Re-projection Error (MLRE), and comparison against factory stereo calibration.

Datasets

The paper does not introduce new datasets. Instead, it uses data collected from a custom sensor suite comprising an Ouster OS1 64 Channel LIDAR, a Velodyne VLP-32 LIDAR, a Basler Ace camera, and a Karmin2 Stereo Vision System. The experiments involve collecting multiple observations of a planar calibration target from various viewpoints.

Key Insights and Ideas

  • Robustness to Initialization: PPC-Cal and PBPC-Cal are generally robust to random initial guesses for calibration parameters, meaning they can often converge to the correct solution regardless of the starting point. MSG-Cal, however, showed divergence in a few cases, suggesting it can get stuck in local minima.
  • Sensor Noise Impact: The Ouster OS1 LIDAR is identified as a noisy sensor. PPC-Cal's performance degrades significantly with this noisy sensor, while PBPC-Cal remains relatively unaffected. MSG-Cal's performance also suffers when the noisy Ouster LIDAR is part of the system, likely due to the graph optimization propagating errors.
  • PBPC-Cal's Advantage: The addition of the "point to back-projected plane" constraint in PBPC-Cal provides more geometric information, making it more resilient to noisy sensor data and leading to more consistent calibration results.
  • MSG-Cal's Trade-offs: While MSG-Cal offers the potential for joint calibration of multiple sensors, its performance is sensitive to the quality of individual sensor data. When the noisy Ouster LIDAR is removed, MSG-Cal's performance improves and becomes comparable to PPC-Cal.

Key Results

  • Accuracy (MLRE): PBPC-Cal consistently achieved the lowest Mean Line Re-projection Error (MLRE) across various sensor pairs, indicating higher accuracy. For example, for the VLP-32 and Right Stereo Camera pair, PBPC-Cal achieved an MLRE of 1.88719 pixels, significantly better than PPC-Cal (2.94664 pixels) and MSG-Cal (11.3415 pixels).
  • Accuracy (Stereo Calibration Comparison): When comparing against factory stereo calibration, PBPC-Cal also showed smaller errors in translation and rotation compared to PPC-Cal and MSG-Cal, especially for the Ouster LIDAR.
  • Robustness: PPC-Cal and PBPC-Cal demonstrated good robustness to random initialization, converging in most trials. MSG-Cal showed divergence in some trials.

Limitations and Future Work

  • Noisy Sensor Sensitivity: MSG-Cal's performance is significantly impacted by noisy sensors like the Ouster OS1. The authors hypothesize this is because the graph optimization propagates errors from noisy nodes to others.
  • Data Collection for PBPC-Cal: PBPC-Cal requires more careful data collection to ensure all edges of the target are visible and not parallel to LIDAR scan lines, making it more tedious than PPC-Cal.
  • Future Work: The authors propose integrating PBPC-Cal into the pair-wise calibration step of MSG-Cal. This would leverage PBPC-Cal's robustness to noisy sensors while retaining the benefits of MSG-Cal's joint global optimization for multi-sensor systems. They also aim to explore using PBPC-Cal in the pair-wise calibration step of MSG-Cal to combine the strengths of both approaches.

Summary updated

Abstract

In this paper we perform an experimental comparison of three different target based 3D-LIDAR camera calibration algorithms. We briefly elucidate the mathematical background behind each method and provide insights into practical aspects like ease of data collection for all of them. We extensively evaluate these algorithms on a sensor suite which consists multiple cameras and LIDARs by assessing their robustness to random initialization and by using metrics like Mean Line Re-projection Error (MLRE) and Factory Stereo Calibration Error. We also show the effect of noisy sensor on the calibration result from all the algorithms and conclude with a note on which calibration algorithm should be used under what circumstances.
BibTeX
@inproceedings{mishra2020experimental,
  author = {Subodh Mishra and Philip R. Osteen and Gaurav Pandey and Srikanth Saripalli},
  title = {{Experimental Evaluation of 3D-LIDAR Camera Extrinsic Calibration}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9340911},
}
Figure