TL;DR
This paper introduces a novel, target-free method for calibrating cameras and LiDAR sensors on autonomous vehicles. By leveraging monocular depth estimation and geometric mutual information, their approach achieves high accuracy and robustness to initial errors, eliminating the need for ground truth data or manual feature engineering.
What Problem Does This Paper Address?
Accurate calibration between sensors like cameras and LiDAR is crucial for autonomous systems to effectively fuse data and make reliable decisions. This extrinsic calibration, which determines the relative pose between sensors, is typically performed once during manufacturing. However, sensors can drift over time, necessitating online calibration methods. Existing methods often rely on ground truth data (which is expensive and difficult to acquire at scale), artificial calibration targets, or hand-engineered features, all of which limit their practicality for real-world, continuous calibration. This paper addresses the challenge of achieving robust, accurate, and automated online camera-LiDAR extrinsic calibration without these limitations.
Methods
The core of this work lies in a "depth-to-depth" (D2D) mutual information (MI) approach. Instead of relying on intensity features (which are sensitive to lighting and surface properties), the authors propose using geometric depth information.
Here's a breakdown of their method:
- Virtual Depth Sensor: They leverage state-of-the-art monocular depth estimation networks to infer a depth map from camera images. This effectively creates a "virtual depth sensor" that provides geometric information from the camera.
- Geometric Mutual Information: The algorithm then calculates the mutual information between the inferred camera depth map and the LiDAR's inherent depth measurements. Mutual information is a measure of statistical dependence between two random variables. Maximizing MI between the depth features from both sensors indicates that the extrinsic calibration is accurate, as it signifies the highest degree of shared information.
- Target-Free and Automatic: This approach is "target-free" because it doesn't require any special calibration targets like checkerboards. It's also automatic as it relies on features directly extractable from sensor data.
- Optimization: The extrinsic calibration parameters (rotation and translation) are optimized iteratively to maximize the calculated mutual information. They use Powell's BOBYQA algorithm, a derivative-free optimization method, which is well-suited for this problem.
Datasets
The proposed method was evaluated on two publicly available datasets:
- KITTI: A widely used dataset for autonomous driving research, featuring standard camera and LiDAR data.
- KITTI-360: A more recent dataset that includes 180° field-of-view fisheye camera images, providing a more challenging scenario for calibration.
For depth estimation, they utilized pre-trained networks:
- FeatDepth: Used for the KITTI sequence.
- MiDaS: Used for the KITTI-360 sequence.
Key Insights and Ideas
- Depth is More Informative than Intensity: The paper strongly demonstrates that geometric depth features provide significantly more reliable and structured information for camera-LiDAR calibration compared to intensity features. This is a crucial insight, as intensity-based methods are prone to failure under varying environmental conditions.
- Monocular Depth Estimation as a Game-Changer: The advancement in monocular depth estimation networks has unlocked the potential for robust geometric feature extraction from camera images, making this D2D approach feasible and effective.
- Robustness to Initialization: A key advantage highlighted is the algorithm's remarkable robustness to initial calibration errors. This is critical for online calibration, where initial guesses might not be very accurate.
- Explainability: The geometric nature of the features and the MI metric make the approach more explainable than black-box deep learning methods.
Key Results
The paper presents compelling results from two main experiments:
- Rotation Only Experiment (3 DoF):
- The D2D method achieved significantly higher "hit rates" (successful calibrations) compared to intensity-to-intensity (I2I) methods, even with initial errors as large as 20°. For example, at 10° error, D2D achieved hit rates of 96.5% and 100% on KITTI and KITTI-360, respectively, while I2I hit rates were near zero.
- D2D also demonstrated superior precision, with optimized rotation parameters varying by less than 0.02°.
- Rotation and Translation Experiment (6 DoF):
- Even in this more challenging scenario with translation errors up to 50cm, D2D consistently outperformed I2I in terms of hit rates. For instance, at 0.5° rotation and 50cm translation error, D2D achieved hit rates of 88% and 76.5% on KITTI and KITTI-360, respectively, compared to I2I's 9.5% and 22%.
- D2D recovered ground truth calibrations with high accuracy, within 0.18° and 9.9cm for KITTI, and 0.08° and 5.4cm for KITTI-360.
Limitations and Future Work
- Limitations:
- The performance of the D2D method is dependent on the quality of the monocular depth estimation network. Degraded depth estimation quality could impact calibration accuracy.
- While robust, the paper acknowledges that the 6 DoF case is more challenging, and further exploration of optimization strategies might be beneficial.
- Future Work:
- Investigating the temporal consistency and data dependency of the calibration.
- Understanding the impact of occlusions and large baseline camera-LiDAR systems.
- Exploring more advanced optimization strategies for the 6 DoF case.
- Researching the effect of degraded monocular depth estimation quality on calibration performance.
- Extending the geometric mutual information approach to continuous online intrinsic camera calibration (COIC).
Summary updated