Indirect Correspondence-Based Robust Extrinsic Calibration of LiDAR and Camera

Sungdae Sim, Juil Sock, Kiho KwakSensors 2016 · DOI

TL;DR

This paper introduces a robust and accurate method for calibrating LiDAR and camera sensors, crucial for autonomous systems. It proposes novel techniques for feature weighting and outlier rejection, and extends this to a joint calibration framework for multiple sensors using loop-closing constraints, outperforming existing methods in accuracy and data efficiency.

What Problem Does This Paper Address?

Autonomous vehicles and robots rely heavily on fusing data from multiple sensors like LiDAR and cameras to perceive their environment accurately. However, to effectively combine data from these different sensor modalities, their relative spatial relationship (extrinsic parameters) must be precisely known. Estimating these parameters, especially the six degrees of freedom transformation (rotation and translation), is a challenging problem. Existing methods often struggle with issues like:

  • Correspondence ambiguity: LiDAR points don't directly map to camera pixels, making it hard to find corresponding features.
  • Sensor modality differences: LiDAR operates outside the visual spectrum, and some features (like intensity edges) are only present in one sensor.
  • Outlier sensitivity: Noisy data or incorrect feature correspondences can significantly degrade calibration accuracy.
  • Scalability: Calibrating multiple sensors often involves pairwise calibration, which can lead to accumulated errors.

This paper aims to address these challenges by proposing a more robust and accurate extrinsic calibration method for LiDAR-camera pairs and extending it to a joint calibration for multi-sensor systems.

Methods

The paper presents a multi-step approach for LiDAR-camera extrinsic calibration:

  1. Calibration Target and Data Acquisition: They use a v-shaped calibration target with checkerboard patterns. This target allows for the extraction of both 3D features from LiDAR (depth edges, center point) and 2D features from camera images (line features corresponding to the target's edges and centerline). Data is acquired with the target in various poses and ranges.
  2. Feature Definition and Extraction:
    • Image Features: Three line features (left edge, center, right edge) are extracted from the camera images.
  3. LiDAR Features: Three 3D point features are extracted from LiDAR scans: left depth edge, right depth edge, and the center point. A key innovation here is the use of virtual feature points instead of real LiDAR points for edge features. These virtual points are placed at the expected value of the true edge locations, mitigating issues with sparse LiDAR data and potential inaccuracies in directly identifying edge points.
  4. Optimization: The core of the calibration is minimizing an error function that measures the distance between projected LiDAR features and corresponding image features.
  5. Baseline Approach: Minimizes the squared sum of distances between projected 3D LiDAR points and image line features.
  6. Feature Weighting (Extension 1): Recognizes that the center feature is more accurate than edge features. They introduce weights to give more importance to the center feature's distance error, improving overall accuracy. Weights are determined based on the average residuals of each feature.
  7. Robust Optimization (Extension 2): Addresses the issue of outliers in the data. Instead of simple squared error, they employ a Huber penalty function. This function penalizes large errors less severely than a standard squared error, making the calibration less sensitive to outliers. A maximum distance threshold ($d_{max}$) is defined based on camera focal length and LiDAR angular resolution.
  8. Joint Calibration (Extension to Multi-Sensor Systems): The authors extend their robust approach to jointly calibrate a system with a LiDAR and multiple cameras. This is achieved by:
  9. Loop-Closing Constraints: For systems with more than two sensors, loop-closing constraints are used to ensure consistency. If a transformation is applied sequentially and then reversed, it should return to the original state (e.g., $T_{1 \to 2} T_{2 \to 1} = I$).
  10. Total Error Function: The overall objective function is a weighted sum of individual sensor pair errors (LiDAR-camera, camera-camera). For camera-camera calibration, they use pixel distances between projected 3D points derived from stereo vision. This joint optimization framework estimates all extrinsic parameters simultaneously, leading to more accurate and consistent results than sequential pairwise calibration.

Datasets

  • Synthetic Datasets: Used to evaluate the effect of image pixel noise and LiDAR distance noise on calibration accuracy. Synthetic data was generated with varying noise levels and a fixed number of scan/image pairs.
  • Real Datasets:
    • LiDAR-Camera Pair: A SICK LMS-221 LiDAR and a PointGrey Flea2 camera were used. Data was collected using the v-shaped target in various poses and ranges.
  • LiDAR-Multiple Cameras: A Velodyne HDL-32 3D LiDAR and two PointGrey Flea2 cameras were used. Data was collected with the v-shaped target.
  • Ground Truth Generation: For evaluation, they generated ground truth by manually marking LiDAR scan lines on the calibration target using an IR camera and colored tape.

Key Insights and Ideas

  • Virtual Feature Points: Using virtual points for edge features, placed at the expected location rather than directly measured points, significantly improves robustness to sparse LiDAR data and edge detection inaccuracies.
  • Weighted Feature Importance: Not all features are created equal. Giving more weight to more reliable features (like the center point) demonstrably improves calibration accuracy.
  • Robustness to Outliers is Crucial: Real-world sensor data is often noisy. Employing robust cost functions like Huber's penalty is essential for achieving reliable calibration, especially when manual feature extraction is involved.
  • Joint Calibration with Loop Closing: For multi-sensor systems, a unified optimization framework with loop-closing constraints is superior to sequential pairwise calibration. It prevents error accumulation and ensures global consistency across all sensors.
  • Data Efficiency: The proposed robust method achieves comparable or better accuracy with fewer data pairs than previous methods, making calibration faster and more practical.

Key Results

  • Robustness: The robust approach reduced RMS error by up to 20% compared to the baseline and 5% compared to the weighted approach when using 150 pairs with 20% noisy data.
  • Comparison to State-of-the-Art: The proposed method achieved a line alignment error of 1.87 pixels, significantly better than Wasielewski (2.7 pixels) and Li (5.5 pixels).
  • Data Efficiency: The method achieved comparable accuracy with half the data compared to previous methods. For instance, with 50 pairs, it showed 43% better performance than Wasielewski with 100 pairs.
  • Pose and Range Independence: The proposed method performed consistently across various ranges and poses of the calibration target, unlike some other methods that struggled with specific conditions (e.g., far range).
  • Joint Calibration Performance: The joint calibration method outperformed methods without loop-closing constraints in synthetic experiments, showing better accuracy and robustness to noise. In real-world tests, it achieved comparable accuracy with fewer data pairs than other methods.

Limitations and Future Work

  • Manual Feature Extraction: The paper acknowledges that line features are manually selected, which can introduce noise. While their robust approach mitigates this, fully automatic feature extraction would be ideal.
  • Sensor Configuration: The joint calibration method is described for a specific configuration (LiDAR + two cameras). Extending it to more general multi-sensor configurations might be a future direction.
  • Target Dependence: While the v-shaped target is effective, future work could explore targetless calibration methods.
  • Computational Cost: While more data-efficient, the optimization process itself might still be computationally intensive for real-time applications, though not explicitly detailed as a limitation.

The authors suggest that their joint calibration algorithm can be applied to systems with overlapping sensing regions and correspondences, leading to more accurate results than existing state-of-the-art methods without loop-closing constraints.

Summary updated

BibTeX
@article{sim2016indirect,
  author = {Sungdae Sim and Juil Sock and Kiho Kwak},
  title = {{Indirect Correspondence-Based Robust Extrinsic Calibration of LiDAR and Camera}},
  journal = {Sensors},
  year = {2016},
  doi = {10.3390/s16060933},
}
Figure