Line-Based Registration of Panoramic Images and LiDAR Point Clouds for Mobile Mapping

Tingting Cui, Shunping Ji, Jie Shan, Jianya Gong, Kejian Liu2017 · DOI

TL;DR

This paper introduces a novel line-based registration method to precisely align panoramic images and LiDAR point clouds from mobile mapping systems. By leveraging straight-line features and a rigorous panoramic camera model, the method significantly improves the accuracy of 2D-3D data fusion, crucial for applications like autonomous driving and urban modeling.

What Problem Does This Paper Address?

Creating detailed 3D models of urban environments is increasingly important for applications like autonomous driving and urban planning. Mobile Mapping Systems (MMS) are efficient tools for collecting data, typically using cameras for high-resolution imagery and LiDAR for 3D point clouds. However, fusing these two data types requires accurate 2D-3D registration. Existing methods often struggle with misalignments caused by system calibration errors, unsynchronized data acquisition, or unreliable GPS signals. Furthermore, many MMS now use panoramic cameras, which require specialized sensor models beyond the standard frame camera models used in prior research. This paper addresses the challenge of accurately registering panoramic images with LiDAR point clouds, specifically by developing a method that accounts for the unique geometry of panoramic cameras.

Methods

The core of this paper is a line-based registration method that uses straight-line features as primitives for aligning panoramic images and LiDAR point clouds.

Here's a breakdown of their approach:

  • System Configuration: They first detail the MMS setup, including the coordinate systems of the vehicle, LiDAR, and the panoramic camera rig (composed of multiple fisheye lenses).
  • Panoramic Camera Model: They propose and utilize a rigorous panoramic camera model that accounts for the multi-camera setup and their individual projection centers, distinguishing it from simpler spherical camera models. This model is crucial for accurately projecting 3D LiDAR points into the panoramic image space.
  • Line Feature Extraction:
    • LiDAR: Straight lines are extracted from the 3D LiDAR point cloud. For buildings, this involves region growing segmentation, boundary point detection, and RANSAC fitting. For street light poles and curbs, specialized algorithms are used to identify and fit linear features.
  • Panoramic Images: Straight lines are detected in the rectified panoramic images.
  • Line-Based Transformation Model: A mathematical model is established to relate 3D LiDAR line segments to their corresponding 2D projections in the panoramic image. This model incorporates the rigorous panoramic camera model.
  • Registration Solution: A non-linear least squares method is employed to iteratively solve for the unknown transformation parameters (rotation and translation) between the LiDAR and panoramic image coordinate systems. RANSAC is used to robustly handle outliers and potential mismatches between extracted line features.

Datasets

The authors used a dataset collected on Hankou North Street in Wuhan City. This dataset includes:

  • LiDAR Point Cloud: Approximately 1.2 million points, classified into buildings, trees, poles, streets, and moving cars.
  • Panoramic Imagery: Captured by a Ladybug 3 camera, with images exposed at approximately 7.5m intervals along the driving path.
  • GPS/IMU Data: Post-processed with RTK technology, providing high-accuracy positioning (up to 0.1m).

The paper does not introduce a new benchmark dataset but rather uses a real-world dataset to evaluate their proposed method.

Key Insights and Ideas

  • The Importance of a Rigorous Panoramic Model: The paper strongly suggests that using a simple spherical camera model is insufficient for accurate registration with panoramic cameras. The rigorous multi-camera rig model, which accounts for individual lens properties, leads to significantly better results. This highlights the need for specialized sensor models when dealing with complex imaging systems.
  • Line Features as Robust Primitives: Straight lines, particularly those from buildings, poles, and curbs, are identified as reliable and discriminative features for registration. Their detectability in both LiDAR point clouds and panoramic images makes them ideal for bridging the gap between the two sensor modalities.
  • Line-Based Registration Outperforms Point-Based: The paper implicitly supports the idea that line-based registration can achieve higher accuracy than methods relying solely on point features, a concept rooted in traditional photogrammetry.
  • Addressing Real-World Misalignments: The method is designed to compensate for practical issues like calibration errors and synchronization inaccuracies, which are common in mobile mapping scenarios.

Key Results

  • Improved Accuracy: The proposed line-based registration method, using the panoramic camera model, achieved better registration accuracy compared to using a spherical camera model.
  • RMSE (pixels) for the panoramic model was 4.244, while for the spherical model it was 4.718.
  • Average residuals decreased from 12.0 to 2.9 pixels with the panoramic model after registration.
  • Visual Alignment: Figures demonstrate a clear visual improvement in the alignment of LiDAR point clouds and panoramic images after registration, with features like building edges and poles becoming well-matched.
  • Increased Overlap Rate: Statistical evaluation showed an increase in the "overlap rate" of linear features between the LiDAR and image datasets after registration, indicating better correspondence. This rate increased by up to 2% for some lenses.

Limitations and Future Work

  • LiDAR Point Cloud Errors: The authors acknowledge that errors inherent in the LiDAR point cloud itself (e.g., calibration errors of multiple scanners) can affect registration accuracy.
  • Automation of Feature Correspondence: While line features are extracted, finding reliable correspondences between LiDAR and image lines can still be challenging and may require more advanced techniques.
  • Future Work Suggestions:
    • Utilizing Other LiDAR Attributes: Incorporating features like LiDAR intensity could further enhance registration accuracy.
  • Advanced Correspondence Matching: Exploring local statistical similarity measures like mutual information for more robust feature matching.
  • Improved Automation: Developing fully automated methods for feature extraction and correspondence finding.

Summary updated

BibTeX
@misc{cui2017line,
  author = {Tingting Cui and Shunping Ji and Jie Shan and Jianya Gong and Kejian Liu},
  title = {{Line-Based Registration of Panoramic Images and LiDAR Point Clouds for Mobile Mapping}},
  year = {2017},
  doi = {10.3390/s17010070},
}

Loading related content...

Figure