REGISTRATION OF MMS LiDAR POINTS AND PANORAMIC IMAGE SEQUENCE USING RELATIVE ORIENTATION MODEL

Na Zhu, Bisheng Yang, Yongxing Jia2020 · DOI

Abstract

This paper proposes a robust method for registering Mobile Mapping System (MMS) LiDAR point clouds with panoramic image sequences using a Relative Orientation Model (ROM). By leveraging the geometric relationships between consecutive panoramic images, the method solves for relative position and attitude parameters, enabling accurate data fusion even in environments where GPS/IMU data may be unreliable. The approach utilizes SURF for feature matching and RANSAC for outlier removal. Experimental results across two distinct datasets demonstrate high registration accuracy, with mean errors of 7.36 pixels and 3.75 pixels, respectively.

Introduction

Mobile Mapping Systems (MMS) are essential for creating high-definition maps for autonomous driving and urban modeling. These systems simultaneously capture 3D LiDAR points and 2D panoramic images. However, accurately fusing these data types—ensuring that every 3D point corresponds correctly to its 2D image pixel—remains a significant challenge.

Existing registration methods often suffer from two primary limitations: a heavy reliance on high-precision GPS/IMU data, which can fail in urban canyons or under dense foliage, and a dependence on specific environmental features like road markings or lamp posts. This research introduces a "scene-independent" registration method that relies on the inherent overlap between consecutive panoramic images rather than external landmarks or continuous hardware signals.

Methodology

The core of the proposed method is the application of the Relative Orientation Model (ROM) to spherical panoramic imagery. Instead of attempting to register LiDAR points to every image independently, the system calculates the relative movement between image pairs to propagate registration from a known starting point.

Relative Orientation Models

Two variations of the ROM are employed based on the available data:

  • PA-ROM (Position/Attitude): Applied when both the camera's position and its orientation (tilt/rotation) are unknown.
  • A-ROM (Attitude): Utilized when the camera's position is known (e.g., via GPS), but its orientation requires refinement.

Feature Matching and Refinement

To solve the ROM equations, the system automatically extracts feature points from panoramic image pairs using the SURF (Speeded Up Robust Features) algorithm. Because moving objects (like cars) or repetitive textures can create false matches, the RANSAC algorithm is combined with the ROM to filter out outliers, ensuring that only stable, static features are used for orientation calculations.

Experimental Evaluation

The method was evaluated using two road scene datasets captured with a Ladybug5 panoramic camera:

  • Dataset I: A 250-meter road segment comprising 10 panoramic images and 3.8 million LiDAR points.
  • Dataset II: A 350-meter segment comprising 7 panoramic images and 8.2 million LiDAR points.
Dataset I: Panoramic image sequence and corresponding LiDAR point cloud map.
Dataset II: Panoramic image sequence and LiDAR data for the second test site.

Results and Discussion

The registration accuracy was assessed by manually selecting checkpoints between the LiDAR points and the panoramic images.

Accuracy and Stability

In Dataset I, the mean registration error was 7.36 pixels, while Dataset II achieved a superior accuracy of 3.75 pixels. The study also compared "Continuous Registration" (matching sequential frames like 1-2, 2-3) against "Discontinuous Registration" (matching non-sequential frames like 1-3, 1-4). The results indicated that continuous registration is significantly more stable, as increasing the distance between image pairs reduces the overlapping area and the number of reliable feature matches.

Error Accumulation

A key finding was the presence of "drift" over long sequences. In Dataset I, the registration error increased from 4.93 pixels at the start of the sequence to 10.67 pixels by the tenth image. This accumulation occurs because each frame's orientation is calculated relative to the previous one, allowing small errors to compound over time.

Conclusion

The ROM-based approach provides a flexible and accurate solution for MMS data registration that is less dependent on external hardware or specific road features. While the method demonstrates high precision in diverse environments, future work is needed to address error accumulation. Integrating global optimization techniques, such as loop closure commonly used in SLAM (Simultaneous Localization and Mapping), could mitigate drift and enable the registration of much longer image sequences.

Summary updated

Abstract

We propose using the relative orientation model (ROM) of panoramic to register the MMS LiDAR points and panoramic image sequence, which has the wide applicability. The feature points, extracted and matched from panoramic image pairs, are used to solve the relative position and attitude parameters in the ROM, then, combining the absolute position and attitude parameters of the initial panoramic image, the MMS LiDAR points and panoramic image sequence are registered. First, we propose the position/attitude ROM (PA-ROM) and attitude ROM (A-ROM) of panoramic images respectively, which are apply to the position/attitude parameters both unknown and only the attitude parameters unknown. Second, we automatically extract and match feature points from panoramic image pairs using the SURF algorithm, as these mismatching points will affect the registration accuracy, the RANSAC algorithm and ROM were used to choose the best matching points automatically. Finally, we select the feature points manually from MMS LiDAR points and panoramic image sequence as the checkpoints, and then compare the registration accuracy of continuous/discontinuous panoramic image pairs. The results show that MMS LiDAR points and panoramic image sequence are registered accurately based on ROM (7.36 and 3.75 pixels in dataset I and II), what's more, our registration method just tackle the image pairs (uninvolved LiDAR points), so it is suitable for more road scenes.
BibTeX
@inproceedings{zhu2020registration,
  author = {Na Zhu and Bisheng Yang and Yongxing Jia},
  title = {{REGISTRATION OF MMS LiDAR POINTS AND PANORAMIC IMAGE SEQUENCE USING RELATIVE ORIENTATION MODEL}},
  booktitle = {˜The},
  year = {2020},
  doi = {10.5194/isprs-archives-xliii-b1-2020-291-2020},
}
Figure