Line-based Camera Pose Estimation in Point Cloud of Structured Environments

Huai Yu, Weikun Zhen, Yang Wen, Sebastian Scherer

Abstract

Accurate registration of 2D imagery with point clouds is a key technology for image-LiDAR point cloud fusion, camera to laser scanner calibration and camera localization. Despite continuous improvements, automatic registration of 2D and 3D data without using additional textured information still faces great challenges. In this paper, we propose a new 2D-3D registration method to estimate 2D-3D line feature correspondences and the camera pose in untextured point clouds of structured environments. Specifically, we first use geometric constraints between vanishing points and 3D parallel lines to compute all feasible camera rotations. Then, we utilize a hypothesis testing strategy to estimate the 2D-3D line correspondences and the translation vector. By checking the consistency with computed correspondences, the best rotation matrix can be found. Finally, the camera pose is further refined using non-linear optimization with all the 2D-3D line correspondences. The experimental results demonstrate the effectiveness of the proposed method on the synthetic and real dataset (outdoors and indoors) with repeated structures and rapid depth changes.

Introduction

The "modality gap" remains a significant challenge in computer vision: 2D images provide rich texture and color, whereas 3D point clouds offer precise geometric coordinates. Traditional registration methods often rely on appearance-based descriptors (like SIFT) to establish correspondences. However, these methods frequently fail in untextured environments or when faced with significant changes in lighting, weather, or seasons.

Furthermore, 2D-3D registration is inherently a non-convex problem. Simultaneous estimation of rotation and translation without a reliable initial guess often leads to local optima, resulting in globally incorrect poses. This research addresses these issues by proposing a global solution that leverages the geometric regularities of structured, human-made environments.

Methodology

The proposed pipeline exploits the "Manhattan World" assumption, which posits that structured environments are characterized by sets of parallel and orthogonal lines. The process is divided into four main stages:

  1. Line Extraction: 2D line segments are extracted from images using the Line Segment Detector (LSD) algorithm, while 3D line segments are extracted from the point cloud.
  2. Rotation Estimation: To avoid local optima, the method decouples rotation from translation. 2D lines are clustered to identify vanishing points, and 3D lines are clustered to find primary orientations. By aligning these vanishing directions, a small set of feasible rotation candidates is generated.
  3. Translation and Correspondence: For each rotation candidate, a hypothesis-and-test strategy (similar to RANSAC) is employed. The algorithm selects 2D-3D line pairs to calculate a translation vector and verifies it by checking the overlap length of projected lines.
  4. Refinement: The final camera pose is refined through non-linear optimization using the g2o library, minimizing the projection error across all identified inlier line correspondences.
The proposed 2D-3D registration pipeline, illustrating the flow from input data to final 6DOF pose estimation.

Results and Discussion

The method was rigorously evaluated using both synthetic Monte Carlo simulations and real-world datasets, including indoor lounges and outdoor campus buildings with repetitive structures and rapid depth changes.

Key findings include:

  • Robustness: The vanishing point-based approach significantly outperformed standard RANSAC-based methods, particularly in scenes with high outlier fractions.
  • Accuracy: Real-world testing yielded rotation errors of less than 1.5 degrees and position errors under 0.4 meters.
  • Efficiency: The global registration process is computationally efficient, taking approximately 8 seconds per image in a Matlab implementation.
  • Handling Ambiguity: The use of overlap constraints effectively prevents degenerate solutions in scenes with repetitive features, such as rows of windows.
Final estimated camera 6DOF pose visualized against the 3D point cloud.

Conclusion

By prioritizing geometric structure over visual appearance, this method provides a robust solution for registering 2D images to untextured 3D point clouds. Decoupling rotation via vanishing points proves to be a critical step in achieving a reliable global pose estimate. While highly effective in structured environments, future work will explore extending these principles to more natural, less regular settings.

Summary updated

Abstract

Accurate registration of 2D imagery with point clouds is a key technology for image-LiDAR point cloud fusion, camera to laser scanner calibration and camera localization. Despite continuous improvements, automatic registration of 2D and 3D data without using additional textured information still faces great challenges. In this paper, we propose a new 2D-3D registration method to estimate 2D-3D line feature correspondences and the camera pose in untextured point clouds of structured environments. Specifically, we first use geometric constraints between vanishing points and 3D parallel lines to compute all feasible camera rotations. Then, we utilize a hypothesis testing strategy to estimate the 2D-3D line correspondences and the translation vector. By checking the consistency with computed correspondences, the best rotation matrix can be found. Finally, the camera pose is further refined using non-linear optimization with all the 2D-3D line correspondences. The experimental results demonstrate the effectiveness of the proposed method on the synthetic and real dataset (outdoors and indoors) with repeated structures and rapid depth changes.
BibTeX
@misc{yu2019line,
  author = {Huai Yu and Weikun Zhen and Yang Wen and Sebastian Scherer},
  title = {{Line-based Camera Pose Estimation in Point Cloud of Structured Environments}},
  year = {2019},
  eprint = {1912.05013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.05013},
}
Figure