Complete Scene Reconstruction by Merging Images and Laser Scans

Xiang Gao, Shuhan Shen, Lingjie Zhu, Tianxin Shi, Zhiheng Wang, Zhanyi Hu

Abstract

Large-scale 3D reconstruction of complex architectural sites often faces a trade-off between the global coverage of image-based methods and the local precision of laser scanning. This paper introduces a hybrid pipeline that automates the integration of these two modalities. By using an initial image-based model to plan optimal laser scan locations and employing a coarse-to-fine merging strategy based on synthetic image matching and generalized bundle adjustment, the authors achieve high-accuracy, complete models without the need for manual targets or overlapping scans.

The Accuracy-Completeness Dilemma

In the field of 3D reconstruction, practitioners typically choose between two primary data sources:

  1. Image-based modeling (e.g., Structure-from-Motion): Highly flexible and capable of broad global coverage, but prone to "holes" in low-texture areas or complex geometries.
  2. Laser scanning: Extremely precise but operationally intensive, often requiring numerous overlapping scans and manual target placement for registration.

The proposed framework addresses these limitations by using images as a primary global framework and laser scans as targeted supplements to fill gaps and refine details.

Methodology

The reconstruction pipeline consists of three main phases: image capturing, automated laser planning, and data merging.

Overview of the hybrid 3D reconstruction pipeline, illustrating the three-step process from image capture to final merging.

1. Multi-view and Multi-scale Image Capturing

The process begins with capturing ground-level and aerial (UAV) images. These are processed using standard Structure-from-Motion (SfM) techniques to generate a sparse point cloud and a preliminary surface mesh.

2. Automated Laser Scanning Planning

To optimize field efficiency, the initial SfM mesh is analyzed to identify regions of low confidence—typically areas with low texture or high structural complexity. An integer linear programming model then determines the minimum number of laser scan locations required to cover these "problem areas."

3. Coarse-to-Fine Merging

The integration of laser data into the image-based model occurs in two steps:

  • Coarse Registration: Virtual images are synthesized from the laser point clouds to match the viewpoints of real captured images. This allows for standard 2D feature matching to align the disparate data sources.
  • Fine Merging (Generalized Bundle Adjustment): A joint optimization minimizes both 2D-3D reprojection errors from images and 3D-3D spatial errors from laser points. An adaptive balancing factor ensures that both data types contribute optimally to the final model.
Hardware used for data acquisition, including motorized camera mounts, drones, and terrestrial laser scanners in field operations.

Experimental Evaluation

The pipeline was tested on two significant heritage sites: the Nan-chan Temple (NCT) and the Fo-guang Temple (FGT). These sites feature intricate Tang Dynasty wooden architecture, which is notoriously difficult to reconstruct due to dark interiors and complex bracket systems.

Key Results

  • Precision: The method achieved an RMS error of 19.42 mm for NCT and 27.68 mm for the much larger FGT site, outperforming existing state-of-the-art frameworks.
  • Efficiency: Automated planning reduced the required number of scans for the NCT site from an estimated 43 (for pure laser scanning) to just 11.
  • Completeness: Qualitative analysis showed that fine details, such as roof sculptures and interior brackets, were significantly more defined compared to image-only reconstructions.

Conclusion

By treating laser scanning as a targeted supplement rather than a primary capture method, this research significantly reduces the manual effort required for high-fidelity 3D modeling. Future work aims to integrate handheld RGB-D sensors to further improve coverage in confined spaces where tripod-mounted scanners are impractical.

Summary updated

BibTeX
@article{gao2020complete,
  author = {Xiang Gao and Shuhan Shen and Lingjie Zhu and Tianxin Shi and Zhiheng Wang and Zhanyi Hu},
  title = {{Complete Scene Reconstruction by Merging Images and Laser Scans}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2020},
  doi = {10.1109/tcsvt.2019.2943892},
}
Figure