Abstract
Generating high-fidelity 3D building models typically requires expensive LiDAR data or multi-view stereo imagery. This paper introduces the Keypoints Inference By Segmentation (KIBS) method, a deep learning pipeline that achieves Level of Detail 2 (LOD2) reconstruction using only a single, non-orthogonal satellite image. By decomposing the problem into 2D roof segmentation and discrete height classification of corners, the approach offers a scalable and efficient solution for urban digital twin creation.
Introduction
Creating 3D digital twins of cities is essential for modern urban planning, environmental analysis, and disaster management. However, a significant bottleneck exists in reaching Level of Detail 2 (LOD2)—a standard that includes specific roof shapes and slopes. Most current methods rely on aerial LiDAR or stereo-photogrammetry, both of which are costly and computationally intensive. While single-image reconstruction exists, it is often limited to LOD1 (simple "shoe box" geometries) or simple indoor scenes.
Scaling LOD2 reconstruction to a city-wide level, where a single satellite image may contain hundreds of thousands of complex roof planes, remains a major challenge in computer vision. This research addresses this gap by leveraging the visual cues present in non-orthogonal satellite imagery.
The KIBS Pipeline
The authors propose the KIBS (Keypoints Inference By Segmentation) pipeline, which utilizes the Mask R-CNN architecture in a two-stage process:
- 2D Roof Segmentation: A Mask R-CNN (ResNet-50 + FPN) identifies and segments individual roof sections from 0.38m resolution RGB satellite imagery.
- 3D Corner Extraction: The segmented masks are blended back into the original image to provide "contextual focus." A second Mask R-CNN then performs panoptic segmentation to identify roof corners. Crucially, the researchers treat height-to-ground as a discrete classification task (defining 19 classes from 1m to 19m) rather than a continuous regression, which improves training stability and precision.
- Geometry Reconstruction: Once at least three corners of a roof section are identified with their respective heights, Delaunay triangulation is used to calculate 3D plane coefficients. The final mesh is generated using Kinetic Shape Reconstruction (KSR) to ensure clean, manifold surfaces.
Experimental Setup and Results
The study utilized Maxar Worldview-3 satellite imagery (0.38m/pixel) covering two regions in France: Mourmelon-le-Grand (73 $km^2$) for training and Sissonne (25 $km^2$) for testing generalization.
The KIBS method demonstrated high efficiency and accuracy:
- Segmentation Accuracy: Achieved a Jaccard index (IoU) of 88.55% on the primary dataset and 75.21% on the generalization dataset.
- Height Precision: The mean error for roof heights was recorded at 1.60 meters and 2.06 meters, respectively.
- Efficiency: The system can reconstruct entire urban areas in minutes, significantly faster than manual or procedural geometric approaches.
Discussion and Key Insights
A core insight of this work is that obliquity is an advantage. Tilted satellite views provide essential visual cues—such as wall inclination, shadows, and ridge perspectives—that the deep learning model exploits to infer height from a single 2D frame. Additionally, the "blending strategy" prevents the second stage of the model from being overwhelmed by the density of points in raw satellite data.
Despite its success, the model has limitations. It is sensitive to specific satellite viewing and solar angles, and the current height classification is capped at 19 meters, making it unsuitable for skyscrapers. Future work will focus on developing modular learning suites to handle varying viewing angles and moving toward a fully end-to-end neural architecture.
Summary updated