3D detection of roof sections from a single satellite image and application to LOD2-building reconstruction

Johann Lussange, Mulin Yu, Yuliya Tarabalka, Florent LafargearXiv 2023 · DOI · arXiv

Abstract

Generating high-fidelity 3D building models typically requires expensive LiDAR data or multi-view stereo imagery. This paper introduces the Keypoints Inference By Segmentation (KIBS) method, a deep learning pipeline that achieves Level of Detail 2 (LOD2) reconstruction using only a single, non-orthogonal satellite image. By decomposing the problem into 2D roof segmentation and discrete height classification of corners, the approach offers a scalable and efficient solution for urban digital twin creation.

Introduction

Creating 3D digital twins of cities is essential for modern urban planning, environmental analysis, and disaster management. However, a significant bottleneck exists in reaching Level of Detail 2 (LOD2)—a standard that includes specific roof shapes and slopes. Most current methods rely on aerial LiDAR or stereo-photogrammetry, both of which are costly and computationally intensive. While single-image reconstruction exists, it is often limited to LOD1 (simple "shoe box" geometries) or simple indoor scenes.

Comparison of Level of Detail (LOD) standards for 3D building models.

Scaling LOD2 reconstruction to a city-wide level, where a single satellite image may contain hundreds of thousands of complex roof planes, remains a major challenge in computer vision. This research addresses this gap by leveraging the visual cues present in non-orthogonal satellite imagery.

The KIBS Pipeline

The authors propose the KIBS (Keypoints Inference By Segmentation) pipeline, which utilizes the Mask R-CNN architecture in a two-stage process:

  1. 2D Roof Segmentation: A Mask R-CNN (ResNet-50 + FPN) identifies and segments individual roof sections from 0.38m resolution RGB satellite imagery.
  2. 3D Corner Extraction: The segmented masks are blended back into the original image to provide "contextual focus." A second Mask R-CNN then performs panoptic segmentation to identify roof corners. Crucially, the researchers treat height-to-ground as a discrete classification task (defining 19 classes from 1m to 19m) rather than a continuous regression, which improves training stability and precision.
  3. Geometry Reconstruction: Once at least three corners of a roof section are identified with their respective heights, Delaunay triangulation is used to calculate 3D plane coefficients. The final mesh is generated using Kinetic Shape Reconstruction (KSR) to ensure clean, manifold surfaces.
The relationship between abstract LOD models and the real-world satellite imagery context used in the KIBS pipeline.

Experimental Setup and Results

The study utilized Maxar Worldview-3 satellite imagery (0.38m/pixel) covering two regions in France: Mourmelon-le-Grand (73 $km^2$) for training and Sissonne (25 $km^2$) for testing generalization.

The KIBS method demonstrated high efficiency and accuracy:

  • Segmentation Accuracy: Achieved a Jaccard index (IoU) of 88.55% on the primary dataset and 75.21% on the generalization dataset.
  • Height Precision: The mean error for roof heights was recorded at 1.60 meters and 2.06 meters, respectively.
  • Efficiency: The system can reconstruct entire urban areas in minutes, significantly faster than manual or procedural geometric approaches.

Discussion and Key Insights

A core insight of this work is that obliquity is an advantage. Tilted satellite views provide essential visual cues—such as wall inclination, shadows, and ridge perspectives—that the deep learning model exploits to infer height from a single 2D frame. Additionally, the "blending strategy" prevents the second stage of the model from being overwhelmed by the density of points in raw satellite data.

Despite its success, the model has limitations. It is sensitive to specific satellite viewing and solar angles, and the current height classification is capped at 19 meters, making it unsuitable for skyscrapers. Future work will focus on developing modular learning suites to handle varying viewing angles and moving toward a fully end-to-end neural architecture.

Summary updated

Abstract

Reconstructing urban areas in 3D out of satellite raster images has been a long-standing and challenging goal of both academical and industrial research. The rare methods today achieving this objective at a Level Of Details $2$ rely on procedural approaches based on geometry, and need stereo images and/or LIDAR data as input. We here propose a method for urban 3D reconstruction named KIBS(\textit{Keypoints Inference By Segmentation}), which comprises two novel features: i) a full deep learning approach for the 3D detection of the roof sections, and ii) only one single (non-orthogonal) satellite raster image as model input. This is achieved in two steps: i) by a Mask R-CNN model performing a 2D segmentation of the buildings' roof sections, and after blending these latter segmented pixels within the RGB satellite raster image, ii) by another identical Mask R-CNN model inferring the heights-to-ground of the roof sections' corners via panoptic segmentation, unto full 3D reconstruction of the buildings and city. We demonstrate the potential of the KIBS method by reconstructing different urban areas in a few minutes, with a Jaccard index for the 2D segmentation of individual roof sections of $88.55\%$ and $75.21\%$ on our two data sets resp., and a height's mean error of such correctly segmented pixels for the 3D reconstruction of $1.60$ m and $2.06$ m on our two data sets resp., hence within the LOD2 precision range.
BibTeX
@misc{lussange2023d,
  author = {Johann Lussange and Mulin Yu and Yuliya Tarabalka and Florent Lafarge},
  title = {{3D detection of roof sections from a single satellite image and application to LOD2-building reconstruction}},
  year = {2023},
  eprint = {2307.05409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.05409},
}
Figure