Abstract
In this work, a novel automatic 3D building reconstruction approach is proposed to extract accurate LoD1 building models from multi-view aerial images. The proposed approach consists of three main parts. The first step is to generate digital surface models (DSMs) from aerial images, which is implemented with the Smart3D software and can be replaced by other open-source multi-view stereo (MVS) algorithms as well. The second step is to produce structured 2D building footprints using combined deep learning and regularization. The initial building segmentation maps are obtained by the multi-scale aggregation fully convolutional network (MA-FCN), which takes both the images and DSM as input, through supervised learning. The initial segmentation maps are further refined with another segmentation maps that are derived from the DSM. After that, the contour extraction and regularization technology are applied to produce structured building footprints. In the last step, the elevations of the top and base of each individual building are reliably estimated by adopting an adaptive terrain generation approach and the neighbourhood buffer analysis. The georeferenced building footprint polygons and the elevations of building top and base form the watertight LoD1 building models. The qualitative and quantitative evaluations in Jinghai District, Tianjin, China demonstrate the robustness and effectiveness of the proposed approach.
Introduction
Creating accurate 3D city models is a cornerstone for modern urban planning, disaster management, and the development of "smart cities." Traditionally, this process has been a significant bottleneck, relying heavily on expensive LiDAR data or potentially outdated cadastral maps. While aerial imagery is abundant, reconstructing 3D shapes from raw pixels is challenging due to complex urban backgrounds, shadows, and vegetation that can mimic building structures. This paper introduces a fully automated pipeline designed to bridge this gap, producing "watertight" Level of Detail 1 (LoD1) 3D polygons directly from multi-view aerial images.
Methodology
The proposed workflow integrates deep learning with classical geometric processing across three major stages:
- DSM Generation: Multi-View Stereo (MVS) algorithms are employed to generate a Digital Surface Model (DSM) from five-view oblique aerial images, providing the initial 3D representation of the scene.
- Structured 2D Footprint Extraction: The core of the segmentation process is the Multi-scale Aggregation Fully Convolutional Network (MA-FCN). Unlike standard approaches, this network utilizes the DSM as a fourth input channel alongside RGB data. This multimodal fusion allows the network to distinguish between features that look similar in 2D (like grey roofs and grey roads) but differ significantly in height. The output is further refined using terrain filtering and regularized via the Douglas-Peucker algorithm to ensure clean, straight-edged polygons.
- Height Estimation: To extrude 2D footprints into 3D blocks, the system uses an Adaptive Network of Ground Points (NGPs) and neighborhood buffer analysis. By fitting a local plane to the terrain immediately surrounding each building, the method accurately estimates base and top elevations even on sloped terrain.
Results and Discussion
The system was evaluated in the Jinghai District of Tianjin, China, covering a 3.96 km² area with a mix of residential and industrial structures. The integration of DSM data into the MA-FCN (MA-FCN+) significantly improved performance, boosting the average Intersection over Union (IoU) from 0.55 to 0.70. Further refinement (MA-FCN++) reached an IoU of 0.74.
In terms of vertical accuracy, the method achieved a Mean Absolute Error (MAE) of 0.31 meters, outperforming traditional Multi-directional Slope Dependent (MSD) methods. The resulting models are robust enough to maintain functionality even under rigorous 3D IoU thresholds, proving their utility for automated high-fidelity modeling.
Conclusion
This research demonstrates that high-quality 3D building reconstruction is possible without specialized LiDAR equipment. By treating elevation data as an input to the segmentation process rather than just an output, the researchers achieved superior accuracy in complex urban environments. While the current system focuses on LoD1 (flat-roof) models and faces some challenges with extreme vegetation or shadows, it provides a robust foundation for future work toward LoD2 reconstruction, which will involve modeling individual roof planes.
Summary updated