3D Building Reconstruction from Monocular Remote Sensing Images

Weijia Li, Lingxuan Meng, Jinwang Wang, Conghui He, Gui-Song Xia, Dahua Lin

Abstract

Accurate 3D building reconstruction from a single (monocular) remote sensing image is a significant challenge in urban planning and geospatial analysis. Traditional methods often rely on expensive LiDAR data or multi-view stereo imagery. This paper presents MTBR-Net, a multi-task deep learning framework designed to reconstruct 3D vector models from monocular images. By decomposing the problem into semantic and offset-related sub-tasks, the proposed method achieves a 40% improvement in height estimation accuracy compared to previous state-of-the-art techniques.

The Challenge of Monocular 3D Reconstruction

Creating 3D city models typically requires high-cost aerial LiDAR or multi-view imagery. While monocular satellite or drone images are more accessible, converting them into 3D models is difficult due to several factors:

  1. Partial Invisibility: In oblique shots, building footprints are often obscured by the building's own structure.
  2. Shadows and Occlusions: Tall structures cast shadows and block the view of adjacent smaller buildings.
  3. Scale Variance: The vast range of building heights makes direct scalar height regression difficult for standard neural networks.

Existing monocular methods often struggle with complex, high-rise urban environments captured from varying oblique angles. MTBR-Net aims to overcome these limitations by treating 3D reconstruction as a multi-task learning problem.

The MTBR-Net Framework

The researchers proposed a two-stage approach consisting of the MTBR-Net multi-task network and a subsequent 3D model optimization module.

1. Multi-Task Learning (MTBR-Net)

Instead of regressing a single height value, the network handles seven concurrent sub-tasks:

  • Semantic Tasks: Predicting roof/facade segments, footprints, building "skeletons" (edges), and edge orientations.
  • Offset Tasks: Predicting the "offset vector"—the 2D displacement between a roof's image position and its ground footprint. This vector serves as a robust proxy for height.
The MTBR-Net architecture and optimization pipeline.

A critical innovation is the Offset-based Feature Warping Module. This module uses predicted offsets to "warp" roof features to the footprint's location, enabling the network to accurately predict hidden ground boundaries.

2. 3D Model Optimization

To convert pixel-level (raster) predictions into usable geometric (vector) shapes, the framework employs a template-matching approach for height refinement and a polygonization method to generate clean 3D boxes.

Data and Evaluation

To address the noise and limited viewing angles of existing datasets like DFC19, the researchers introduced a new large-scale dataset featuring over 200,000 annotated buildings across major Chinese cities. This dataset includes diverse oblique angles and manually labeled roofs, facades, and footprints, providing a high-quality ground truth for training.

Key Results and Insights

The MTBR-Net framework demonstrates significant performance gains:

  • Height Accuracy: On the DFC19 dataset, the Root Mean Square Error (RMSE) for height estimation dropped from 16.87m to 9.57m.
  • Segmentation Quality: The F1-score for identifying building components improved by 2% to 4% over previous leaders like HR-Net.
  • Robustness: The model generalizes well to unseen cities, indicating it has learned fundamental geometric principles rather than specific urban layouts.
Qualitative results showing monocular input images and the corresponding 3D-aware segmentation masks.

A key insight from this work is that predicting height as a vector (offset field) is more effective than scalar regression. This approach captures both height and perspective simultaneously, which is more compatible with the way convolutional neural networks process image textures.

Limitations and Future Work

Despite its success, MTBR-Net faces challenges in ultra-dense urban areas where buildings are touching. Additionally, the current model assumes relatively simple roof structures and may struggle with complex residential designs like gables or hips. Future research will focus on incorporating architectural priors and more sophisticated structural constraints to further refine the reconstruction of complex urban environments.

Summary updated

Abstract

3D building reconstruction from monocular remote sensing imagery is an important research problem and an economic solution to large-scale city modeling, compared with reconstruction from LiDAR data and multi-view imagery. However, several challenges such as the partial invisibility of building footprints and facades, the serious shadow effect, and the extreme variance of building height in large-scale areas, have restricted the existing monocular image based building reconstruction studies to certain application scenes, i.e., modeling simple low-rise buildings from near-nadir images. In this study, we propose a novel 3D building reconstruction method for monocular remote sensing images, which tackles the above difficulties, thus providing an appealing solution for more complicated scenarios. We design a multi-task building reconstruction network, named MTBR-Net, to learn the geometric property of oblique images, the key components of a 3D building model and their relations via four semantic-related and three offset-related tasks. The network outputs are further integrated by a prior knowledge based 3D model optimization method to produce the the final 3D building models. Results on a public 3D reconstruction dataset and a novel released dataset demonstrate that our method improves the height estimation performance by over 40% and the segmentation F1-score by 2% - 4% compared with current state-of-the-art.
BibTeX
@inproceedings{li2021d,
  author = {Weijia Li and Lingxuan Meng and Jinwang Wang and Conghui He and Gui-Song Xia and Dahua Lin},
  title = {{3D Building Reconstruction from Monocular Remote Sensing Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01232},
}
Figure