Abstract
Accurate 3D building reconstruction from a single (monocular) remote sensing image is a significant challenge in urban planning and geospatial analysis. Traditional methods often rely on expensive LiDAR data or multi-view stereo imagery. This paper presents MTBR-Net, a multi-task deep learning framework designed to reconstruct 3D vector models from monocular images. By decomposing the problem into semantic and offset-related sub-tasks, the proposed method achieves a 40% improvement in height estimation accuracy compared to previous state-of-the-art techniques.
The Challenge of Monocular 3D Reconstruction
Creating 3D city models typically requires high-cost aerial LiDAR or multi-view imagery. While monocular satellite or drone images are more accessible, converting them into 3D models is difficult due to several factors:
- Partial Invisibility: In oblique shots, building footprints are often obscured by the building's own structure.
- Shadows and Occlusions: Tall structures cast shadows and block the view of adjacent smaller buildings.
- Scale Variance: The vast range of building heights makes direct scalar height regression difficult for standard neural networks.
Existing monocular methods often struggle with complex, high-rise urban environments captured from varying oblique angles. MTBR-Net aims to overcome these limitations by treating 3D reconstruction as a multi-task learning problem.
The MTBR-Net Framework
The researchers proposed a two-stage approach consisting of the MTBR-Net multi-task network and a subsequent 3D model optimization module.
1. Multi-Task Learning (MTBR-Net)
Instead of regressing a single height value, the network handles seven concurrent sub-tasks:
- Semantic Tasks: Predicting roof/facade segments, footprints, building "skeletons" (edges), and edge orientations.
- Offset Tasks: Predicting the "offset vector"—the 2D displacement between a roof's image position and its ground footprint. This vector serves as a robust proxy for height.
A critical innovation is the Offset-based Feature Warping Module. This module uses predicted offsets to "warp" roof features to the footprint's location, enabling the network to accurately predict hidden ground boundaries.
2. 3D Model Optimization
To convert pixel-level (raster) predictions into usable geometric (vector) shapes, the framework employs a template-matching approach for height refinement and a polygonization method to generate clean 3D boxes.
Data and Evaluation
To address the noise and limited viewing angles of existing datasets like DFC19, the researchers introduced a new large-scale dataset featuring over 200,000 annotated buildings across major Chinese cities. This dataset includes diverse oblique angles and manually labeled roofs, facades, and footprints, providing a high-quality ground truth for training.
Key Results and Insights
The MTBR-Net framework demonstrates significant performance gains:
- Height Accuracy: On the DFC19 dataset, the Root Mean Square Error (RMSE) for height estimation dropped from 16.87m to 9.57m.
- Segmentation Quality: The F1-score for identifying building components improved by 2% to 4% over previous leaders like HR-Net.
- Robustness: The model generalizes well to unseen cities, indicating it has learned fundamental geometric principles rather than specific urban layouts.
A key insight from this work is that predicting height as a vector (offset field) is more effective than scalar regression. This approach captures both height and perspective simultaneously, which is more compatible with the way convolutional neural networks process image textures.
Limitations and Future Work
Despite its success, MTBR-Net faces challenges in ultra-dense urban areas where buildings are touching. Additionally, the current model assumes relatively simple roof structures and may struggle with complex residential designs like gables or hips. Future research will focus on incorporating architectural priors and more sophisticated structural constraints to further refine the reconstruction of complex urban environments.
Summary updated