TL;DR
This paper introduces HTC-DC Net, a novel deep learning architecture for estimating building heights from single remote sensing images. It addresses the challenge of long-tailed height distributions and the ill-posed nature of monocular height estimation by combining a classification-regression approach with a "Head-Tail Cut" mechanism and distribution-based constraints.
What Problem Does This Paper Address?
Accurate 3D geo-information is crucial for various applications, but obtaining it from remote sensing data, especially at scale, is challenging due to the high cost of 3D sensors like LiDAR. While optical imagery is abundant, extracting precise height information from a single image (monocular height estimation) is an ill-posed problem. Existing methods often struggle with the inherent long-tailed distribution of height values in real-world data, where low-height objects (background) are overwhelmingly common, leading to biased models that underestimate building heights. This paper aims to improve the accuracy and robustness of monocular height estimation, particularly for buildings, by developing a more effective deep learning approach.
Methods
The proposed HTC-DC Net employs a classification-regression paradigm. It consists of three main components:
- Backbone Network: A standard encoder-decoder architecture (U-Net or EfficientNet) is used to extract multi-level features from the input image.
- HTC-AdaBins Module: This is the core innovation, adapting the AdaBins module for monocular height estimation. It incorporates:
- Local and Global Branches: A convolutional local branch captures local features, while a Vision Transformer (ViT) encoder in the global branch captures long-range dependencies.
- Head-Tail Cut (HTC): This mechanism explicitly separates foreground (buildings, height > 1m) and background pixels. This is crucial for addressing the long-tailed distribution, allowing for different treatments and preventing the majority background from negatively impacting foreground predictions.
- Adaptive Bins: The module predicts adaptive bin edges and probabilities for height discretization, tailored to each input image.
- Hybrid Regression Process: This module smooths the discrete bin predictions into a continuous height map. It uses a weighted average of bin centers, guided by the predicted bin probabilities.
- Distribution-Based Constraints (DCs): These constraints are applied to the predicted bin probabilities during training. They regularize the probabilities by assuming underlying distributions (e.g., Gaussian for foreground, uniform for background) and encouraging the predicted distribution to align with these assumptions, leading to more accurate height predictions.
Datasets
The paper evaluates HTC-DC Net on three datasets:
- DFC19: A dataset with multi-date satellite images and ground truth DSMs and semantic maps, featuring a Ground Sampling Distance (GSD) of 1.3m.
- GBH (Global Building Height): A newly proposed dataset comprising optical remote sensing images, nDSMs, and building footprint maps, with a GSD of 3m. This dataset was introduced to address the lack of diversity in existing benchmarks.
- ISPRS Vaihingen: A dataset with aerial orthophotos, nDSMs from LiDAR, and semantic labels, featuring a GSD of 0.09m.
Key Insights and Ideas
- Head-Tail Cut is Crucial for Long-Tailed Data: Explicitly separating foreground (buildings) and background pixels is highly effective in mitigating the negative impact of the dominant background class on building height estimation. This is a key insight for handling the inherent imbalance in remote sensing height data.
- Classification-Regression is Superior to Pure Regression: Shifting from a direct regression approach to a classification-regression paradigm, where height is first discretized into bins and then regressed, significantly improves performance.
- Adaptive Bins and Distribution Constraints Enhance Accuracy: Tailoring bin discretizations to individual images and applying distribution-based constraints during training help to regularize predictions and improve accuracy, especially for challenging cases.
- Multi-level Feature Fusion Benefits Building Height: Utilizing features from multiple levels of the backbone network, particularly for building-related metrics, leads to better performance.
Key Results
HTC-DC Net consistently outperforms existing state-of-the-art methods across all tested datasets (DFC19, GBH, and ISPRS Vaihingen). For instance, on the DFC19 dataset, HTC-DC Net B7 achieved the best RMSE, outperforming competitors by significant margins. On the GBH dataset, HTC-DC Net B7 also showed superior performance compared to U-Net, the best among competitors. Qualitative results demonstrate that HTC-DC Net produces sharper building boundaries and more accurate height predictions, especially for taller buildings, compared to other methods that often produce blurred or underestimated heights.
Limitations and Future Work
The authors acknowledge that domain shifts between different cities remain a challenge for large-scale applications, leading to performance drops in cities with distinct urban morphologies. They suggest that future work could focus on addressing these domain shifts through domain generalization techniques.
Summary updated