HTC-DC Net: Monocular Height Estimation From Single Remote Sensing Images

Sining Chen, Yilei Shi, Zhitong Xiong, Xiao Xiang Zhu

TL;DR

This paper introduces HTC-DC Net, a novel deep learning architecture for estimating building heights from single remote sensing images. It addresses the challenge of long-tailed height distributions and the ill-posed nature of monocular height estimation by combining a classification-regression approach with a "Head-Tail Cut" mechanism and distribution-based constraints.

What Problem Does This Paper Address?

Accurate 3D geo-information is crucial for various applications, but obtaining it from remote sensing data, especially at scale, is challenging due to the high cost of 3D sensors like LiDAR. While optical imagery is abundant, extracting precise height information from a single image (monocular height estimation) is an ill-posed problem. Existing methods often struggle with the inherent long-tailed distribution of height values in real-world data, where low-height objects (background) are overwhelmingly common, leading to biased models that underestimate building heights. This paper aims to improve the accuracy and robustness of monocular height estimation, particularly for buildings, by developing a more effective deep learning approach.

Methods

The proposed HTC-DC Net employs a classification-regression paradigm. It consists of three main components:

  • Backbone Network: A standard encoder-decoder architecture (U-Net or EfficientNet) is used to extract multi-level features from the input image.
  • HTC-AdaBins Module: This is the core innovation, adapting the AdaBins module for monocular height estimation. It incorporates:
  • Local and Global Branches: A convolutional local branch captures local features, while a Vision Transformer (ViT) encoder in the global branch captures long-range dependencies.
  • Head-Tail Cut (HTC): This mechanism explicitly separates foreground (buildings, height > 1m) and background pixels. This is crucial for addressing the long-tailed distribution, allowing for different treatments and preventing the majority background from negatively impacting foreground predictions.
  • Adaptive Bins: The module predicts adaptive bin edges and probabilities for height discretization, tailored to each input image.
  • Hybrid Regression Process: This module smooths the discrete bin predictions into a continuous height map. It uses a weighted average of bin centers, guided by the predicted bin probabilities.
  • Distribution-Based Constraints (DCs): These constraints are applied to the predicted bin probabilities during training. They regularize the probabilities by assuming underlying distributions (e.g., Gaussian for foreground, uniform for background) and encouraging the predicted distribution to align with these assumptions, leading to more accurate height predictions.

Datasets

The paper evaluates HTC-DC Net on three datasets:

  • DFC19: A dataset with multi-date satellite images and ground truth DSMs and semantic maps, featuring a Ground Sampling Distance (GSD) of 1.3m.
  • GBH (Global Building Height): A newly proposed dataset comprising optical remote sensing images, nDSMs, and building footprint maps, with a GSD of 3m. This dataset was introduced to address the lack of diversity in existing benchmarks.
  • ISPRS Vaihingen: A dataset with aerial orthophotos, nDSMs from LiDAR, and semantic labels, featuring a GSD of 0.09m.

Key Insights and Ideas

  • Head-Tail Cut is Crucial for Long-Tailed Data: Explicitly separating foreground (buildings) and background pixels is highly effective in mitigating the negative impact of the dominant background class on building height estimation. This is a key insight for handling the inherent imbalance in remote sensing height data.
  • Classification-Regression is Superior to Pure Regression: Shifting from a direct regression approach to a classification-regression paradigm, where height is first discretized into bins and then regressed, significantly improves performance.
  • Adaptive Bins and Distribution Constraints Enhance Accuracy: Tailoring bin discretizations to individual images and applying distribution-based constraints during training help to regularize predictions and improve accuracy, especially for challenging cases.
  • Multi-level Feature Fusion Benefits Building Height: Utilizing features from multiple levels of the backbone network, particularly for building-related metrics, leads to better performance.

Key Results

HTC-DC Net consistently outperforms existing state-of-the-art methods across all tested datasets (DFC19, GBH, and ISPRS Vaihingen). For instance, on the DFC19 dataset, HTC-DC Net B7 achieved the best RMSE, outperforming competitors by significant margins. On the GBH dataset, HTC-DC Net B7 also showed superior performance compared to U-Net, the best among competitors. Qualitative results demonstrate that HTC-DC Net produces sharper building boundaries and more accurate height predictions, especially for taller buildings, compared to other methods that often produce blurred or underestimated heights.

Limitations and Future Work

The authors acknowledge that domain shifts between different cities remain a challenge for large-scale applications, leading to performance drops in cities with distinct urban morphologies. They suggest that future work could focus on addressing these domain shifts through domain generalization techniques.

Summary updated

Abstract

3D geo-information is of great significance for understanding the living environment; however, 3D perception from remote sensing data, especially on a large scale, is restricted. To tackle this problem, we propose a method for monocular height estimation from optical imagery, which is currently one of the richest sources of remote sensing data. As an ill-posed problem, monocular height estimation requires well-designed networks for enhanced representations to improve performance. Moreover, the distribution of height values is long-tailed with the low-height pixels, e.g., the background, as the head, and thus trained networks are usually biased and tend to underestimate building heights. To solve the problems, instead of formalizing the problem as a regression task, we propose HTC-DC Net following the classification-regression paradigm, with the head-tail cut (HTC) and the distribution-based constraints (DCs) as the main contributions. HTC-DC Net is composed of the backbone network as the feature extractor, the HTC-AdaBins module, and the hybrid regression process. The HTC-AdaBins module serves as the classification phase to determine bins adaptive to each input image. It is equipped with a vision transformer encoder to incorporate local context with holistic information and involves an HTC to address the long-tailed problem in monocular height estimation for balancing the performances of foreground and background pixels. The hybrid regression process does the regression via the smoothing of bins from the classification phase, which is trained via DCs. The proposed network is tested on three datasets of different resolutions, namely ISPRS Vaihingen (0.09 m), DFC19 (1.3 m) and GBH (3 m). Experimental results show the superiority of the proposed network over existing methods by large margins. Extensive ablation studies demonstrate the effectiveness of each design component.
BibTeX
@article{chen2023htc,
  author = {Sining Chen and Yilei Shi and Zhitong Xiong and Xiao Xiang Zhu},
  title = {{HTC-DC Net: Monocular Height Estimation From Single Remote Sensing Images}},
  journal = {IEEE TGRS},
  year = {2023},
  doi = {10.1109/tgrs.2023.3321255},
}
Figure