Efficient Rural Building Segmentation via a Multilevel Decoding Network

Bowen Xu, Liang Dong, Gui-Song Xia, Liangpei ZhangIEEE JSTARS 2023 · DOI · OpenReview

Abstract

This article addresses the problem of building segmentation for rural areas with high-resolution remote sensing images. Due to the irregular spatial distribution of rural buildings, it is often challenging to perform pixel-wise dense prediction to entire areas like the usual segmentation task to extract buildings. Specifically, we present a multilevel decoding network model that classifies the input image on the patch and image levels according to the distribution of buildings. A scene head module is used to identify scenes defined as patches that contain buildings. Depending on the scene classification results, a decode gate is taken to determine the level of prediction. This hierarchical extraction strategy reduces the amount of inference time. Experiments on our constructed rural building dataset with large-scale images validate the high efficiency of the proposed method.

1. Introduction

Building extraction is a fundamental task in remote sensing, yet rural environments present unique challenges that standard urban-centric models often fail to address. Rural buildings are characterized by sparse distribution, scattered across vast landscapes where the majority of the image contains empty fields. This leads to a significant foreground-background imbalance, making it computationally wasteful to perform dense pixel-wise segmentation on massive images where buildings occupy less than 5% of the area. Furthermore, there is a notable lack of representative data, as most existing benchmarks focus on urban architectural styles.

2. Proposed Methodology

The proposed Multilevel Decoding Network adopts a "coarse-to-fine" hierarchical strategy to optimize both accuracy and efficiency. The architecture is built upon three core components:

  • Shared Encoder: Utilizes a ResNet-50 backbone with an Atrous Spatial Pyramid Pooling (ASPP) module to extract multiscale features.
  • Scene Head: This module performs coarse classification by dividing the input image into a grid and predicting which patches contain buildings.
  • Decode Gate & Multilevel Decoder: Acting as the system's control unit, the Decode Gate determines the level of prediction based on a threshold $\gamma$. If buildings are scattered, it triggers a full-image decoding path; if they are concentrated, it triggers a patch-level path, decoding only the relevant features and skipping empty background areas.
The architecture of the proposed Multilevel Decoding Network, featuring the Scene Head and Decode Gate.

This approach leverages the semantic connection between rough location (classification) and refined extraction (segmentation), allowing the model to prune unnecessary computations at the decoding stage.

3. Dataset Construction

To address the data gap, the researchers constructed a new Rural Building Dataset using 0.3-meter high-resolution RGB imagery from Google Earth. The dataset covers Qiaoshiying Town in Inner Mongolia and consists of 100 large-scale tiles (approximately 6400 x 6400 pixels each). The data was manually labeled and split into 60 images for training, 7 for validation, and 33 for testing to ensure a robust evaluation of rural architectural layouts.

4. Experimental Results

The proposed model was evaluated against industry standards including UNet, DeepLabV3, and ABCNet.

  • Accuracy: The model achieved state-of-the-art performance with an IoU of 84.99% and an F1-score of 91.89%, outperforming both DeepLabV3 (84.22% IoU) and UNet (82.49% IoU).
  • Efficiency: The model demonstrated superior speed, completing the test set inference in 465.9 seconds, compared to 554.0s for DeepLabV3. While the lightweight ABCNet was faster, its accuracy (56.26% IoU) was insufficient for reliable extraction.
Qualitative segmentation results comparing the proposed method with ground truth and other models.

Qualitative analysis shows that the Scene Head effectively filters out "false alarms" in empty fields, leading to cleaner segmentation maps with fewer false positives compared to standard dense prediction models.

5. Conclusion and Future Work

The Multilevel Decoding Network successfully balances the trade-off between efficiency and accuracy in rural building extraction. While highly effective, future work will focus on adaptive grid division to handle varying scene complexities and domain adaptation techniques to ensure the model's generalizability across different global rural architectural styles. Additionally, addressing the occasional omission of very small, isolated structures remains a priority for further refinement.

Summary updated

Abstract

This article addresses the problem of building segmentation for rural areas with high-resolution remote sensing images. Due to the irregular spatial distribution of rural buildings, it is often challenging to perform pixel-wise dense prediction to entire areas like the usual segmentation task to extract buildings. Specifically, we present a multilevel decoding network model that classifies the input image on the patch and image levels according to the distribution of buildings. A scene head module is used to identify scenes defined as patches that contain buildings. Depending on the scene classification results, a decode gate is taken to determine the level of prediction. This hierarchical extraction strategy reduces the amount of inference time. Experiments on our constructed rural building dataset with large-scale images validate the high efficiency of the proposed method.
BibTeX
@article{xu2023efficient,
  author = {Bowen Xu and Liang Dong and Gui-Song Xia and Liangpei Zhang},
  title = {{Efficient Rural Building Segmentation via a Multilevel Decoding Network}},
  journal = {IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing},
  year = {2023},
  doi = {10.1109/jstars.2023.3344210},
}
Figure