Method of Building Detection in Optical Remote Sensing Images Based on SegFormer

Meilin Li, Jie Rui, Songkun Yang, Zhi Liu, Liqiu Ren, Li Ma, Qing Li, Xu Su, Xibing ZuoSensors 2023 · DOI

TL;DR

This paper introduces an improved SegFormer-based method for detecting buildings in remote sensing images. By incorporating a transposed convolutional sampling module and an atrous spatial pyramid pooling (ASPP) decoding module, the approach enhances feature extraction and fusion, leading to more accurate building detection with better edge integrity and fewer false positives/negatives.

What Problem Does This Paper Address?

Extracting building information from remote sensing images is crucial for urban planning, disaster monitoring, and environmental management. However, this task is challenging due to the complex nature of building features, variations in scale, imaging conditions, and the presence of similar-looking non-building objects. Traditional image processing methods are often time-consuming and labor-intensive, while existing deep learning models struggle with issues like lost semantic and detailed features, poor edge integrity, and a high rate of missing or false detections, especially in intricate environments or for small buildings. This paper aims to address these limitations by proposing an enhanced SegFormer architecture specifically tailored for robust and accurate building detection in remote sensing imagery.

Methods

The core of this work lies in modifying the SegFormer architecture to improve its building detection capabilities. The authors introduce two main enhancements:

  • Transposed Convolutional Sampling Module: This module is integrated into the decoder. It uses transposed convolutions to fuse multi-scale semantic and detailed features from different layers. This helps to recover lost information, mitigate the impact of spatial heterogeneity and background clutter, and reinforce the network's ability to capture building features. Multiple normalization and activation layers are also added to stabilize training and improve feature representation.
  • Atrous Spatial Pyramid Pooling (ASPP) Decoding Module: This module is fused with the decoder to capture multi-scale contextual information. It employs atrous convolutions with different dilation rates to extract features at various receptive field sizes, addressing the loss of detail in local buildings and long-distance contextual information. This module helps in obtaining clearer building edge information, particularly for buildings affected by inclination or shadows.

The authors also mention the use of a "Semantic-Guided Fusion Module" which fuses shallow-layer details with deep-layer semantic information, and a "shuffle" operation for feature channel interaction, to further enhance feature representation and fusion across different resolutions.

Datasets

The proposed method was trained and evaluated on three publicly available remote sensing image datasets:

  • Aerial Image Segmentation Dataset (AISD): Contains building, road, water, and vegetation labels. The study specifically uses the building labels. It covers areas like Berlin, Chicago, Paris, Potsdam, and Zurich with a resolution of 0.3 m.
  • Massachusetts Buildings Dataset (MBD): Focuses on buildings in Boston and its suburbs, with a resolution of 1 m.
  • Wuhan University (WHU) Aerial Remote Sensing Image Dataset: Includes global urban satellite images with a resolution of 0.3 m.

The paper provides a detailed breakdown of the train, validation, and test splits for each dataset, along with their respective resolutions.

Key Insights and Ideas

  • The power of combining architectures: The paper demonstrates that by strategically combining the strengths of Transformer-based models (SegFormer) with convolutional techniques (transposed convolution, ASPP), significant improvements in semantic segmentation tasks like building detection can be achieved.
  • Multi-scale context is king for buildings: Buildings, especially in remote sensing, exhibit a wide range of scales and contextual relationships. The ASPP module's ability to capture features at multiple scales is crucial for accurately identifying both large structures and small, intricate buildings.
  • Transposed convolution for feature recovery: The authors highlight how transposed convolutions can effectively "upsample" and fuse features, recovering information that might be lost in standard downsampling operations, which is vital for preserving building details and edges.
  • Normalization and activation layers are not just for regularization: Beyond preventing overfitting, the strategic placement of these layers within the transposed convolutional framework appears to enhance feature expression and network stability, leading to better detection.

Key Results

The proposed method achieved significant improvements across all tested datasets:

  • mIoU improvements: The method enhanced the mean Intersection over Union (mIoU) by 17.68% on AISD, 30.44% on MBD, and 15.26% on WHU compared to the original SegFormer.
  • Overall superior performance: In comparison experiments against HRnet, PSPNet, U-Net, and DeepLabv3+, the proposed method consistently outperformed them, achieving the highest mIoU, F1 score, precision, and recall on all three datasets.
  • Qualitative improvements: Visualizations showed that the improved network provides better edge integrity, more regular boundary transitions, and a reduced number of missing and false detections, particularly for tiny buildings. It also demonstrates stronger robustness in complex environments.
  • Efficiency: While achieving superior results, the proposed method's computational complexity (GFLOPS, parameters, and computation time) increased only slightly compared to the original SegFormer, making it a cost-effective solution.

Limitations and Future Work

  • Spectral Imaging Interference: The authors acknowledge that spectral similarity in remote sensing images can still interfere with detection. They propose exploring building detection methods that can suppress this spectral imaging interference in future work.
  • Generalization to other domains: While the method shows strong performance on the tested datasets, its generalization to even more diverse remote sensing scenarios (e.g., different sensor types, extreme weather conditions) could be further investigated.

Summary updated

Abstract

An appropriate detection network is required to extract building information in remote sensing images and to relieve the issue of poor detection effects resulting from the deficiency of detailed features. Firstly, we embed a transposed convolution sampling module fusing multiple normalization activation layers in the decoder based on the SegFormer network. This step alleviates the issue of missing feature semantics by adding holes and fillings, cascading multiple normalizations and activation layers to hold back over-fitting regularization expression and guarantee steady feature parameter classification. Secondly, the atrous spatial pyramid pooling decoding module is fused to explore multi-scale contextual information and to overcome issues such as the loss of detailed information on local buildings and the lack of long-distance information. Ablation experiments and comparison experiments are performed on the remote sensing image AISD, MBD, and WHU dataset. The robustness and validity of the improved mechanism are demonstrated by control groups of ablation experiments. In comparative experiments with the HRnet, PSPNet, U-Net, DeepLabv3+ networks, and the original detection algorithm, the mIoU of the AISD, the MBD, and the WHU dataset is enhanced by 17.68%, 30.44%, and 15.26%, respectively. The results of the experiments show that the method of this paper is superior to comparative methods such as U-Net. Furthermore, it is better for integrity detection of building edges and reduces the number of missing and false detections.
BibTeX
@inproceedings{li2023method,
  author = {Meilin Li and Jie Rui and Songkun Yang and Zhi Liu and Liqiu Ren and Li Ma and Qing Li and Xu Su and Xibing Zuo},
  title = {{Method of Building Detection in Optical Remote Sensing Images Based on SegFormer}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23031258},
}
Figure