RSBuilding: Towards General Remote Sensing Image Building Extraction and Change Detection with Foundation Model

Mingze Wang, Keyan Chen, Lili Su, Cilin Yan, Sheng Xu, Haotian Zhang, Pengcheng Yuan, Xiaolong Jiang, Baochang Zhang

Introduction

Building understanding in remote sensing typically involves two distinct tasks: building extraction (segmenting footprints from a single image) and change detection (identifying structural changes between temporal image pairs). Despite their shared focus on buildings, these tasks are often treated independently, leading to missed semantic synergies and poor generalization across different geographical regions.

RSBuilding is a unified foundation model designed to bridge this gap. By integrating both tasks into a single framework and training on a massive dataset of 245,000 images, RSBuilding achieves state-of-the-art performance and demonstrates remarkable zero-shot generalization capabilities.

The RSBuilding framework conceptualizes building extraction and change detection as a unified task, processing temporal image pairs to generate multiple semantic masks.

Methodology

RSBuilding utilizes a Siamese (twin-stream) architecture to process image pairs ($Img_1, Img_2$) captured at different times. The model is built upon three core components:

  1. Siamese Encoder: Employs high-capacity backbones such as Vision Transformers (ViT) or Swin Transformers to extract robust features from input imagery. These backbones are often initialized with weights from large-scale models like SAM (Segment Anything Model).
  2. Multi-level Feature Enhancer: To account for the significant variation in building scales—from small residential sheds to large industrial complexes—this module constructs a feature pyramid to enrich multi-scale information.
  3. Cross-Attention Decoder with Task Prompts: This is the model's primary innovation. Instead of using separate decoders for each task, RSBuilding employs a unified decoder that accepts learnable "Task Prompts." These prompts (e.g., Extraction or Change Detection) query the image features to produce specific masks ($m_1, m_2, m_{cd}$).
The RSBuilding architecture consists of a Siamese backbone and neck for feature extraction, followed by a prompt-driven decoder and task-specific heads.

Federated Training Strategy

A significant challenge in multi-task remote sensing is the lack of dual-labeled datasets. Most benchmarks provide either building footprints or change masks, but rarely both. RSBuilding addresses this through a federated training strategy that allows the model to learn from incomplete data, ensuring that the extraction and change detection branches provide mutual support rather than interference.

Datasets and Generalization

The model was pre-trained on the RSBuilding Dataset, a curated collection of 245,000 images spanning various resolutions and geographies. This dataset aggregates several well-known benchmarks, including WHU, INRIA, and Map Challenge for extraction, and LEVIR-CD and S2Looking for change detection. Dual-label datasets like BANDON and xView2 were also included to facilitate joint learning.

This large-scale pre-training allows RSBuilding to function as a true foundation model. It can be applied to entirely new datasets, such as WHU-EAST ASIA, in a zero-shot manner, often outperforming specialized models that were fully supervised on that specific data.

Key Results

RSBuilding establishes new benchmarks across multiple tasks:

  • Building Extraction: Achieved a 92.15% IoU on the WHU dataset, surpassing specialized models like BuildFormer.
  • Change Detection: Reached an 86.19% IoU on the LEVIR-CD dataset.
  • Multi-Task Performance: On the BANDON dataset, which requires simultaneous task execution, the model achieved 80.55% IoU for extraction and 58.68% for change detection.
  • Zero-Shot Generalization: In tests on the unseen SECOND dataset, the model achieved a high IoU of 58.96% without any fine-tuning.

Conclusion and Future Work

RSBuilding demonstrates that unifying building extraction and change detection leads to more robust semantic representations. By leveraging task prompts and large-scale data, it provides a flexible and generalizable solution for remote sensing analysis.

Future research will focus on improving computational efficiency for single-image tasks, exploring dynamic input mechanisms for longer time series, and optimizing large Transformer backbones for faster inference on standard hardware.

Summary updated

Abstract

The intelligent interpretation of buildings plays a significant role in urban planning and management, macroeconomic analysis, population dynamics, etc. Remote sensing image building interpretation primarily encompasses building extraction and change detection. However, current methodologies often treat these two tasks as separate entities, thereby failing to leverage shared knowledge. Moreover, the complexity and diversity of remote sensing image scenes pose additional challenges, as most algorithms are designed to model individual small datasets, thus lacking cross-scene generalization. In this paper, we propose a comprehensive remote sensing image building understanding model, termed RSBuilding, developed from the perspective of the foundation model. RSBuilding is designed to enhance cross-scene generalization and task universality. Specifically, we extract image features based on the prior knowledge of the foundation model and devise a multi-level feature sampler to augment scale information. To unify task representation and integrate image spatiotemporal clues, we introduce a cross-attention decoder with task prompts. Addressing the current shortage of datasets that incorporate annotations for both tasks, we have developed a federated training strategy to facilitate smooth model convergence even when supervision for some tasks is missing, thereby bolstering the complementarity of different tasks. Our model was trained on a dataset comprising up to 245,000 images and validated on multiple building extraction and change detection datasets. The experimental results substantiate that RSBuilding can concurrently handle two structurally distinct tasks and exhibits robust zero-shot generalization capabilities.
BibTeX
@article{wang2024rsbuilding,
  author = {Mingze Wang and Keyan Chen and Lili Su and Cilin Yan and Sheng Xu and Haotian Zhang and Pengcheng Yuan and Xiaolong Jiang and Baochang Zhang},
  title = {{RSBuilding: Towards General Remote Sensing Image Building Extraction and Change Detection with Foundation Model}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  doi = {10.1109/tgrs.2024.3439395},
}
Figure