Introduction
Building understanding in remote sensing typically involves two distinct tasks: building extraction (segmenting footprints from a single image) and change detection (identifying structural changes between temporal image pairs). Despite their shared focus on buildings, these tasks are often treated independently, leading to missed semantic synergies and poor generalization across different geographical regions.
RSBuilding is a unified foundation model designed to bridge this gap. By integrating both tasks into a single framework and training on a massive dataset of 245,000 images, RSBuilding achieves state-of-the-art performance and demonstrates remarkable zero-shot generalization capabilities.
Methodology
RSBuilding utilizes a Siamese (twin-stream) architecture to process image pairs ($Img_1, Img_2$) captured at different times. The model is built upon three core components:
- Siamese Encoder: Employs high-capacity backbones such as Vision Transformers (ViT) or Swin Transformers to extract robust features from input imagery. These backbones are often initialized with weights from large-scale models like SAM (Segment Anything Model).
- Multi-level Feature Enhancer: To account for the significant variation in building scales—from small residential sheds to large industrial complexes—this module constructs a feature pyramid to enrich multi-scale information.
- Cross-Attention Decoder with Task Prompts: This is the model's primary innovation. Instead of using separate decoders for each task, RSBuilding employs a unified decoder that accepts learnable "Task Prompts." These prompts (e.g., Extraction or Change Detection) query the image features to produce specific masks ($m_1, m_2, m_{cd}$).
Federated Training Strategy
A significant challenge in multi-task remote sensing is the lack of dual-labeled datasets. Most benchmarks provide either building footprints or change masks, but rarely both. RSBuilding addresses this through a federated training strategy that allows the model to learn from incomplete data, ensuring that the extraction and change detection branches provide mutual support rather than interference.
Datasets and Generalization
The model was pre-trained on the RSBuilding Dataset, a curated collection of 245,000 images spanning various resolutions and geographies. This dataset aggregates several well-known benchmarks, including WHU, INRIA, and Map Challenge for extraction, and LEVIR-CD and S2Looking for change detection. Dual-label datasets like BANDON and xView2 were also included to facilitate joint learning.
This large-scale pre-training allows RSBuilding to function as a true foundation model. It can be applied to entirely new datasets, such as WHU-EAST ASIA, in a zero-shot manner, often outperforming specialized models that were fully supervised on that specific data.
Key Results
RSBuilding establishes new benchmarks across multiple tasks:
- Building Extraction: Achieved a 92.15% IoU on the WHU dataset, surpassing specialized models like BuildFormer.
- Change Detection: Reached an 86.19% IoU on the LEVIR-CD dataset.
- Multi-Task Performance: On the BANDON dataset, which requires simultaneous task execution, the model achieved 80.55% IoU for extraction and 58.68% for change detection.
- Zero-Shot Generalization: In tests on the unseen SECOND dataset, the model achieved a high IoU of 58.96% without any fine-tuning.
Conclusion and Future Work
RSBuilding demonstrates that unifying building extraction and change detection leads to more robust semantic representations. By leveraging task prompts and large-scale data, it provides a flexible and generalizable solution for remote sensing analysis.
Future research will focus on improving computational efficiency for single-image tasks, exploring dynamic input mechanisms for longer time series, and optimizing large Transformer backbones for faster inference on standard hardware.
Summary updated