Self-supervised Learning for Building Damage Assessment from Large-Scale xBD Satellite Imagery Benchmark Datasets

Zaishuo Xia, Zelin Li, Yanbing Bai, Jinze Yu, Bruno Adriano

Abstract

Rapid building damage assessment is critical for effective disaster response, yet traditional deep learning models are hindered by a heavy reliance on large-scale, human-labeled datasets. This paper presents a self-supervised learning (SSL) framework designed to bridge the label gap in disaster scenarios. By integrating global contrastive learning with local image reconstruction in an asymmetric twin network, the proposed model learns robust features from unlabeled satellite imagery. Experimental results on the xBD benchmark demonstrate that this approach significantly outperforms standard ImageNet pre-training, particularly in data-scarce environments, achieving an F1-score of 0.390 with only 1% of available labels.

Introduction

When natural disasters such as hurricanes, earthquakes, or floods occur, emergency responders require immediate, actionable intelligence to prioritize rescue and recovery efforts. Satellite imagery provides a high-level view of affected regions, but the manual assessment of building damage is time-consuming and labor-intensive. While deep learning models offer a path toward automation, they typically suffer from a "data hunger" problem, requiring thousands of expertly labeled pre- and post-disaster image pairs.

In the immediate aftermath of a new disaster, such labels are rarely available. Furthermore, existing models often rely on pre-training from generic datasets like ImageNet, which lack the specific "top-down" perspective and semantic nuances of remote sensing data. This research addresses these challenges by developing a framework capable of learning from unlabeled satellite data, thereby reducing the dependency on manual annotations.

Methodology

The researchers propose a two-stage framework that shifts away from traditional supervised paradigms toward a more flexible, self-supervised approach.

Stage 1: Self-Supervised Pre-training

The core of the framework is an asymmetric twin network architecture designed to learn features from unlabeled images through two parallel tasks:

  1. Global Contrastive Learning: Utilizing the DINO (Self-Distillation with No Labels) framework, the model compares different augmented views of the same image. A "student" network is trained to predict the output of a "teacher" network. To ensure the model learns meaningful representations rather than trivial solutions, the authors employ sharpening and centering techniques.
  2. Local Image Reconstruction: Recognizing that satellite imagery contains critical small-scale details, the authors integrated a lightweight decoder. This forces the encoder to reconstruct the original image from its internal features, ensuring the model captures local semantic information, such as building edges and structural textures.

Stage 2: Downstream Fine-tuning

The pre-trained encoder is subsequently integrated into a Siamese Network. This architecture features two identical branches with shared weights to process pre-disaster and post-disaster images simultaneously. The features from both branches are fused and passed to a UPerNet semantic segmentation head. The model produces two primary outputs: a localization mask identifying building footprints and a classification output categorizing damage on a four-level scale (from "no damage" to "destroyed").

Dataset and Experimental Setup

The study utilizes the xBD dataset, the largest and most diverse benchmark for building damage assessment. The dataset comprises 22,068 high-resolution images covering 5,000 km² and includes 19 different disaster types across 15 countries. Annotations include polygons for building footprints and standardized damage levels.

To evaluate the model's efficiency, the researchers compared their SSL approach against standard ImageNet pre-training and other state-of-the-art self-supervised methods, such as MoCo v3 and standard DINO, across varying levels of label availability.

Results and Discussion

The proposed model demonstrated superior performance and robustness, particularly when training data was limited:

  • The 1% Label Challenge: When fine-tuned with only 1% of the xBD labels, the model achieved an F1-score of 0.390. This significantly outperformed ImageNet pre-training (0.321) and standard DINO (0.366), proving that domain-specific SSL is more effective than generic pre-training.
  • Scaling Performance: With 20% of the labels, the model reached an F1-score of 0.636, maintaining a competitive edge over other initialization methods.
  • Feature Visualization: Attention matrix visualizations revealed that the self-supervised encoder naturally learns to focus on building structures without any human guidance. The resulting attention maps closely align with ground-truth building masks.
  • Convergence and Efficiency: Training curves indicated that the proposed model converges faster and reaches a higher performance plateau than models initialized with random weights or ImageNet features.

Conclusion and Future Work

This research demonstrates that self-supervised learning can effectively mitigate the data scarcity problem in disaster response. By combining global and local feature learning, the model captures the intricate details necessary for accurate damage assessment from satellite imagery.

Future research directions include exploring deeper transformer architectures to capture more complex patterns and incorporating multi-temporal image sequences to monitor the progression of damage and recovery over time. Additionally, addressing the inherent class imbalance in disaster data—where "destroyed" buildings are significantly rarer than "undamaged" ones—remains a critical area for further refinement.

Summary updated

Abstract

In the field of post-disaster assessment, for timely and accurate rescue and localization after a disaster, people need to know the location of damaged buildings. In deep learning, some scholars have proposed methods to make automatic and highly accurate building damage assessments by remote sensing images, which are proved to be more efficient than assessment by domain experts. However, due to the lack of a large amount of labeled data, these kinds of tasks can suffer from being able to do an accurate assessment, as the efficiency of deep learning models relies highly on labeled data. Although existing semi-supervised and unsupervised studies have made breakthroughs in this area, none of them has completely solved this problem. Therefore, we propose adopting a self-supervised comparative learning approach to address the task without the requirement of labeled data. We constructed a novel asymmetric twin network architecture and tested its performance on the xBD dataset. Experiment results of our model show the improvement compared to baseline and commonly used methods. We also demonstrated the potential of self-supervised methods for building damage recognition awareness.
BibTeX
@inproceedings{xia2022self,
  author = {Zaishuo Xia and Zelin Li and Yanbing Bai and Jinze Yu and Bruno Adriano},
  title = {{Self-supervised Learning for Building Damage Assessment from Large-Scale xBD Satellite Imagery Benchmark Datasets}},
  booktitle = {DEXA},
  year = {2022},
  url = {https://openreview.net/forum?id=gT4m6cHn93},
}
Figure