Remote Sensing Change Detection With Transformers Trained from Scratch

Mubashir Noman, Mustansar Fiaz, Hisham Cholakkal, Sanath Narayan, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan

Abstract

Current transformer-based change detection (CD) approaches either employ a pre-trained model trained on large-scale image classification ImageNet dataset or rely on first pre-training on another CD dataset and then fine-tuning on the target benchmark. This current strategy is driven by the fact that transformers typically require a large amount of training data to learn inductive biases, which is insufficient in standard CD datasets due to their small size. We develop an end-to-end CD approach with transformers that is trained from scratch and yet achieves state-of-the-art performance on four public benchmarks. Instead of using conventional self-attention that struggles to capture inductive biases when trained from scratch, our architecture utilizes a shuffled sparse-attention operation that focuses on selected sparse informative regions to capture the inherent characteristics of the CD data. Moreover, we introduce a change-enhanced feature fusion (CEFF) module to fuse the features from input image pairs by performing a per-channel re-weighting. Our CEFF module aids in enhancing the relevant semantic changes while suppressing the noisy ones. Extensive experiments on four CD datasets reveal the merits of the proposed contributions, achieving gains as high as 14.27% in intersection-over-union (IoU) score, compared to the best-published results in the literature. Code is available at https://github.com/mustansarfiaz/ScratchFormer.

Introduction

In remote sensing, Change Detection (CD) is the task of identifying relevant semantic changes—such as new buildings or roads—between multi-temporal images while ignoring "noisy" changes like seasonal vegetation shifts, shadows, or lighting. While Convolutional Neural Networks (CNNs) excel at local feature extraction, they often struggle with long-range dependencies. Conversely, Transformers capture global context but are notoriously "data-hungry," typically requiring pre-training on massive datasets like ImageNet to overcome their lack of inherent inductive biases. Given that remote sensing CD datasets are relatively small, training standard Transformers from scratch often leads to overfitting. ScratchFormer addresses this by introducing architectural innovations that allow for efficient training directly on CD benchmarks without external pre-training.

Methodology

ScratchFormer utilizes a Siamese hierarchical framework designed to learn effectively from scratch. The architecture centers on two primary innovations:

Shuffled Sparse Attention (SSA)

Standard self-attention has quadratic complexity and lacks spatial inductive biases, making it difficult to train on small datasets. SSA instead performs token-mixing over a sparse, data-dependent subset of features. By using learnable offsets to focus on the most informative regions, SSA reduces computational overhead and acts as a regularizer, enabling the model to learn robust features even with limited data.

Change-Enhanced Feature Fusion (CEFF)

To combine features from the "pre-change" and "post-change" streams, ScratchFormer employs the CEFF module. Unlike simple concatenation or subtraction, CEFF uses global average pooling and per-channel re-weighting to explicitly highlight semantic changes while suppressing channels that represent noise (e.g., shadows or weather variations).

Architecture of ScratchFormer featuring the Shuffled Sparse Attention Layer and Change-enhanced Features Fusion Module.

Experimental Results

The model was evaluated on four major benchmarks: LEVIR-CD, DSIFN-CD, WHU-CD, and CDD-CD. ScratchFormer consistently outperformed existing state-of-the-art (SOTA) models, including those utilizing ImageNet pre-training.

Quantitative Performance

As shown in the table below, ScratchFormer achieves significant improvements across all metrics. Notably, on the DSIFN-CD dataset, it reached an IoU of 90.75%, representing a 14.27% absolute gain over the previous best result (ChangeFormer).

Quantitative comparison of ScratchFormer against state-of-the-art methods across four benchmarks.

Key Insights

  • Efficiency: On the DSIFN-CD dataset, ScratchFormer achieved performance comparable to the previous state-of-the-art in just 10% of the training time.
  • Sparsity as a Strength: Forcing the model to attend to a sparse subset of pixels effectively introduces a beneficial inductive bias that prevents the transformer from being distracted by noise in small datasets.
  • Challenging the Pre-training Paradigm: The results demonstrate that tailored architectural design can compensate for the lack of massive pre-training data in specialized domains like remote sensing.

Conclusion

ScratchFormer provides a blueprint for deploying Transformers in data-constrained environments. By optimizing the attention mechanism and feature fusion for the specific characteristics of change detection, the model achieves superior performance without the need for millions of ImageNet samples. Future work will explore the applicability of shuffled sparse attention in other domains such as medical imaging.

Summary updated

Abstract

Current transformer-based change detection (CD) approaches either employ a pre-trained model trained on large-scale image classification ImageNet dataset or rely on first pre-training on another CD dataset and then fine-tuning on the target benchmark. This current strategy is driven by the fact that transformers typically require a large amount of training data to learn inductive biases, which is insufficient in standard CD datasets due to their small size. We develop an end-to-end CD approach with transformers that is trained from scratch and yet achieves state-of-the-art performance on four public benchmarks. Instead of using conventional self-attention that struggles to capture inductive biases when trained from scratch, our architecture utilizes a shuffled sparse-attention operation that focuses on selected sparse informative regions to capture the inherent characteristics of the CD data. Moreover, we introduce a change-enhanced feature fusion (CEFF) module to fuse the features from input image pairs by performing a per-channel re-weighting. Our CEFF module aids in enhancing the relevant semantic changes while suppressing the noisy ones. Extensive experiments on four CD datasets reveal the merits of the proposed contributions, achieving gains as high as 14.27\% in intersection-over-union (IoU) score, compared to the best-published results in the literature. Code is available at \url{https://github.com/mustansarfiaz/ScratchFormer}.
BibTeX
@article{noman2024remote,
  author = {Mubashir Noman and Mustansar Fiaz and Hisham Cholakkal and Sanath Narayan and Rao Muhammad Anwer and Salman Khan and Fahad Shahbaz Khan},
  title = {{Remote Sensing Change Detection With Transformers Trained from Scratch}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  url = {https://openreview.net/forum?id=ahaT7IielI},
}
Figure