Remote Sensing Change Detection With Transformers Trained From Scratch

Mubashir Noman, Mustansar Fiaz, Hisham Cholakkal, Sanath Narayan, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan

TL;DR

ScratchFormer is a novel transformer-based architecture for remote sensing change detection that breaks the industry's reliance on massive pre-training. By introducing shuffled sparse attention and a specialized feature fusion module, it achieves state-of-the-art results on four major benchmarks while being trained entirely from scratch—outperforming models that had the head start of ImageNet pre-training.

What Problem Does This Paper Address?

In remote sensing, Change Detection (CD) is the task of identifying relevant semantic changes (like new buildings or roads) between images taken at different times, while ignoring "noisy" changes like seasonal vegetation shifts, shadows, or lighting.

Currently, the field is divided:

  1. CNNs are great at picking up local patterns but struggle with long-range dependencies (e.g., understanding how a change in one corner of a large industrial complex relates to another).
  2. Transformers excel at long-range dependencies but are notoriously "data-hungry." Because they lack the inherent "inductive biases" (like spatial locality) that CNNs have, they usually require pre-training on massive datasets like ImageNet.

The problem? Remote sensing CD datasets are relatively small. Training a standard transformer from scratch on these small datasets usually leads to overfitting and poor convergence. ScratchFormer aims to fix this by baking efficiency and smarter attention directly into the architecture.

Methods

The authors propose a Siamese (two-stream) hierarchical framework with two primary innovations:

  • Shuffled Sparse Attention (SSA): Instead of the standard "dense" self-attention—which compares every pixel to every other pixel and has quadratic complexity—SSA performs token-mixing over a sparse, data-dependent subset of features. It uses learnable offsets to "choose" which regions are most informative. This reduces the computational load and acts as a form of regularization, helping the model learn relevant features even with limited data.
  • Change-Enhanced Feature Fusion (CEFF): Most models combine features from the "before" and "after" images using simple subtraction or concatenation. CEFF is more sophisticated; it uses global average pooling and per-channel re-weighting to explicitly highlight semantic changes while suppressing channels that represent noise (like shadows or weather).

Datasets

The model was evaluated on four widely recognized public CD benchmarks:

  • LEVIR-CD: Focused on building changes in high-resolution Google Earth imagery.
  • DSIFN-CD: A challenging dataset covering six cities in China with high-resolution satellite pairs.
  • WHU-CD: Another building-focused dataset with very high-resolution aerial imagery.
  • CDD-CD: Specifically designed to test a model's ability to handle seasonal variations.

Key Insights and Ideas

  • Sparse is Stronger: The most surprising finding is that less attention can lead to better results. By forcing the model to look at a sparse subset of pixels, the authors effectively introduced an inductive bias that prevents the transformer from getting "distracted" by the noise inherent in small datasets.
  • Efficiency Drives Convergence: ScratchFormer doesn't just reach a higher peak; it gets there faster. On the DSIFN-CD dataset, ScratchFormer achieved performance comparable to the previous state-of-the-art in just 10% of the training time.
  • The "Pre-training Myth": The paper challenges the assumption that transformers must be pre-trained for specialized tasks. By tailoring the attention mechanism to the nature of the data (sparse informative regions), they proved that architectural design can compensate for a lack of massive external data.

Key Results

ScratchFormer set new State-of-the-Art (SOTA) records across all four datasets:

  • DSIFN-CD: Achieved an IoU (Intersection over Union) of 90.75%, a massive 14.27% absolute gain over the previous best-published result (ChangeFormer).
  • LEVIR-CD: Reached an IoU of 84.81%, outperforming models that utilized ImageNet pre-training.
  • Ablation Success: Removing the SSA or CEFF modules led to significant performance drops, proving that both the sparse attention and the smart fusion are critical to the model's success.

Limitations and Future Work

While ScratchFormer dominates the remote sensing benchmarks, the authors note a few areas for further exploration:

  • Domain Expansion: The current study is strictly limited to remote sensing. The authors suggest that the "training from scratch" benefits of shuffled sparse attention should be tested on natural images and medical imaging.
  • Complexity vs. Sparsity: While SSA reduces the token count, the overhead of calculating data-dependent offsets is a factor to monitor as image resolutions continue to scale.

In summary, ScratchFormer provides a blueprint for using Transformers in data-constrained environments, proving that with the right architectural tweaks, you don't need millions of ImageNet samples to build a world-class vision model.

Summary updated

Abstract

Current transformer-based change detection (CD) approaches either employ a pretrained model trained on a large-scale image classification ImageNet dataset or rely on first pretraining on another CD dataset and then fine-tuning on the target benchmark. This current strategy is driven by the fact that transformers typically require a large amount of training data to learn inductive biases, which is insufficient in standard CD datasets due to their small size. We develop an end-to-end CD approach with transformers that is trained from scratch and yet achieves state-of-the-art performance on five benchmarks. Instead of using conventional self-attention that struggles to capture inductive biases when trained from scratch, our architecture utilizes a shuffled sparse-attention operation that focuses on selected sparse informative regions to capture the inherent characteristics of the CD data. Moreover, we introduce a change-enhanced feature fusion (CEFF) module to fuse the features from input image pairs by performing a per-channel re-weighting. Our CEFF module aids in enhancing the relevant semantic changes while suppressing the noisy ones. Extensive experiments on five CD datasets reveal the merits of the proposed contributions, achieving gains as high as 1.35% in intersection over union (IoU) score, compared to the best-published results in the literature. The code is available at https://github.com/mustansarfiaz/ScratchFormer .
BibTeX
@article{noman2024remote,
  author = {Mubashir Noman and Mustansar Fiaz and Hisham Cholakkal and Sanath Narayan and Rao Muhammad Anwer and Salman Khan and Fahad Shahbaz Khan},
  title = {{Remote Sensing Change Detection With Transformers Trained From Scratch}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  url = {https://openreview.net/forum?id=ahaT7IielI},
}

Loading related content...

Figure