VcT: Visual Change Transformer for Remote Sensing Image Change Detection

Bo Jiang, Zitian Wang, Xixi Wang, Ziyan Zhang, Lan Chen, Xiao Wang, Bin Luo

TL;DR

The Visual change Transformer (VcT) shifts the focus of remote sensing change detection from merely identifying "what changed" to carefully mining "what stayed the same." By using a novel Reliable Token Mining (RTM) module—combining Graph Neural Networks and K-means clustering—VcT captures consistent background context to significantly reduce false positives caused by lighting, shadows, and sensor variations.

What Problem Does This Paper Address?

Remote sensing change detection (CD) is the task of identifying pixel-level differences between two images of the same location taken at different times. It sounds simple, but it’s notoriously difficult because of "irrelevant changes." Variations in sunlight intensity, atmospheric conditions, soil moisture, and different sensor resolutions can make the same building look completely different, leading traditional models to flag them as "changed."

Most existing deep learning models focus on enhancing the features of the changed regions. However, the authors argue that these models often ignore the unchanged background context. Without a solid grasp of the consistent background, it is difficult for a model to distinguish between a true structural change (like a new building) and a pseudo-change (like a shadow moving).

Methods

The authors introduce the Visual change Transformer (VcT), which follows a sophisticated multi-stage pipeline:

  1. Feature Extraction: A shared ResNet18 backbone extracts initial feature maps from the image pair.
  2. Reliable Token Mining (RTM): This is the core innovation. Instead of picking tokens randomly or manually, the model uses a Graph Neural Network (GNN) to generate a coarse change map. It then identifies "reliable" background tokens (areas with the lowest probability of change) and refines them using K-means clustering to ensure a diverse and representative set of background anchors.
  3. Transformer Interaction:
    • Self-Attention: Models the intra-image relationships within the selected tokens.
  4. Cross-Attention: Models the inter-image dependencies, allowing the model to "communicate" background information between the two time-steps.
  5. Anchor-Primary Attention (APA): This module fuses the refined, globally-aware tokens back into the original high-resolution feature maps.
  6. Prediction Head: A CNN-based decoder subtracts the enhanced features and produces the final binary change map.

Datasets

The model was evaluated on three widely recognized high-resolution (HSR) remote sensing benchmarks:

  • LEVIR-CD: A large-scale building change detection dataset (0.5m/pixel) from Google Earth.
  • WHU-CD: Focuses on building changes before and after a 6.3 magnitude earthquake.
  • DSIFN-CD: Contains high-resolution images covering six major cities in China with complex land covers.

Key Insights and Ideas

  • Background as an Anchor: The "aha moment" here is treating the unchanged background as a reliable reference point. By aligning the background features, the model can more easily "see through" noise like lighting shifts.
  • GNNs for Coarse Priors: Using a GNN to model the structured spatial information for an initial coarse map proves more effective than standard CNNs for selecting which pixels are "reliable" enough to become tokens.
  • Two-Stage Selection: The transition from Top-K selection to K-means clustering is vital. Top-K selection alone might pick many similar pixels from one small area; K-means ensures the "reliable tokens" represent the entire diversity of the background (trees, roads, water, etc.).
  • Efficiency Matters: Despite using Transformers, VcT is remarkably lightweight. It uses only ~3.57M parameters, making it comparable to the popular BIT model but significantly smaller than other SOTA models like IFNet (~50M).

Key Results

VcT achieved state-of-the-art (SOTA) performance across all three datasets:

  • LEVIR-CD: Achieved an F1-score of 90.04% and an IoU of 81.89%, outperforming the baseline BIT model by a significant margin.
  • WHU-CD: Reached an F1-score of 89.58%, showing strong robustness in post-disaster scenarios.
  • Efficiency: In terms of computational cost, VcT maintains a low footprint (10.64 GFLOPs), proving that global context modeling doesn't have to be computationally prohibitive.

Qualitatively, the visualizations show that VcT produces much "cleaner" change maps with fewer fragmented noise pixels in unchanged areas compared to previous methods.

Limitations and Future Work

The authors identify a few areas for improvement:

  • Fixed Token Strategy: The model currently uses a fixed "Top-K" selection. In extreme cases where almost everything has changed (or nothing has), a fixed K might be sub-optimal. Adaptive token selection is a suggested future path.
  • Semantic Gap: The detector sometimes finds changes in "non-building" regions (like vehicles or trees) that aren't labeled in the ground truth, leading to technical "errors" that are actually correct observations.
  • Foundation Models: The authors suggest that integrating large-scale pre-trained models (like SAM or Grounding DINO) could further improve the semantic understanding of the scene.

Summary updated

Abstract

Given two remote sensing images, the goal of visual change detection task is to detect significantly changed areas between them. Existing visual change detectors usually adopt convolutional neural networks (CNNs) or transformers for feature representation learning and focus on learning effective representation for the changed regions between images. Although good performance can be obtained by enhancing the features of the change regions, however, these works are still limited mainly due to the ignorance of mining the unchanged background context information. It is known that one main challenge for change detection is how to obtain the consistent representations for two images involving different variations, such as spatial variation and sunlight intensity. In this work, we demonstrate that carefully mining the common background information provides an important cue to learn the consistent representations for the two images which thus obviously facilitates the visual change detection problem. Based on this observation, we propose a novel visual change transformer (VcT) model for visual change detection problem. To be specific, a shared backbone network is first used to extract the feature maps for the given image pair. Then, each pixel of feature map is regarded as a graph node and the graph neural network (GNN) is proposed to model the structured information for coarse change map prediction. Top- $K$ reliable tokens can be mined from the map and refined by using the clustering algorithm. Then, these reliable tokens are enhanced by first utilizing self/cross-attention (CA) schemes and then interacting with original features via an anchor-primary attention (APA) learning module. Finally, the prediction head is proposed to get a more accurate change map. Extensive experiments on multiple benchmark datasets validated the effectiveness of our proposed VcT model. The source code and pre-trained models are available at https://github.com/Event-AHU/VcT_Remote_Sensing_Change_Detection .
BibTeX
@article{jiang2023vct,
  author = {Bo Jiang and Zitian Wang and Xixi Wang and Ziyan Zhang and Lan Chen and Xiao Wang and Bin Luo},
  title = {{VcT: Visual Change Transformer for Remote Sensing Image Change Detection}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2023},
  url = {https://openreview.net/forum?id=bXR57OEIt9},
}

Loading related content...

Figure