Diffusion-RSCC: Diffusion Probabilistic Model for Change Captioning in Remote Sensing Images

Xiaofei Yu, Yitong Li, Jie Ma, Chang Li, Hanlin WuIEEE TGRS 2024 · arXiv

Introduction

Remote Sensing Image Change Captioning (RSICC) is a specialized task that involves generating natural language descriptions of semantic changes between bi-temporal satellite images. This task is critical for environmental monitoring, urban planning, and disaster assessment. However, RSICC faces significant challenges: the "pixel problem," where non-semantic changes (lighting, shadows) confuse models; data scarcity in annotated remote sensing datasets; and the limitations of traditional auto-regressive decoders that often produce rigid, repetitive sentence structures.

To address these issues, we present Diffusion-RSCC, a generative framework that treats change captioning as a probabilistic diffusion process. By iteratively denoising Gaussian noise into descriptive text, the model achieves greater linguistic flexibility and better alignment with visual features compared to standard LSTM or Transformer-based approaches.

Example of RSICC task showing ground truth and model prediction.

Methodology

The Diffusion Framework

Diffusion-RSCC operates through a two-stage process. In the Forward Process, ground-truth captions are converted into continuous word embeddings, and Gaussian noise is gradually added until the signal is destroyed. The Reverse Process involves a learned Condition Denoiser that starts from pure noise and iteratively recovers the caption, guided by the bi-temporal image features.

Condition Denoiser Architecture

The core of the model is the Condition Denoiser, which maps visual changes to textual sequences. It consists of three primary components:

  1. Feature Extraction: A ResNet101 backbone extracts high-level features from the "before" and "after" images, which are then used to compute a residual map representing the semantic changes ($I_{di}$).
  2. Cross-Mode Fusion (CMF): This module employs multi-head cross-attention to fuse visual change features directly into the textual diffusion sequence, ensuring that the generated words are grounded in the observed pixels.
  3. Stacking Self-Attention (SSA): A series of self-attention blocks refine the internal coherence of the sentence. Ablation studies indicate that a configuration of three SSA blocks provides the optimal balance between grammatical structure and semantic accuracy.
The overall architecture of Diffusion-RSCC, illustrating the forward/reverse diffusion processes and the internal structure of the Condition Denoiser.

Experimental Evaluation

Dataset

The model was evaluated on the LEVIR-CC dataset, the current benchmark for RSICC. It contains 10,077 image pairs with over 50,000 descriptive sentences covering five change types: buildings, roads, parking lots, vegetation, and water. The images feature a high resolution of 0.5m/pixel and span time intervals of 5 to 15 years.

Quantitative Results

Diffusion-RSCC demonstrates state-of-the-art performance across all standard metrics on LEVIR-CC, outperforming established baselines like DUDA and RSICCformer:

  • BLEU-4: 60.9% (vs. 57.6% for DUDA)
  • METEOR: 37.8%
  • CIDEr: 125.6
  • BARTScore: -417.7
  • MoverScore: 32.3

The improvement in BARTScore and MoverScore highlights the model's superior ability to generate linguistically diverse and semantically accurate captions that go beyond simple n-gram matching.

Discussion and Conclusion

Diffusion-RSCC represents a significant shift from deterministic to generative modeling in remote sensing. By leveraging the diffusion process, the model effectively mitigates overfitting on small datasets and produces more descriptive outputs, such as identifying "neatly arranged villas" instead of generic "buildings."

While the iterative nature of diffusion models introduces higher computational costs compared to single-pass decoders, the gains in factual correctness and descriptive richness are substantial. Future work will focus on optimizing the computational efficiency of the reverse process and extending the model to handle more complex, multi-object change scenarios.

Summary updated

Abstract

Remote sensing image change captioning (RSICC) aims at generating human-like language to describe the semantic changes between bitemporal remote sensing image (RSI) pairs. It provides valuable insight into environmental dynamics and land management. Unlike conventional change captioning (CC) tasks, RSICC involves not only retrieving relevant information across different modalities and generating fluent captions but also mitigating the impact of pixel-level differences on the terrain change localization. Pixel-level discrepancies over a long time span decrease caption accuracy. To address these problems, we propose a probabilistic diffusion-based model that leverages its remarkable generative capability to produce flexible captions. In the training phase, we construct a condition denoiser to efficiently map the real caption distribution to a standard Gaussian distribution. This denoiser incorporates cross-mode fusion (CMF) and stacking self-attention (SSA) modules to enhance cross-modal alignment and reduce pixel interference, thereby improving caption accuracy. In the training phase, the condition denoiser provides a new strategy for mean value estimation and helps to generate captions step by step. Extensive experiments on the LEVIR-CC dataset and DUBAI-CC dataset demonstrate the effectiveness of our Diffusion-RSCC and each of its individual components. The quantitative results showcase superior performance over existing methods across both traditional and newly introduced metrics. The code is available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Fay-Y/Diffusion-RSCC</uri>.
BibTeX
@misc{yu2024diffusion,
  author = {Xiaofei Yu and Yitong Li and Jie Ma and Chang Li and Hanlin Wu},
  title = {{Diffusion-RSCC: Diffusion Probabilistic Model for Change Captioning in Remote Sensing Images}},
  year = {2024},
  eprint = {2405.12875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.12875},
}
Figure