Exploring Difference Semantic Prior Guidance for Remote Sensing Image Change Captioning

Yunpeng Li, Xiangrong Zhang, Guanchun Wang, T. ZhangRemote 2026 · DOI

TL;DR

This paper introduces a novel framework for Remote Sensing Image Change Captioning (RSICC) that leverages "difference semantic prior guidance" to generate more accurate and contextually rich descriptions of changes between bi-temporal satellite images. By effectively combining deep and shallow visual features with semantic understanding, their model significantly improves captioning performance on benchmark datasets.

What Problem Does This Paper Address?

Remote Sensing Image Change Captioning (RSICC) aims to automatically describe changes detected between two satellite images taken at different times. This is crucial for applications like urban planning, disaster assessment, and environmental monitoring. However, existing RSICC methods often struggle with:

  • Contextual Awareness: Failing to consistently distinguish between changed and unchanged areas, leading to inaccurate descriptions.
  • Semantic Prior Guidance: Difficulty in effectively incorporating high-level semantic knowledge about the changes, especially for subtle or weak changes.
  • Cross-Modal Consistency: Bridging the gap between visual features of changes and the textual descriptions.

This paper addresses these challenges by proposing a model that explicitly guides the captioning process with semantic understanding derived from the detected visual differences.

Methods

The proposed RSICC framework, named "Difference Semantic Prior Guidance Network," consists of three core modules:

  1. Context-Aware Difference Learning Module: This module extracts multi-scale features from both "before" and "after" images using a Siamese encoder (ResNet-101). It employs a dual-branch difference extraction layer:
  2. Symmetric Difference Context Encoding: Captures common and difference context features using symmetric operations (addition, multiplication, subtraction) and Transformer layers to enhance consistency.
  3. Shallow Difference Auxiliary Encoding: Focuses on distilling fine-grained details and weak changes using shallow features and Transformer layers.
  4. An adaptive fusion block then combines these deep and shallow representations.
  5. Difference Comprehending Module: This module aims to learn semantic knowledge from the extracted difference features. It takes the shallow difference features and a set of pre-defined "change words" (nouns from annotated sentences) as input. It uses cross-attention mechanisms to align visual change information with semantic cues, generating a set of predicted change semantics.
  6. Change Caption Decoder: A Transformer-based decoder that utilizes a cross refined attention mechanism. This attention layer balances the fused difference features (from the context-aware module) and the predicted change semantics (from the comprehending module) to generate the final textual description. This allows for better interaction between visual change content and semantic understanding.

The overall training objective is a weighted combination of cross-entropy loss for caption generation, a ranking loss for difference learning, and a focal loss for difference comprehending.

Datasets

The proposed model was evaluated on two widely used benchmark datasets:

  • LEVIR-CC: Contains 10,077 RSI pairs with a time span of 5-14 years, focusing on changes in buildings. It has a maximum sentence length of 39 words.
  • Dubai-CC: Features bi-temporal RSIs from Dubai urban environments collected between 2000 and 2010. Images are cropped to 50x50 pixels and then upsampled to 256x256 for consistency. This dataset has fewer changing scenarios and shorter annotated sentences (max 27 words).

Key Insights and Ideas

  • Dual-Branch Difference Extraction: Separating deep and shallow feature extraction allows the model to capture both high-level contextual changes and fine-grained details, which is crucial for comprehensive change understanding.
  • Symmetric Difference Encoding: Using symmetric operations (like T1-to-T2 and T2-to-T1) helps in more robustly identifying consistent changes and reducing noise.
  • Semantic Prior Guidance: Explicitly incorporating "change words" and using a difference comprehending module to learn semantic representations significantly improves the model's ability to generate relevant and accurate captions.
  • Cross Refined Attention: This mechanism in the decoder is key to effectively integrating visual change information with semantic cues, leading to more coherent and informative captions.
  • Weak Change Detection: The shallow difference branch is particularly effective in capturing subtle changes that might be missed by methods focusing solely on deep features.

Key Results

The proposed model achieved state-of-the-art performance on both LEVIR-CC and Dubai-CC datasets, particularly excelling in the CIDEr metric.

  • On the LEVIR-CC dataset, the model achieved a CIDEr score of 143.34%, which is a 3.11% improvement over the previous best (SAT-Cap). It also showed strong performance across other metrics like BLEU, METEOR, and ROUGE_L.
  • On the Dubai-CC dataset, the model achieved a CIDEr score of 1.0042, outperforming other methods.
  • Qualitative results (Figure 3 and 4) demonstrate that the model can generate more accurate and detailed change captions, correctly identifying and describing various changes, including those involving vegetation removal, new construction, and road development.

Limitations and Future Work

  • Complexity and Inference Speed: The paper acknowledges that their model, due to its complex structure, has a disadvantage in terms of parameter quantity and inference time compared to some lighter models. This is noted as a limitation for industrial implementation.
  • Future Work: The authors suggest exploring lightweight RSICC models to improve efficiency. They also propose focusing on improving model accessibility with multi-task encoders and investigating the use of knowledge graphs with diffusion decoders to capture long-range dependencies and enhance predictive accuracy.

Summary updated

Abstract

Understanding complex change scenes is a crucial challenge in remote sensing field. Remote sensing image change captioning (RSICC) task has emerged as a promising approach to translate appeared changes between bi-temporal remote sensing images into textual descriptions, enabling users to make accurate decisions. Current RSICC methods frequently encounter difficulties in consistency for contextual awareness and semantic prior guidance. Therefore, this study explores difference semantic prior guidance network to reason context-rich sentence for capturing appeared vision changes. Specifically, the context-aware difference module is introduced to guarantee the consistency of unchanged/changed context features, strengthening multi-level changed information to improve the ability of semantic change feature representation. Moreover, to effectively mine higher-level cognition ability to reason salient/weak changes, we employ difference comprehending with shallow change information to realize semantic change knowledge learning. In addition, the designed parallel cross refined attention in Transformer decoder can balance vision difference and semantic knowledge for implicit knowledge distilling, enabling fine-grained perception changes of semantic details and reducing pseudochanges. Compared with advanced algorithms on the LEVIR-CC and Dubai-CC datasets, experimental results validate the outstanding performance of the designed model in RSICC tasks. Notably, on the LEVIR-CC dataset, it reaches a CIDEr score of 143.34%, representing a 3.11% improvement over the most competitive SAT-cap.
BibTeX
@inproceedings{li2026exploring,
  author = {Yunpeng Li and Xiangrong Zhang and Guanchun Wang and T. Zhang},
  title = {{Exploring Difference Semantic Prior Guidance for Remote Sensing Image Change Captioning}},
  booktitle = {Remote},
  year = {2026},
  doi = {10.3390/rs18020232},
}
Figure