Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance

Yongshuo Zhu, Lu Li, Keyan Chen, Chenyang Liu, Fugen Zhou, Zhenwei ShiarXiv 2024 · OpenReview

TL;DR

This paper introduces Semantic-CC, a novel approach to describing changes in remote sensing images. By combining the power of foundation models (like SAM and LLMs) with the precise pixel-level information from change detection, Semantic-CC generates more accurate and detailed captions for image transformations, overcoming limitations of previous methods.

What Problem Does This Paper Address?

Remote sensing image change captioning (RSICC) is crucial for tasks like environmental monitoring and urban planning, but current methods struggle to provide granular, robust, and precise descriptions of changes. They often over-rely on high-level semantic information, missing fine details, and can be misled by irrelevant factors like lighting variations. Existing RSICC models also face challenges in integrating information from different temporal images and bridging the gap between pixel-level details and natural language descriptions, often requiring extensive annotated data.

Methods

Semantic-CC tackles these challenges with a multi-component architecture:

  • Bi-temporal SAM-based Encoder: This component leverages the latent knowledge of the Segment Anything Model (SAM) to extract features from two temporally distinct images. A novel "bi-temporal change semantic filter" (BCSF) is integrated to specifically enhance the interaction and filtering of change-related semantic features between the two images.
  • Multi-task Semantic Aggregation Neck: This module facilitates information flow and interaction between the change detection (CD) and change captioning (CC) tasks. It uses "intra-task attention" to mine task-specific knowledge and "inter-task attention" to amalgamate homogeneous information across tasks, effectively bridging semantic gaps.
  • Change Detection Decoder: A streamlined multi-scale decoder is used to generate pixel-level semantic guidance for the CC task. This provides precise localization of changes, which is crucial for detailed descriptions.
  • Change Caption Decoder: This component is based on the Vicuna LLM. It includes a "Q-Former" to query semantic features, a "change semantic feature enhancer" to generate bi-temporal differential features, and the Vicuna LLM itself to produce the final descriptive sentences.
  • Three-Stage Training Strategy: To ensure stable multi-task learning and prevent negative transfer between CD and CC, a specific training regimen is employed, supervising different tasks at distinct stages.

Datasets

The proposed method is validated on two datasets:

  • LEVIR-CD: A dataset for change detection, containing high-resolution bi-temporal images with annotations for building changes.
  • LEVIR-CC: Derived from LEVIR-CD, this dataset is specifically designed for change captioning, with images resized to 256x256 pixels and corresponding natural language descriptions of changes.

Key Insights and Ideas

  • Foundation Models as Knowledge Source: The paper demonstrates that leveraging pre-trained foundation models (SAM, LLMs) can significantly boost RSICC performance, reducing the reliance on extensive task-specific annotations.
  • Pixel-Level Guidance is Crucial: Integrating precise pixel-level change detection information into the captioning process is key to generating more granular, accurate, and robust descriptions, especially for complex scenarios.
  • Multi-task Learning Synergy: The study highlights the complementary nature of CD and CC, showing that jointly training them can lead to superior performance in both tasks, rather than hindering each other.
  • Bridging Semantic Gaps: The proposed aggregation neck and feature enhancer are critical for effectively transferring and integrating information between the low-level pixel representations and the high-level semantic representations required for natural language generation.

Key Results

  • State-of-the-Art Performance: Semantic-CC achieves state-of-the-art results on the LEVIR-CC dataset for change captioning, outperforming existing methods across various metrics like METEOR, ROUGE-L, CIDEr, and BLEU scores. For example, it achieved a ROUGE-L score of 0.7776 and a CIDEr score of 1.3851.
  • High-Precision Change Detection: The model also demonstrates competitive performance in change detection on the LEVIR-CD dataset, achieving metrics comparable to specialized CD methods.
  • Qualitative Improvements: Visual comparisons show that Semantic-CC generates more accurate and detailed captions, correctly identifying quantities, locations, and types of changes, especially in challenging scenarios with complex urban development or vegetation changes.

Limitations and Future Work

  • Dataset Specificity: While validated on LEVIR-CC and LEVIR-CD, the generalizability to other remote sensing datasets with different characteristics (e.g., varying resolutions, geographic regions, or change types) could be further explored.
  • Computational Cost: The use of large foundation models and multi-task learning can be computationally intensive, and further optimization for efficiency might be beneficial.
  • Future Directions: The authors suggest exploring more advanced foundation models and investigating more sophisticated multi-task learning strategies to further enhance performance and robustness. The potential for applying this framework to other remote sensing tasks beyond change captioning is also a promising avenue.

Summary updated

Abstract

Remote sensing image change captioning (RSICC) aims to articulate the changes in objects of interest within bi-temporal remote sensing images using natural language. Given the limitations of current RSICC methods in expressing general features across multi-temporal and spatial scenarios, and their deficiency in providing granular, robust, and precise change descriptions, we introduce a novel change captioning (CC) method based on the foundational knowledge and semantic guidance, which we term Semantic-CC. Semantic-CC alleviates the dependency of high-generalization algorithms on extensive annotations by harnessing the latent knowledge of foundation models, and it generates more comprehensive and accurate change descriptions guided by pixel-level semantics from change detection (CD). Specifically, we propose a bi-temporal SAM-based encoder for dual-image feature extraction; a multi-task semantic aggregation neck for facilitating information interaction between heterogeneous tasks; a straightforward multi-scale change detection decoder to provide pixel-level semantic guidance; and a change caption decoder based on the large language model (LLM) to generate change description sentences. Moreover, to ensure the stability of the joint training of CD and CC, we propose a three-stage training strategy that supervises different tasks at various stages. We validate the proposed method on the LEVIR-CC and LEVIR-CD datasets. The experimental results corroborate the complementarity of CD and CC, demonstrating that Semantic-CC can generate more accurate change descriptions and achieve optimal performance across both tasks.
BibTeX
@misc{zhu2024semantic,
  author = {Yongshuo Zhu and Lu Li and Keyan Chen and Chenyang Liu and Fugen Zhou and Zhenwei Shi},
  title = {{Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance}},
  year = {2024},
  eprint = {2407.14032},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.14032},
}
Figure