RSCaMa: Remote Sensing Image Change Captioning With State Space Model

Chenyang Liu, Keyan Chen, Bowen Chen, Haotian Zhang, Zhengxia Zou, Zhenwei Shi

TL;DR

This paper introduces RSCaMa, a novel approach for describing changes in satellite imagery using State Space Models (SSMs), specifically Mamba. By integrating Mamba's efficient spatial and temporal modeling capabilities, RSCaMa significantly improves the accuracy and detail of change captions compared to existing methods, paving the way for more advanced remote sensing analysis.

What Problem Does This Paper Address?

Remote Sensing Image Change Captioning (RSICC) is the task of generating natural language descriptions of changes observed between two or more satellite images of the same area over time. This involves identifying not just what has changed (e.g., new buildings, deforestation), but also where and how (e.g., added, disappeared). Existing methods often struggle with effectively modeling both the spatial details of changes and their temporal evolution simultaneously. Many rely on Convolutional Neural Networks (CNNs) with limited receptive fields or Transformers, which can be computationally expensive. There's a need for models that can efficiently capture complex spatial-temporal relationships in remote sensing data for accurate change description.

Methods

The core of RSCaMa is its innovative use of Mamba, a recent State Space Model (SSM) known for its global receptive field and linear complexity, which addresses limitations of prior CNN and Transformer-based approaches. The model is built around "CaMa layers" that iteratively refine bi-temporal features.

Key novel components include:

  • Spatial Difference-aware SSM (SD-SSM): This module enhances spatial change perception. It leverages Mamba's capabilities by incorporating bidirectional scans (forward and backward) on flattened visual tokens. Crucially, it multiplies the output of these bidirectional SSMs with "differencing features" (derived from the difference between the two input images) to sharpen the model's focus on actual changes.
  • Temporal-Traversing SSM (TT-SSM): This module tackles temporal modeling. Recognizing Mamba's potential for temporal scanning, TT-SSM rearranges the bi-temporal features into an interleaved sequence. This "temporal cross-wise" scanning allows for more effective interaction and understanding of temporal information between the two image sequences.

These CaMa layers, composed of SD-SSM and TT-SSM, are stacked to iteratively refine the bi-temporal features, leading to improved joint spatial-temporal modeling. The processed features are then fed into a language decoder to generate the final caption.

Datasets

The primary dataset used for training and evaluation is LEVIR-CC. This is described as a large-scale RSICC dataset containing 10,077 image pairs and 50,385 descriptive captions, covering changes over 5 to 15 years across 20 regions in Texas, USA.

Key Insights and Ideas

  • Mamba's Potential for RSICC: The paper strongly advocates for Mamba as a powerful alternative to traditional CNNs and Transformers in the RSICC domain, highlighting its efficiency and effectiveness in capturing both spatial and temporal dynamics.
  • Synergy of SD-SSM and TT-SSM: The combination of spatial and temporal SSM modules is crucial. SD-SSM sharpens the focus on what and where changes occur, while TT-SSM enhances the understanding of when and how these changes evolve over time.
  • Importance of Differential Features: Explicitly using the difference between bi-temporal images within the SD-SSM module is key to guiding the model towards relevant changes, rather than just general image features.
  • Token-wise Temporal Scanning: The TT-SSM's strategy of interleaving tokens from both images for cross-wise scanning is more effective than simple concatenation, demonstrating the benefit of a structured temporal interaction.
  • Transformer Decoder's Dominance: For caption generation, the Transformer decoder with cross-attention outperformed Mamba and GPT-style decoders, suggesting that while Mamba excels at feature extraction, established Transformer architectures remain strong for language generation in this context.

Key Results

  • State-of-the-Art Performance: RSCaMa achieved state-of-the-art results on the LEVIR-CC dataset, outperforming previous methods, including PromptCC, by significant margins on key metrics like BLEU-4 (+1.70%) and Sm (+1.11%).
  • Efficiency: Despite its advanced capabilities, RSCaMa maintains a competitive number of parameters and computational complexity compared to Transformer-based methods.
  • Ablation Studies:
    • Removing either SD-SSM or TT-SSM led to a noticeable drop in performance, confirming their individual contributions.
  • The TT-SSM* (concatenation) variant performed worse than the proposed token-wise cross-scanning, validating the TT-SSM's design.
  • The optimal number of CaMa layers was found to be 3.
  • Qualitative Improvements: Visual examples show RSCaMa generating more accurate and detailed captions, correctly identifying object categories, locations, and relationships between changed objects.

Limitations and Future Work

  • Language Decoder Choice: While RSCaMa's feature extraction is strong, the Transformer decoder was found to be the best performer for caption generation. The paper suggests exploring novel attention mechanisms to further enhance Mamba's cross-modal understanding capabilities for word generation.
  • Combinatorial Exploration: The authors believe Mamba has significant potential and suggest further research into combining it with other architectures like CNNs, Transformers, and MLPs for even more advanced RSICC models.
  • Iterative Refinement: The CaMa layers provide iterative refinement. Future work could explore more sophisticated iterative strategies or different numbers of layers.

Summary updated

Abstract

Remote sensing image change captioning (RSICC) aims to describe surface changes between multitemporal remote sensing images in language, including the changed object categories, locations, and dynamics of changing objects (e.g., added or disappeared). This poses challenges to spatial and temporal modeling of bi-temporal features. Despite previous methods progressing in the spatial change perception, there are still weaknesses in joint spatial-temporal modeling. To address this, in this letter, we propose a novel RSCaMa model, which achieves efficient joint spatial-temporal modeling through multiple CaMa layers, enabling iterative refinement of bi-temporal features. To achieve efficient spatial modeling, we introduce the recently popular Mamba [a state space model (SSM)] with a global receptive field and linear complexity into the RSICC task and propose the Spatial Difference-aware SSM (SD-SSM), overcoming limitations of previous convolutional neural network (CNN)- and Transformer-based methods in the receptive field and computational complexity. SD-SSM enhances the model’s ability to capture spatial changes sharply. In terms of efficient temporal modeling, considering the potential correlation between the temporal scanning characteristics of Mamba and the temporality of the RSICC, we propose the Temporal-Traversing SSM (TT-SSM), which scans bi-temporal features in a temporal crosswise manner, enhancing the model’s temporal understanding and information interaction. Experiments validate the effectiveness of the efficient joint spatial-temporal modeling and demonstrate the outstanding performance of RSCaMa and the potential of the Mamba in the RSICC task. Additionally, we systematically compare three different language decoders, including Mamba, generative pre-trained Transformer (GPT)-style decoder, and Transformer decoder, providing valuable insights for future RSICC research. The code will be available at https://github.com/Chen-Yang-Liu/RSCaMa .
BibTeX
@article{liu2024rscama,
  author = {Chenyang Liu and Keyan Chen and Bowen Chen and Haotian Zhang and Zhengxia Zou and Zhenwei Shi},
  title = {{RSCaMa: Remote Sensing Image Change Captioning With State Space Model}},
  journal = {IEEE Geosci. Remote. Sens. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=MJKtsUZQAU},
}
Figure