CSSA: A Cross-Modal Spatial–Semantic Alignment Framework for Remote Sensing Image Captioning

Xiao Han, Zhaoji Wu, Yunpeng Li, Xiangrong Zhang, Guanchun Wang, Biao HouRemote 2026 · DOI

TL;DR

This paper introduces CSSA, a novel framework for generating natural language descriptions of remote sensing images. It tackles the challenge of aligning visual and textual information by employing a cross-modal contrastive learning mechanism and a dynamic geometry Transformer, leading to more accurate and spatially aware captions.

What Problem Does This Paper Address?

Generating descriptive captions for remote sensing images (RSIs) is a complex task. Unlike typical images, RSIs are captured from a top-down perspective, leading to scattered objects with irregular shapes and sparsity. Existing image captioning methods often struggle with these characteristics, failing to fully capture the spatial relationships between objects and the nuances of the scene. This results in captions that may lack semantic accuracy or spatial coherence. The core problem is bridging the gap between the visual information in noisy, spatially complex RSIs and the structured nature of natural language descriptions.

Methods

The proposed CSSA framework consists of two main novel components:

  • Multi-Branch Cross-Modal Contrastive Learning (MCCL): This mechanism aims to reduce the "modality gap" between images and text. It uses contrastive learning to encourage similar image-text pairs to have close representations in a shared semantic space, while pushing dissimilar pairs apart. This is achieved by using text representations to guide the extraction of image features, promoting semantic alignment. The MCCL also incorporates image and text augmentation branches to enhance feature robustness.
  • Dynamic Geometry Transformer (DG-former): To address the spatial challenges of RSIs, the DG-former is introduced. It enhances a standard Transformer by dynamically incorporating geometry information. This module helps the model understand the spatial layout and relationships between scattered objects, which is crucial for accurate captioning. The DG-former uses a novel "Multi-head dynamic geometry enhancement Attention" (MGA) mechanism and a "Dynamic Geometry Positional Encoding" (DGP) module that adjusts the influence of geometry information across different layers of the Transformer.

Datasets

The authors evaluated their CSSA framework on three publicly available remote sensing image captioning datasets:

  • Sydney-Captions: Contains 613 images with five descriptive sentences per image, covering seven types of ground objects.
  • UCM-Captions: Based on the UC-Merced land use dataset, it includes 21 categories of ground objects, with 100 images per category and five descriptive sentences per image.
  • RSICD: The largest of the three, with 10,921 images from various sources, each with five sentence descriptions.

Key Insights and Ideas

  • Geometry is Key for RSIs: The paper strongly emphasizes that traditional image captioning approaches, which often overlook spatial relationships, are insufficient for remote sensing imagery. The unique characteristics of RSIs (scattered objects, overhead view) necessitate explicit modeling of geometry.
  • Text Guides Vision: The MCCL mechanism demonstrates that using textual information to guide the learning of visual features can significantly improve semantic alignment and reduce the noise inherent in RSIs.
  • Dynamic Geometry Adaptation: The DG-former's ability to dynamically adjust the influence of geometry information across different Transformer layers is crucial. This allows the model to focus more on local spatial details in earlier layers and global relationships in later layers, adapting to the varying needs of RSI understanding.
  • Combined Approach Yields Superiority: The significant improvements observed when combining both MCCL and DG-former highlight the synergistic effect of addressing both semantic and spatial challenges simultaneously.

Key Results

The CSSA framework achieved state-of-the-art performance across all three benchmark datasets.

  • Quantitative Improvements: On the Sydney-Captions dataset, CSSA achieved a 15.9% improvement in CIDEr score compared to the "RASG" method. Similar gains were observed on UCM-Captions. Compared to Transformer-based methods like "KE," CSSA showed improvements of 10.1% and 5.9% on Sydney-Captions and UCM-Captions, respectively. On the RSICD dataset, CSSA scored first in all eight evaluation metrics.
  • Qualitative Improvements: Visualizations showed that CSSA generated more accurate and detailed captions. For instance, it correctly identified the number of objects (e.g., "four baseball fields" vs. "a baseball field"), described subtle details like "white waves," and avoided misidentifying objects (e.g., "railway station" vs. "residential area"). Ablation studies confirmed the individual contributions of the DGP and MCCL modules to these improvements.

Limitations and Future Work

  • Limitations: The paper acknowledges that the current combination of modules is a "preliminary option" and that "various alternative image encoder and text decoder models can be used to implement the proposed framework, and they may also achieve a better performance." The study also notes that the influence of batch size on contrastive learning was relatively minor, suggesting potential for further optimization in other parameters.
  • Future Work: The authors suggest that exploring different encoder and decoder architectures could further enhance performance. The paper also implicitly points towards future work in refining the dynamic geometry adjustment strategies and exploring other cross-modal alignment techniques.

Summary updated

Abstract

Remote sensing image captioning (RSIC) aims to generate natural language descriptions for the given remote sensing image, which requires a comprehensive and in-depth understanding of image content and summarizes it with sentences. Most RSIC methods have successful vision feature extraction, but the representation of spatial features or fusion features fails to fully consider cross-modal differences between remote sensing images and texts, resulting in unsatisfactory performance. Thus, we propose a novel cross-modal spatial–semantic alignment (CSSA) framework for an RSIC task, which consists of a multi-branch cross-modal contrastive learning (MCCL) mechanism and a dynamic geometry Transformer (DG-former) module. Specifically, compared to discrete text, remote sensing images present a noisy property, interfering with the extraction of valid vision features. Therefore, we present an MCCL mechanism to learn consistent representation between image and text, achieving cross-modal semantic alignment. In addition, most objects are scattered in remote sensing images and exhibit a sparsity property due to the overhead view. However, the Transformer structure mines the objects’ relationships without considering the geometry information of the objects, leading to suboptimal capture of the spatial structure. To address this, a DG-former is designed to realize spatial alignment by introducing geometry information. We conduct experiments on three publicly available datasets (Sydney-Captions, UCM-Captions and RSICD), and the superior results demonstrate its effectiveness.
BibTeX
@inproceedings{han2026cssa,
  author = {Xiao Han and Zhaoji Wu and Yunpeng Li and Xiangrong Zhang and Guanchun Wang and Biao Hou},
  title = {{CSSA: A Cross-Modal Spatial–Semantic Alignment Framework for Remote Sensing Image Captioning}},
  booktitle = {Remote},
  year = {2026},
  doi = {10.3390/rs18030522},
}
Figure