Exploring Multi-Level Attention and Semantic Relationship for Remote Sensing Image Captioning

Zhenghang Yuan, Xuelong Li, Qi Wang

Abstract

Remote sensing image captioning (RSIC) is an emerging field that aims to translate high-level semantic information and object interactions in aerial imagery into natural language. Despite advances in convolutional neural networks (CNNs) and recurrent neural networks (RNNs), RSIC faces two major hurdles: dramatic variations in object scales and the underutilization of visual relationships between ground objects. This paper proposes a framework utilizing multi-level attention and multi-label attribute graph convolution to address these issues. The multi-level attention module adaptively focuses on both spatial features and specific scales, while the attribute graph convolution module learns effective relationship-aware features. Experiments on UCM-captions, Sydney-captions, and RSICD datasets demonstrate the framework's superior performance.

What Problem Does This Paper Address?

Remote sensing image captioning is significantly more complex than standard image captioning due to the unique characteristics of overhead imagery. The authors identify two primary limitations in existing models:

  1. The Scale Problem: In aerial imagery, objects of the same class can vary wildly in size depending on the sensor's altitude and the object's actual dimensions. For example, a large airliner and a small private jet might appear in the same scene, but standard CNNs with fixed receptive fields often fail to represent both effectively.
Significant scale variation of airplanes in remote sensing imagery.
  1. The Relationship Problem: Most models treat detected attributes (like "houses," "roads," or "trees") as independent entities. However, in remote sensing, the spatial and semantic context—such as "houses" being "neatly arranged" along a "road"—is crucial for generating high-quality, descriptive captions.
Remote sensing images featuring diverse objects like tennis courts, rivers, and buildings, highlighting the need for semantic relationship modeling.

Proposed Methodology

The proposed framework employs an encoder-decoder architecture with two innovative components designed to overcome the identified limitations:

Multi-Level Attention Module

Instead of relying solely on the final layer of a CNN, the model extracts features from multiple layers (specifically conv4_3 and conv5_3 of a VGG16 backbone). It then applies two types of attention:

  • Spatial Attention: Identifies relevant regions within the image.
  • Scale-wise (Channel) Attention: Determines which feature scale is most informative for the current word being generated, effectively allowing the model to "zoom" in or out as needed.

Multi-Label Attribute Graph Convolution

To capture the relationships between objects, the authors first train a multi-label classifier to predict attributes (nouns and adjectives). They then build an attribute graph where nodes represent words and edges represent their co-occurrence frequency in the training data. A Graph Convolutional Network (GCN) processes this graph to produce "relationship-aware" attribute features, which are then integrated into the decoder.

The Decoder

A modified Long Short-Term Memory (LSTM) network serves as the decoder, integrating global image features, multi-level attended features, and GCN-refined attribute features to generate the final descriptive sentence.

Experimental Results

The model was evaluated on three major benchmarks: UCM-Captions, Sydney-Captions, and RSICD.

  • State-of-the-Art Performance: The proposed method outperformed several baselines, including FC-ATT+LSTM and SM-ATT+LSTM. On the UCM-Captions dataset, it achieved a BLEU-4 score of 0.6623 and a CIDEr score of 3.1684.
  • Robustness: On the large-scale RSICD dataset, the model showed superior performance across all metrics (BLEU, METEOR, ROUGE_L), proving its effectiveness in handling diverse scenes and scales.
  • Ablation Studies: Results confirmed that the Multi-Level Attention and the Attribute GCN are complementary; the best performance was achieved when both modules were used together.

Conclusion and Future Work

This work demonstrates that explicitly modeling object scales and semantic relationships significantly improves remote sensing image captioning. While the current model relies on a pre-trained attribute classifier, future research could explore dynamic graphs that adapt to the specific spatial layout of individual images, further refining the model's ability to describe complex aerial scenes. Additionally, optimizing the computational overhead for real-time processing remains an important goal for practical applications.

Summary updated

Abstract

Remote sensing image captioning, which aims to understand high-level semantic information and interactions of different ground objects, is a new emerging research topic in recent years. Though image captioning has developed rapidly with convolutional neural networks (CNNs) and recurrent neural networks (RNNs), the image captioning task for remote sensing images still suffers from two main limitations. One limitation is that the scales of objects in remote sensing images vary dramatically, which makes it difficult to obtain an effective image representation. Another limitation is that the visual relationship in remote sensing images is still underused, which should have great potential to improve the final performance. In order to deal with these two limitations, an effective framework for captioning the remote sensing image is proposed in this paper. The framework is based on multi-level attention and multi-label attribute graph convolution. Specifically, the proposed multi-level attention module can adaptively focus not only on specific spatial features, but also on features of specific scales. Moreover, the designed attribute graph convolution module can employ the attribute-graph to learn more effective attribute features for image captioning. Extensive experiments are conducted and the proposed method achieves superior performance on UCM-captions, Sydney-captions and RSICD dataset.
BibTeX
@inproceedings{yuan2019exploring,
  author = {Zhenghang Yuan and Xuelong Li and Qi Wang},
  title = {{Exploring Multi-Level Attention and Semantic Relationship for Remote Sensing Image Captioning}},
  booktitle = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2962195},
}
Figure