Abstract
Remote sensing image captioning (RSIC) is an emerging field that aims to translate high-level semantic information and object interactions in aerial imagery into natural language. Despite advances in convolutional neural networks (CNNs) and recurrent neural networks (RNNs), RSIC faces two major hurdles: dramatic variations in object scales and the underutilization of visual relationships between ground objects. This paper proposes a framework utilizing multi-level attention and multi-label attribute graph convolution to address these issues. The multi-level attention module adaptively focuses on both spatial features and specific scales, while the attribute graph convolution module learns effective relationship-aware features. Experiments on UCM-captions, Sydney-captions, and RSICD datasets demonstrate the framework's superior performance.
What Problem Does This Paper Address?
Remote sensing image captioning is significantly more complex than standard image captioning due to the unique characteristics of overhead imagery. The authors identify two primary limitations in existing models:
- The Scale Problem: In aerial imagery, objects of the same class can vary wildly in size depending on the sensor's altitude and the object's actual dimensions. For example, a large airliner and a small private jet might appear in the same scene, but standard CNNs with fixed receptive fields often fail to represent both effectively.
- The Relationship Problem: Most models treat detected attributes (like "houses," "roads," or "trees") as independent entities. However, in remote sensing, the spatial and semantic context—such as "houses" being "neatly arranged" along a "road"—is crucial for generating high-quality, descriptive captions.
Proposed Methodology
The proposed framework employs an encoder-decoder architecture with two innovative components designed to overcome the identified limitations:
Multi-Level Attention Module
Instead of relying solely on the final layer of a CNN, the model extracts features from multiple layers (specifically conv4_3 and conv5_3 of a VGG16 backbone). It then applies two types of attention:
- Spatial Attention: Identifies relevant regions within the image.
- Scale-wise (Channel) Attention: Determines which feature scale is most informative for the current word being generated, effectively allowing the model to "zoom" in or out as needed.
Multi-Label Attribute Graph Convolution
To capture the relationships between objects, the authors first train a multi-label classifier to predict attributes (nouns and adjectives). They then build an attribute graph where nodes represent words and edges represent their co-occurrence frequency in the training data. A Graph Convolutional Network (GCN) processes this graph to produce "relationship-aware" attribute features, which are then integrated into the decoder.
The Decoder
A modified Long Short-Term Memory (LSTM) network serves as the decoder, integrating global image features, multi-level attended features, and GCN-refined attribute features to generate the final descriptive sentence.
Experimental Results
The model was evaluated on three major benchmarks: UCM-Captions, Sydney-Captions, and RSICD.
- State-of-the-Art Performance: The proposed method outperformed several baselines, including
FC-ATT+LSTMandSM-ATT+LSTM. On the UCM-Captions dataset, it achieved a BLEU-4 score of 0.6623 and a CIDEr score of 3.1684. - Robustness: On the large-scale RSICD dataset, the model showed superior performance across all metrics (BLEU, METEOR, ROUGE_L), proving its effectiveness in handling diverse scenes and scales.
- Ablation Studies: Results confirmed that the Multi-Level Attention and the Attribute GCN are complementary; the best performance was achieved when both modules were used together.
Conclusion and Future Work
This work demonstrates that explicitly modeling object scales and semantic relationships significantly improves remote sensing image captioning. While the current model relies on a pre-trained attribute classifier, future research could explore dynamic graphs that adapt to the specific spatial layout of individual images, further refining the model's ability to describe complex aerial scenes. Additionally, optimizing the computational overhead for real-time processing remains an important goal for practical applications.
Summary updated