Multi-Scale Semantic Segmentation and Spatial Relationship Recognition of Remote Sensing Images Based on an Attention Model

Wei Cui, Fei Wang, Xin He, Dongyou Zhang, Xuxiang Xu, Yao Meng, Ziwei Wang, Jiejun HuangRemote 2019 · DOI

TL;DR

This paper introduces the Multi-Scale Remote Sensing Image Interpretation Network (MSRIN), a novel deep learning architecture that simultaneously performs semantic segmentation of objects and recognizes their spatial relationships in remote sensing images. By leveraging multi-scale segmentation and an attention-based LSTM for image captioning, MSRIN aims to provide a more comprehensive understanding of complex scenes, crucial for applications where object context is vital.

What Problem Does This Paper Address?

Interpreting remote sensing images goes beyond simply identifying objects; understanding how these objects relate to each other spatially is critical for accurate analysis. This is particularly challenging when objects have similar spectral and textural features, making them difficult to distinguish based on appearance alone. Traditional methods often struggle to capture these complex spatial relationships in an end-to-end manner. Existing deep learning approaches, while good at semantic segmentation or image captioning individually, often fail to integrate these capabilities to simultaneously identify objects and their inter-object relationships, leading to incomplete scene understanding.

Methods

The MSRIN employs a parallel network architecture combining three key components:

  • Multi-Scale Semantic Segmentation: The network uses two segmentation modules, a Fully Convolutional Network (FCN) for large-scale objects and a U-Net for small-scale objects. This dual-scale approach captures hierarchical spatial information. The outputs are masked to obtain object locations and boundaries.
  • Attention-Based LSTM for Image Captioning: Features from a pre-trained VGG-19 network are fed into an attention-based LSTM. This module generates image captions that describe both the identified objects (nouns) and their spatial relationships.
  • Object Recognition and Correction Mechanism: A crucial innovation is the mechanism that links the generated captions to the segmented object masks. It uses an attention weight matrix to transfer spatial relationships from the captions to the object masks. To address potential mismatches due to attention location inaccuracies, a multi-scale correction algorithm is proposed. This algorithm first upscales to large-scale object regions and then downscales to identify the correct small-scale object corresponding to the noun in the caption.

Datasets

The authors used a dataset of 1835 remote sensing image patches from Guanggu, China, collected in 2009. This dataset was split into a training set (1167 samples) and a verification set (668 samples). For evaluation, they also utilized the RSICD (Remote Sensing Image Captioning Dataset) as a validation set.

Key Insights and Ideas

  • Hierarchical Interpretation: The core idea of using multi-scale segmentation (large-scale FCN and small-scale U-Net) is to build a hierarchical understanding of scenes, mirroring how we might describe a landscape from broad regions down to specific features.
  • Captioning as a Bridge: Using image captions as an intermediary to transfer spatial relationships from text to image masks is an elegant way to imbue segmented objects with contextual understanding.
  • Attention Correction is Crucial: The paper highlights that standard attention mechanisms in LSTMs can have spatial inaccuracies. The proposed multi-scale correction mechanism is vital for aligning textual descriptions with actual image objects, especially in complex scenes.
  • "With" as a Spatial Connector: The captioning strategy explicitly uses "with" to connect multiple large-scale objects, simplifying the representation of complex scene compositions.

Key Results

The MSRIN demonstrated significant improvements:

  • Matching Rate Increase: The matching rate between samples and mask graphs increased by 67.37 percentage points after correction.
  • Noun-Mask Matching: The matching rate between nouns and mask graphs increased by 41.78 percentage points after correction.
  • Quantitative Metrics: On the RSICD dataset, the proposed MSRIN outperformed a VGG-19+LSTM baseline across various metrics, including Bleu_1 (0.774 vs. 0.583), METEOR (0.359 vs. 0.261), and CIDEr (2.745 vs. 2.033).
  • Reliability: Analysis of generated captions showed high accuracy for nouns (91.08%) and spatial relationship words (89.14%).

Limitations and Future Work

  • Caption Quality Dependency: The method's effectiveness is partly dependent on the quality of the generated captions. If nouns are incorrect, the correction mechanism may be ineffective.
  • Spatial Relationship Correction: While object recognition is improved, the paper notes that if both nouns and spatial relationships are incorrect in the caption, the method is ineffective, suggesting a need for further research in this area.
  • Attention Weight Calculation: The authors plan to improve the attention weight calculation method to achieve more accurate positioning of objects.
  • Data Modality: The current work focuses on optical remote sensing images. Future work will involve verifying the mechanism with other data types, such as SAR images.
  • Evaluation Metrics: The reliance on Bleu scores for evaluation is noted as a limitation due to the burden of sample preparation. Future work will explore more suitable evaluation metrics for remote sensing image captioning.

Summary updated

Abstract

A comprehensive interpretation of remote sensing images involves not only remote sensing object recognition but also the recognition of spatial relations between objects. Especially in the case of different objects with the same spectrum, the spatial relationship can help interpret remote sensing objects more accurately. Compared with traditional remote sensing object recognition methods, deep learning has the advantages of high accuracy and strong generalizability regarding scene classification and semantic segmentation. However, it is difficult to simultaneously recognize remote sensing objects and their spatial relationship from end-to-end only relying on present deep learning networks. To address this problem, we propose a multi-scale remote sensing image interpretation network, called the MSRIN. The architecture of the MSRIN is a parallel deep neural network based on a fully convolutional network (FCN), a U-Net, and a long short-term memory network (LSTM). The MSRIN recognizes remote sensing objects and their spatial relationship through three processes. First, the MSRIN defines a multi-scale remote sensing image caption strategy and simultaneously segments the same image using the FCN and U-Net on different spatial scales so that a two-scale hierarchy is formed. The output of the FCN and U-Net are masked to obtain the location and boundaries of remote sensing objects. Second, using an attention-based LSTM, the remote sensing image captions include the remote sensing objects (nouns) and their spatial relationships described with natural language. Finally, we designed a remote sensing object recognition and correction mechanism to build the relationship between nouns in captions and object mask graphs using an attention weight matrix to transfer the spatial relationship from captions to objects mask graphs. In other words, the MSRIN simultaneously realizes the semantic segmentation of the remote sensing objects and their spatial relationship identification end-to-end. Experimental results demonstrated that the matching rate between samples and the mask graph increased by 67.37 percentage points, and the matching rate between nouns and the mask graph increased by 41.78 percentage points compared to before correction. The proposed MSRIN has achieved remarkable results.
BibTeX
@inproceedings{cui2019multi,
  author = {Wei Cui and Fei Wang and Xin He and Dongyou Zhang and Xuxiang Xu and Yao Meng and Ziwei Wang and Jiejun Huang},
  title = {{Multi-Scale Semantic Segmentation and Spatial Relationship Recognition of Remote Sensing Images Based on an Attention Model}},
  booktitle = {Remote},
  year = {2019},
  doi = {10.3390/rs11091044},
}
Figure