Landslide Image Captioning Method Based on Semantic Gate and Bi-Temporal LSTM

Wenqi Cui, Xin He, Meng Yao, Ziwei Wang, Jie Li, Yuanjie Hao, Weijie Wu, Huiling Zhao, Xianfeng Chen, Wei CuiISPRS Int. J. Geo Inf 2020 · DOI

TL;DR

This paper introduces a novel deep learning model, SG-BiTLSTM, for simultaneously identifying landslides and their surrounding hazard-affected bodies using remote sensing imagery. By combining semantic segmentation with a bi-temporal LSTM and a unique semantic gate, the model generates descriptive captions that pinpoint critical infrastructure like roads and buildings, crucial for emergency response.

What Problem Does This Paper Address?

When landslides occur, understanding not just the landslide's extent but also the spatial relationships with surrounding objects (like roads and buildings) is vital for effective risk assessment and emergency rescue. Traditional methods often struggle with this, either focusing on segmentation alone or lacking the nuanced understanding of spatial relationships. Image captioning offers a way to capture this richer semantic information, but existing methods suffer from accumulated errors during prediction and inaccurate attention mechanisms, leading to potential misjudgments of risk. This paper aims to bridge this gap by developing a more robust and accurate system for interpreting landslide-affected areas.

Methods

The proposed SG-BiTLSTM model is built upon two core components:

  • U-Net Encoder: This network extracts features from remote sensing images and generates semantic segmentation maps, identifying different geographic objects (landslides, buildings, roads, etc.).
  • Bi-Temporal LSTM Decoder: This is the novel part, consisting of two interconnected LSTMs (a language LSTM and a prediction LSTM) and a semantic gate.
  • The bi-temporal nature allows the model to consider information from two time steps, helping to mitigate accumulated errors in caption generation.
  • The semantic gate is a key innovation. It dynamically controls whether the model focuses more on image features or semantic context when generating words in the caption. This is achieved using a customized activation function that adapts based on whether the predicted word is a noun (object) or a relationship word.
  • The model also incorporates an attention correction mechanism to improve the accuracy of locating objects within the images.

Datasets

The study utilizes remote sensing data from the Worldview-1 satellite, captured after the Wenchuan earthquake in 2008. The images have a spatial resolution of 0.5m. The data is divided into 7 classes: landslide, road, greenland, agriculture, building, river, and others. For training and validation, 2910 samples were manually cropped to 224x224 pixels, with 1925 used for training and 985 for validation.

Key Insights and Ideas

  • Bridging Segmentation and Captioning: The paper effectively demonstrates how to fuse semantic segmentation (identifying objects) with image captioning (describing relationships) to provide a more comprehensive understanding of disaster-affected areas.
  • The Power of the Semantic Gate: The semantic gate is crucial for adaptively focusing on relevant information. By dynamically switching between image features and semantic context, it prevents the model from being misled by irrelevant visual cues or contextual ambiguities.
  • Mitigating Accumulated Error: The bi-temporal LSTM architecture, by considering information from two time steps, offers a promising approach to reduce the cascading errors common in sequential prediction tasks like image captioning.
  • Improved Location Accuracy: The attention correction mechanism, coupled with the semantic gate, leads to more precise localization of identified objects, which is critical for actionable information in disaster scenarios.

Key Results

The SG-BiTLSTM model demonstrated significant improvements over baseline models:

  • BLEU-1 Score: Achieved the highest BLEU-1 score of 0.8611, a 5.89% improvement compared to the traditional LSTM baseline.
  • Noun-Object Matching Accuracy: Reached an impressive 87.19% matching accuracy, a substantial increase from the baseline's 44.38%.
  • Location Accuracy: Showed superior performance in both "multiple to multiple" (77.86%) and "1 to 1" (91.54%) sample scenarios compared to the baseline.
  • Semantic Segmentation Accuracy: The total accuracy for semantic segmentation was 0.93, with specific accuracies for landslides, buildings, and roads at 0.94, 0.91, and 0.87, respectively.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Data Source: The reliance on single-temporal remote sensing images limits the ability to assess landslide magnitude.
  • Vegetation Cover: Landslides covered by vegetation are difficult to recognize using spectral and texture information alone.
  • Future Work:
    • Combining deep learning with remote sensing and landslide domain knowledge.
  • Investigating multi-temporal remote sensing images for change detection.
  • Exploring methods to better handle landslides obscured by vegetation.

Summary updated

BibTeX
@article{cui2020landslide,
  author = {Wenqi Cui and Xin He and Meng Yao and Ziwei Wang and Jie Li and Yuanjie Hao and Weijie Wu and Huiling Zhao and Xianfeng Chen and Wei Cui},
  title = {{Landslide Image Captioning Method Based on Semantic Gate and Bi-Temporal LSTM}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2020},
  doi = {10.3390/ijgi9040194},
}
Figure