Detective Networks: Enhancing Disaster Recognition in Images Through Attention Shifting using Optimal Masking

Narongthat Thanyawet, Photchara Ratsamee, Yuki Uranishi, Haruo TakemuraWACV 2025

TL;DR

This paper introduces the "Detective Network" (DeNet), a novel approach to improve disaster recognition in aerial and shipborne imagery by intelligently masking out prominent objects to focus on smaller, crucial disaster-affected areas. DeNet significantly enhances image captioning and classification performance for disaster scenarios, achieving high accuracy in landslide detection.

What Problem Does This Paper Address?

Natural disasters like landslides, floods, and wildfires cause widespread damage, and rapid assessment of these events is crucial for effective recovery efforts. Aerial and shipborne imagery offers a valuable perspective for this assessment. However, a significant challenge arises when disaster-affected areas appear as small, inconspicuous regions within these images, making them difficult for standard computer vision models to detect and recognize. Existing methods often focus on the most prominent features in an image, potentially overlooking these critical, smaller disaster zones. Furthermore, the blurry nature of some disaster-related features can make them easily confused with natural elements. While image captioning models show promise, they tend to prioritize foreground objects, missing the broader context of disaster impact.

Methods

The core innovation is the Detective Network (DeNet), designed to redirect a model's attention from prominent objects to less obvious but critical disaster-related regions. DeNet achieves this through a two-stage masking process:

  1. Masking Candidate Layer: This layer, inspired by Region Proposal Networks (RPNs), uses patches and anchor boxes to generate potential "masked region candidates." It defines parameters for grid patches and anchor boxes within these patches, specifying their position, height, and width. This process aims to cover even small objects.
  2. Masking Layer: This layer refines the candidates from the previous stage into "optimal masks." It utilizes an attention mechanism, similar to those in Transformers, to compute an attention map. This map is then used to generate "Masked Proposals," which are combined with the original image features and scaled. Finally, a sigmoid function is applied to produce the "Optimal Masking" (OM), which highlights the regions to be focused on.

DeNet is integrated into a Vision Encoder-Decoder (VED) framework. The VED uses a Vision Transformer (ViT) as its encoder to extract image features and a BERT-based decoder to generate captions. By feeding the masked images produced by DeNet into this VED, the model is guided to focus on the critical disaster areas.

Datasets

The study utilized the following datasets:

  • British Geological Survey (BGS) [2]: Provided landslide images, which were further annotated with text captions.
  • Kaggle [1]: Contributed a large collection of common scene images used for training and validation.
  • Shipborne [16]: A dataset of landslide and non-landslide images captured during ship surveys. This dataset was used exclusively for testing and evaluation, serving as an unseen benchmark.

The paper notes that captions were generated using common descriptive terms and emphasized objects to provide context for the model.

Key Insights and Ideas

  • Masking Over Prominent Objects is Key: The research found that masking out the most prominent objects in an image is crucial for generating accurate disaster-related captions. This forces the model to look beyond the obvious and identify subtler signs of disaster.
  • Cropping is Detrimental: Unlike masking, cropping peripheral regions of an image to focus on specific areas actually removed meaningful features, leading to less accurate captions and insufficient detail.
  • Brute-Force Masking is Inefficient: While exploring all possible masking combinations (512 in a 3x3 grid) helped identify optimal masking strategies, this brute-force approach was computationally prohibitive, taking over 2 hours to process just 16 images. This highlights the need for a more efficient, learned approach like DeNet.
  • Attention Mechanisms are Powerful for Focusing: The integration of attention mechanisms within the masking layer allows the model to intelligently identify and refine the most relevant areas to focus on.

Key Results

  • Image Captioning: When DeNet was used to preprocess images for an image captioning model, it achieved a remarkable accuracy of 92.91% for landslide detection from side-view image captions and 87.50% for shipborne view detection.
  • Classification Performance: In classification tasks on the BGS dataset, DeNet outperformed other methods with an accuracy of 0.9291, precision of 0.9067, and recall of 0.9510. On the shipborne dataset, while another method (Fusion) had higher accuracy, DeNet achieved superior F1-score, recall, and precision.
  • Computational Efficiency: DeNet offers a significant speedup compared to brute-force masking. While training is more intensive, inference (including masking and captioning) reaches 3 FPS, a vast improvement over the 2 hours required for brute-force processing of 16 images.

Limitations and Future Work

  • Complex Scenes: The authors acknowledge that DeNet's ability to shift attention can be impacted by complex images containing multiple main objects.
  • Caption Dataset Limitations: The study's caption dataset was limited to 30 tokens, which affected the accuracy of DeNet and the VED model.
  • Future Directions: Future research should focus on addressing these limitations by developing methods to manage more intricate scenes with numerous significant objects and by expanding the caption dataset to overcome token limitations.

Summary updated

BibTeX
@inproceedings{thanyawet2025detective,
  author = {Narongthat Thanyawet and Photchara Ratsamee and Yuki Uranishi and Haruo Takemura},
  title = {{Detective Networks: Enhancing Disaster Recognition in Images Through Attention Shifting using Optimal Masking}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure