Crisis-DIAS: Towards Multimodal Damage Analysis - Deployment, Challenges and Assessment

Mansi Agarwal, Maitree Leekha, Ramit Sawhney, Rajiv Ratn ShahAAAI 2020 · DOI

Introduction

In the immediate aftermath of natural disasters, social media platforms like Twitter serve as critical sources of real-time information. However, the high volume of data presents a significant challenge for humanitarian organizations attempting to extract actionable insights. Existing disaster assessment systems often rely on a single modality (text or image), potentially overlooking complementary information. Furthermore, computational efficiency is paramount in crisis scenarios where every second counts.

This paper introduces Crisis-DIAS, a gated multimodal framework designed to automate the identification and assessment of disaster damage. By integrating text and image analysis through a sequential gating mechanism, Crisis-DIAS achieves high accuracy while significantly reducing processing time compared to traditional unimodal systems.

Methodology

Gated Multimodal Architecture

The primary innovation of Crisis-DIAS is its hierarchical gating strategy. Rather than processing every post through the entire complex model, the system operates in three sequential tasks:

  1. Task 1: Informativeness Identification – Filters out irrelevant posts.
  2. Task 2: Infrastructural Damage Identification – Identifies damage to buildings or utilities.
  3. Task 3: Damage Severity Assessment – Categorizes damage as mild or severe.
The Crisis-DIAS gated multimodal framework for sequential damage identification and severity detection.

The outputs of the first two tasks serve as binary gates. If a post is deemed "non-informative" in Task 1, it is immediately discarded, bypassing subsequent, more computationally expensive stages. For the first two tasks, a logical OR operation is applied between the text and image classifiers to minimize false negatives, ensuring that if either modality indicates damage, the report is preserved for further analysis.

Multimodal Pipelines and Fusion

The framework utilizes specialized pipelines for each modality:

  • Linguistic Pipeline: Employs a Recurrent Convolutional Neural Network (RCNN) to capture long-term dependencies and identify key phrases indicative of damage.
  • Vision Pipeline: Uses an Inception-v3 model, pre-trained on ImageNet, to extract high-level spatial features from images.
Detailed architecture of the multimodal fusion component, showing the integration of RCNN for text and Inception blocks for images.

For the final severity assessment (Task 3), an Attention Fusion mechanism is employed. This allows the model to dynamically weight the importance of each modality based on its descriptiveness for a given post.

Data and Experimental Setup

The framework was evaluated using CrisisMMD, a large-scale multimodal dataset covering seven major natural disasters. The dataset provides separate labels for text and images, allowing for the independent training of unimodal pipelines before their integration into the fused architecture.

Results and Discussion

Crisis-DIAS demonstrates superior performance across all tasks, achieving an F1-score of 0.99 for informativeness and damage identification, and 0.97 for severity assessment. These results significantly outperform state-of-the-art unimodal benchmarks.

Beyond accuracy, the system excels in computational efficiency. By utilizing the gating mechanism, Crisis-DIAS can process approximately 100 tweets per minute—twice the speed of traditional frameworks. This efficiency is crucial for real-time deployment in emergency response centers. Furthermore, the multimodal approach reduces bias; for instance, an ambiguous image of a storm can be correctly classified when paired with descriptive text, and vice versa.

Limitations and Future Directions

Despite its success, the current iteration of Crisis-DIAS has limitations. The system primarily focuses on infrastructural damage, currently omitting humanitarian aspects like human casualties or vehicle damage. Additionally, the hierarchical nature of the framework means that an error in the initial gating stage (Task 1) will propagate, potentially leading to the loss of critical information. Future work will explore the use of Generative Adversarial Networks (GANs) for data augmentation and the integration of rumor-detection layers to address the challenge of misinformation on social media.

Summary updated

BibTeX
@inproceedings{agarwal2020crisis,
  author = {Mansi Agarwal and Maitree Leekha and Ramit Sawhney and Rajiv Ratn Shah},
  title = {{Crisis-DIAS: Towards Multimodal Damage Analysis - Deployment, Challenges and Assessment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i01.5369},
}
Figure