Introduction
In the immediate aftermath of natural disasters, social media platforms like Twitter serve as critical sources of real-time information. However, the high volume of data presents a significant challenge for humanitarian organizations attempting to extract actionable insights. Existing disaster assessment systems often rely on a single modality (text or image), potentially overlooking complementary information. Furthermore, computational efficiency is paramount in crisis scenarios where every second counts.
This paper introduces Crisis-DIAS, a gated multimodal framework designed to automate the identification and assessment of disaster damage. By integrating text and image analysis through a sequential gating mechanism, Crisis-DIAS achieves high accuracy while significantly reducing processing time compared to traditional unimodal systems.
Methodology
Gated Multimodal Architecture
The primary innovation of Crisis-DIAS is its hierarchical gating strategy. Rather than processing every post through the entire complex model, the system operates in three sequential tasks:
- Task 1: Informativeness Identification – Filters out irrelevant posts.
- Task 2: Infrastructural Damage Identification – Identifies damage to buildings or utilities.
- Task 3: Damage Severity Assessment – Categorizes damage as mild or severe.
The outputs of the first two tasks serve as binary gates. If a post is deemed "non-informative" in Task 1, it is immediately discarded, bypassing subsequent, more computationally expensive stages. For the first two tasks, a logical OR operation is applied between the text and image classifiers to minimize false negatives, ensuring that if either modality indicates damage, the report is preserved for further analysis.
Multimodal Pipelines and Fusion
The framework utilizes specialized pipelines for each modality:
- Linguistic Pipeline: Employs a Recurrent Convolutional Neural Network (RCNN) to capture long-term dependencies and identify key phrases indicative of damage.
- Vision Pipeline: Uses an Inception-v3 model, pre-trained on ImageNet, to extract high-level spatial features from images.
For the final severity assessment (Task 3), an Attention Fusion mechanism is employed. This allows the model to dynamically weight the importance of each modality based on its descriptiveness for a given post.
Data and Experimental Setup
The framework was evaluated using CrisisMMD, a large-scale multimodal dataset covering seven major natural disasters. The dataset provides separate labels for text and images, allowing for the independent training of unimodal pipelines before their integration into the fused architecture.
Results and Discussion
Crisis-DIAS demonstrates superior performance across all tasks, achieving an F1-score of 0.99 for informativeness and damage identification, and 0.97 for severity assessment. These results significantly outperform state-of-the-art unimodal benchmarks.
Beyond accuracy, the system excels in computational efficiency. By utilizing the gating mechanism, Crisis-DIAS can process approximately 100 tweets per minute—twice the speed of traditional frameworks. This efficiency is crucial for real-time deployment in emergency response centers. Furthermore, the multimodal approach reduces bias; for instance, an ambiguous image of a storm can be correctly classified when paired with descriptive text, and vice versa.
Limitations and Future Directions
Despite its success, the current iteration of Crisis-DIAS has limitations. The system primarily focuses on infrastructural damage, currently omitting humanitarian aspects like human casualties or vehicle damage. Additionally, the hierarchical nature of the framework means that an error in the initial gating stage (Task 1) will propagate, potentially leading to the loss of critical information. Future work will explore the use of Generative Adversarial Networks (GANs) for data augmentation and the integration of rumor-detection layers to address the challenge of misinformation on social media.
Summary updated