Disaster Image Classification by Fusing Multimodal Social Media Data

Zhiqiang Zou, Hongyu Gan, Qunying Huang, Tianhui Cai, Kai CaoISPRS Int. J. Geo Inf 2021 · OpenReview

TL;DR

This paper introduces a novel multimodal fusion approach to classify disaster-related images from social media by combining visual and textual features. Their method significantly outperforms existing unimodal and multimodal approaches, achieving an accuracy of 87.6% on the CrisisMMD dataset, which is crucial for improving disaster response and situational awareness.

What Problem Does This Paper Address?

In the aftermath of natural disasters, social media platforms become a deluge of information, often containing critical details about the situation. However, this data is frequently multimodal (images and text) and riddled with irrelevant or misleading content. Existing methods often rely on analyzing either just images or just text (unimodal), or they attempt to combine them with varying degrees of success. This leaves a gap in effectively and accurately extracting actionable disaster information from the rich, yet noisy, multimodal social media landscape, hindering timely and efficient disaster management.

Methods

The core of this research is a multimodal fusion model designed to classify disaster images. It comprises three key modules:

  1. Image Feature Extractor: This module uses a pre-trained VGG16 convolutional neural network (CNN) to extract visual features from images. A key modification here is adjusting the parameter size in the second-to-last fully connected layer to 500 (down from 1000) to simplify the model and reduce computational load without sacrificing accuracy.
  2. Textual Feature Extractor: Instead of more complex CNNs, the authors opt for FastText to extract textual features. They highlight FastText's advantages: faster training/testing with comparable accuracy to CNNs, and its ability to generate word embeddings automatically, eliminating the need for pre-trained embeddings. The output is a 500-dimensional vector representing the average of word embeddings.
  3. Multimodal Fusion Module: This module takes the extracted visual and textual features and combines them. The approach is a straightforward concatenation of the two 500-dimensional feature vectors into a single 1000-dimensional vector. This combined vector is then fed through three fully connected layers and a SoftMax layer for the final classification. This simple concatenation is presented as an advantage over more complex eigenvector alignment methods.

Datasets

The primary dataset used for training and evaluation is CrisisMMD. This is a multimodal Twitter dataset containing tweets and images collected during seven major natural disaster events (e.g., Hurricane Irma, Mexico Earthquake). The dataset is annotated for three different classification tasks:

  • Task 1: Distinguishing between "Informative" and "Not informative" disaster-related content.
  • Task 2: Categorizing informative content into specific sub-types like "Affected individuals," "Infrastructure and utility damage," etc.
  • Task 3: Assessing the severity of damage ("Severe damage," "Mild damage," "Little or no damage").

The authors filtered the dataset to ensure that only pairs of text and images with the same label were used for experiments, and they also combined certain categories in Task 2 to address data sparsity.

Key Insights and Ideas

  • Multimodality is Key: The research strongly reinforces the idea that combining image and text data is superior to using either modality alone for disaster classification. The multimodal approach consistently yields higher accuracy.
  • Simplicity Can Be Powerful: The authors advocate for a simpler model architecture, particularly in the fusion stage (simple concatenation) and by adjusting VGG16's parameters. This simplicity makes the model easier to train and potentially more efficient.
  • FastText's Practicality: Choosing FastText for text feature extraction highlights its efficiency and ease of use, especially in scenarios where rapid processing and automatic embedding generation are beneficial.
  • Data Imbalance is a Challenge: The study acknowledges that unbalanced datasets, particularly in Task 3 (damage severity), significantly impact accuracy. This points to a common challenge in real-world data analysis.

Key Results

The proposed multimodal fusion method demonstrated significant improvements across all tasks:

  • Task 1 (Informative vs. Not Informative): The multimodal approach achieved an accuracy of 87.6%, outperforming unimodal methods (83.3% for images, 85.2% for text) and previous state-of-the-art by over 3%.
  • Task 2 (Categorization): In the first step, multimodal accuracy reached 92.6%, surpassing unimodal methods. In the second step, it achieved 91.2%, which was better than the image-only model but slightly lower than the text-only model (though the authors argue text-only lacks reliability).
  • Task 3 (Damage Severity): The initial multimodal accuracy was 68.9%. However, after increasing the number of samples for the "L" (Little or no damage) category, the accuracy improved to 79.6%.

The authors also highlight that their multimodal approach missed significantly fewer instances compared to unimodal methods in Task 1, indicating better overall classification.

Limitations and Future Work

  • Data Imbalance: The authors explicitly mention that the low accuracy in Task 3 was due to an unbalanced dataset. They suggest that increasing the number of samples for underrepresented categories is crucial.
  • Spatio-Temporal Information: While geospatial data was used in text feature extraction, the authors believe that further optimization can be achieved by mining spatio-temporal information more effectively. This could involve mapping locations of need or damage severity.
  • Data Fusion Techniques: The paper proposes exploring novel deep learning methods for fusing multimodal data beyond simple concatenation.
  • Handling Unbalanced Data: Developing robust approaches to address the problem of unbalanced samples is identified as a key area for future work.
  • Generative Adversarial Networks (GANs): The authors suggest exploring GANs for damage severity assessment, particularly for handling minimal data scenarios.

Summary updated

Abstract

Social media datasets have been widely used in disaster assessment and management. When a disaster occurs, many users post messages in a variety of formats, e.g., image and text, on social media platforms. Useful information could be mined from these multimodal data to enable situational awareness and to support decision making during disasters. However, the multimodal data collected from social media contain a lot of irrelevant and misleading content that needs to be filtered out. Existing work has mostly used unimodal methods to classify disaster messages. In other words, these methods treated the image and textual features separately. While a few methods adopted multimodality to deal with the data, their accuracy cannot be guaranteed. This research seamlessly integrates image and text information by developing a multimodal fusion approach to identify useful disaster images collected from social media platforms. In particular, a deep learning method is used to extract the visual features from social media, and a FastText framework is then used to extract the textual features. Next, a novel data fusion model is developed to combine both visual and textual features to classify relevant disaster images. Experiments on a real-world disaster dataset, CrisisMMD, are performed, and the validation results demonstrate that the method consistently and significantly outperforms the previously published state-of-the-art work by over 3%, with a performance improvement from 84.4% to 87.6%.
BibTeX
@article{zou2021disaster,
  author = {Zhiqiang Zou and Hongyu Gan and Qunying Huang and Tianhui Cai and Kai Cao},
  title = {{Disaster Image Classification by Fusing Multimodal Social Media Data}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2021},
  url = {https://openreview.net/forum?id=NZXJHvIKmJ},
}
Figure