Using Multimodal Models for Informative Classification of Ambiguous Tweets in Crisis Response

Sumiko Teng, Emily ÖhmanNAACL 2025

TL;DR

This paper tackles the challenge of sifting through noisy, ambiguous social media posts during crises. They demonstrate that multimodal models, which combine text and image information, significantly outperform single-modality models in classifying tweets as "informative" or "not informative," especially when the text and image content don't align. This advancement is crucial for improving the efficiency of humanitarian response efforts by reducing information overload.

What Problem Does This Paper Address?

Social media platforms like X (formerly Twitter) are invaluable for real-time information dissemination during crises. However, the sheer volume and often ambiguous nature of these posts create a significant challenge for humanitarian organizations trying to extract actionable intelligence. A key issue is that the text and accompanying images in a tweet might not align, leading to misinterpretations. For instance, a tweet might discuss a wildfire, but the image could be unrelated or even misleading. This misalignment makes it difficult for both humans and automated systems to accurately determine if a post is genuinely informative. Existing multimodal approaches often rely on datasets where text and image labels are pre-aligned, leaving a gap in understanding how these models perform with naturally occurring, ambiguous social media data.

Methods

The researchers propose and evaluate a multimodal classification model that leverages a hybrid fusion architecture. This approach combines a pre-trained VGG-16 model for image feature extraction and a pre-trained BERT model for textual embeddings. The core of their multimodal strategy is a cross-attention mechanism. This mechanism is designed to efficiently fuse the information from both modalities by aligning text and image embeddings. It learns to identify complementary features and filter out noise, allowing the model to focus on relevant visual and textual cues. The outputs from the text and cross-attention modules are then concatenated and passed through dense layers to produce the final classification probabilities.

For comparison, they also evaluated:

  • Text-Only Model: A fine-tuned BERT base model.
  • Image-Only Model: A fine-tuned VGG-16 model.

Datasets

The study utilizes the CrisisMMD dataset, a multimodal Twitter corpus containing manually annotated tweets and images from seven major natural disasters. For this specific research, they focused on tweets related to the California wildfires in 2017.

A key aspect of their work involved filtering the dataset to ensure consistency between text and image labels. They identified and addressed ambiguous tweets where text and image labels did not align. After this filtering and reclassification process, the dataset comprised 1,303 "informative" instances and 286 "not informative" instances. The authors note a class imbalance, with "informative" tweets being the majority.

Key Insights and Ideas

  • Multimodal Fusion is Key for Ambiguity: The most significant insight is that multimodal models, particularly those employing cross-attention, are substantially better at handling ambiguous social media content than unimodal models. This is because they can leverage complementary information from both text and images to make more robust decisions.
  • Text Modality Dominates in Ambiguous Cases: For ambiguous tweets, the text modality often carries more critical information than the image. This is evidenced by the text-only model achieving a slightly higher F1-score than the image-only model on the ambiguous dataset.
  • Real-World Noise is a Challenge: The study highlights that real-world social media data, with its inherent noise and misalignments, poses significant challenges. However, multimodal models demonstrate resilience in handling this complexity.
  • Class Imbalance is a Persistent Problem: Accurately classifying the minority "not informative" class remains a significant hurdle, even for advanced multimodal models, especially when the minority class is severely underrepresented.

Key Results

  • Overall Performance: The multimodal model consistently outperformed both text-only and image-only models across all metrics (accuracy, precision, recall, F1-score) on both the full and ambiguous datasets.
  • Full Dataset: The multimodal model achieved an accuracy of 84% and a weighted F1-score of 0.81. The text and image-only models performed similarly, with an 81% accuracy and 80% F1-score.
  • Ambiguous Dataset: The multimodal model showed a clear superiority with 90% accuracy and a 0.91 F1-score. The text-only model achieved an 82% accuracy and 0.86 F1-score, while the image-only model had 79% accuracy and 0.85 F1-score.
  • Minority Class Performance: All models struggled with the "not informative" class, particularly in the ambiguous dataset. None achieved a recall greater than 0.33 for this class, and the multimodal model failed to correctly classify any "not informative" tweets in the ambiguous subset.

Limitations and Future Work

Limitations: * Minority Class Prediction: All models faced difficulties predicting the "not informative" class, especially in the ambiguous dataset, due to its extreme rarity. This makes it hard to ascertain if the poor performance is due to model limitations or insufficient data. * Manual Annotation Subjectivity: The manual annotation of ambiguous tweets was performed by a single person, introducing potential subjectivity and inconsistencies with the original dataset's annotation criteria. * Class Imbalance Mitigation: While random oversampling was used, it might lead to overfitting on duplicated minority class instances. More sophisticated techniques like SMOTE are complex for multimodal data.

Future Work: * Data Augmentation: Collecting more data, especially for the minority class, is crucial to reduce reliance on oversampling. * Improved Annotation: Employing multiple annotators and inter-annotator agreement metrics would enhance label reliability. * Hyperparameter Tuning: Systematic hyperparameter optimization could further improve model performance. * Advanced Models: Exploring state-of-the-art multimodal architectures like CLIP could lead to better classification results.

Summary updated

BibTeX
@inproceedings{teng2025using,
  author = {Sumiko Teng and Emily Öhman},
  title = {{Using Multimodal Models for Informative Classification of Ambiguous Tweets in Crisis Response}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure