Evaluating Robustness of LLMs on Crisis-Related Microblogs across Events, Information Types, and Linguistic Features

Muhammad Imran, Abdul Wahab Ziaullah, Kai Chen, Ferda Ofli

TL;DR

This paper evaluates the robustness of six prominent Large Language Models (LLMs) in classifying disaster-related social media posts across 19 real-world events. While proprietary models like GPT-4o show strong generalizability, the study reveals critical weaknesses in handling flood-related data, identifying urgent humanitarian needs, and processing noisy linguistic features like mid-sentence hashtags and typos.

What Problem Does This Paper Address?

During large-scale disasters, microblogging platforms like X (formerly Twitter) become vital lifelines for real-time information. However, the sheer volume of data is overwhelming and highly noisy, filled with irrelevant chatter. Traditionally, supervised machine learning models have been used to filter this data, but they often fail when applied to new disasters or different geographic regions due to "domain shift."

LLMs offer a potential solution thanks to their zero-shot capabilities, but their actual reliability in high-stakes humanitarian contexts remains under-explored. This paper seeks to quantify how well LLMs handle the specific linguistic messiness of crisis data—such as slang, typos, and varying disaster types—across both native and non-native English-speaking regions.

Methods

The authors benchmarked three proprietary models (GPT-3.5, GPT-4, GPT-4o) and three open-source models (Llama-2 13B, Llama-3 8B, Mistral 7B). They evaluated these models on a multi-class classification task using two primary settings:

  • Zero-shot: The model classifies tweets based solely on category definitions.
  • Few-shot: The model is provided with 1, 3, 5, or 10 examples per class to guide its performance.

Beyond standard accuracy metrics, the researchers employed logistic regression to analyze how 17 different linguistic features (e.g., word count, hashtags, emojis, typos) correlate with model success or failure. They also specifically tested the impact of hashtag positioning (beginning, middle, or end of a tweet) on classification accuracy.

Datasets

The study utilizes the HumAID dataset, a large-scale collection of 77,196 human-annotated tweets from 19 major natural disasters (earthquakes, hurricanes, wildfires, and floods) occurring between 2016 and 2019 across 11 countries.

Figure 1: Data distributions for (a) events, (b) information types/classes, (c) disaster types, and (d) native/non-native English countries

The tweets are categorized into ten information types, including "Caution and advice," "Requests or urgent needs," and "Infrastructure damage." The authors focused their analysis on a test split of 15,160 tweets.

Key Insights and Ideas

  • Proprietary Dominance: Proprietary models consistently outperformed open-source alternatives. GPT-4o and GPT-4 demonstrated the best generalizability across different disaster types.
  • The "Flood" Difficulty: Models performed significantly better on earthquake data than on flood data. The authors suggest that the language used during floods might be more ambiguous or less distinct than the formulaic reports often seen during earthquakes.
  • Few-Shot Diminishing Returns: Surprisingly, increasing the number of examples (shots) did not consistently improve performance. For some models, more examples actually introduced confusion, likely because the high variability of social media language makes a few "representative" examples less effective than in cleaner datasets.
  • Linguistic Fragility:
    • Hashtags: Placing hashtags in the middle of a sentence significantly degrades performance, particularly for proprietary models, as it disrupts the syntactic structure.
  • Noise: Features like typos and special characters were negatively correlated with model performance, highlighting a lack of robustness to common social media "noise."
  • Semantic Confusion: Models frequently confused "Requests or urgent needs" (RUN) with "Rescue volunteering or donation efforts" (RVDE). This is a critical failure, as misclassifying a plea for help as a general donation report could delay emergency response.

Key Results

  • Overall Performance: GPT-4o achieved the highest zero-shot F1-score of 0.76, coming close to the state-of-the-art supervised baseline (RoBERTa at 0.78) without requiring task-specific training.
  • Disaster Type Variance: As shown in the radar charts below, earthquakes were the "easiest" for models to handle, while floods consistently yielded the lowest scores.
Figure 2: Performance (F1-scores) of LLMs across disaster types and few-shot settings
  • Information Type Challenges: Most models struggled to identify "Requests or urgent needs," with even the best GPT models often falling below an F1-score of 0.60 for this category.
  • Language Gap: Performance dropped notably when processing data from non-native English-speaking countries, even though the tweets themselves were in English.
Performance across different information types for GPT-4o (left) and Mistral 7B (right).

Limitations and Future Work

The authors acknowledge that their study is limited to text-only analysis. Real-world social media is increasingly multimodal, containing images and videos that provide crucial context during crises.

Future directions include: * Evaluating Large Vision-Language Models (LVLMs) to incorporate visual data. * Conducting deeper qualitative analyses to understand why specific disaster types like floods are harder to classify. * Developing methods to improve LLM robustness against mid-sentence hashtags and other informal linguistic structures common in microblogs.

Summary updated

Abstract

The widespread use of microblogging platforms like X (formerly Twitter) during disasters provides real-time information to governments and response authorities. However, the data from these platforms is often noisy, requiring automated methods to filter relevant information. Traditionally, supervised machine learning models have been used, but they lack generalizability. In contrast, Large Language Models (LLMs) show better capabilities in understanding and processing natural language out of the box. This paper provides a detailed analysis of the performance of six well-known LLMs in processing disaster-related social media data from a large-set of real-world events. Our findings indicate that while LLMs, particularly GPT-4o and GPT-4, offer better generalizability across different disasters and information types, most LLMs face challenges in processing flood-related data, show minimal improvement despite the provision of examples (i.e., shots), and struggle to identify critical information categories like urgent requests and needs. Additionally, we examine how various linguistic features affect model performance and highlight LLMs' vulnerabilities against certain features like typos. Lastly, we provide benchmarking results for all events across both zero- and few-shot settings and observe that proprietary models outperform open-source ones in all tasks.
BibTeX
@misc{imran2024evaluating,
  author = {Muhammad Imran and Abdul Wahab Ziaullah and Kai Chen and Ferda Ofli},
  title = {{Evaluating Robustness of LLMs on Crisis-Related Microblogs across Events, Information Types, and Linguistic Features}},
  year = {2024},
  eprint = {2412.10413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.10413},
}
Figure