TL;DR
This paper evaluates the robustness of six prominent Large Language Models (LLMs) in classifying disaster-related social media posts across 19 real-world events. While proprietary models like GPT-4o show strong generalizability, the study reveals critical weaknesses in handling flood-related data, identifying urgent humanitarian needs, and processing noisy linguistic features like mid-sentence hashtags and typos.
What Problem Does This Paper Address?
During large-scale disasters, microblogging platforms like X (formerly Twitter) become vital lifelines for real-time information. However, the sheer volume of data is overwhelming and highly noisy, filled with irrelevant chatter. Traditionally, supervised machine learning models have been used to filter this data, but they often fail when applied to new disasters or different geographic regions due to "domain shift."
LLMs offer a potential solution thanks to their zero-shot capabilities, but their actual reliability in high-stakes humanitarian contexts remains under-explored. This paper seeks to quantify how well LLMs handle the specific linguistic messiness of crisis data—such as slang, typos, and varying disaster types—across both native and non-native English-speaking regions.
Methods
The authors benchmarked three proprietary models (GPT-3.5, GPT-4, GPT-4o) and three open-source models (Llama-2 13B, Llama-3 8B, Mistral 7B). They evaluated these models on a multi-class classification task using two primary settings:
- Zero-shot: The model classifies tweets based solely on category definitions.
- Few-shot: The model is provided with 1, 3, 5, or 10 examples per class to guide its performance.
Beyond standard accuracy metrics, the researchers employed logistic regression to analyze how 17 different linguistic features (e.g., word count, hashtags, emojis, typos) correlate with model success or failure. They also specifically tested the impact of hashtag positioning (beginning, middle, or end of a tweet) on classification accuracy.
Datasets
The study utilizes the HumAID dataset, a large-scale collection of 77,196 human-annotated tweets from 19 major natural disasters (earthquakes, hurricanes, wildfires, and floods) occurring between 2016 and 2019 across 11 countries.
The tweets are categorized into ten information types, including "Caution and advice," "Requests or urgent needs," and "Infrastructure damage." The authors focused their analysis on a test split of 15,160 tweets.
Key Insights and Ideas
- Proprietary Dominance: Proprietary models consistently outperformed open-source alternatives. GPT-4o and GPT-4 demonstrated the best generalizability across different disaster types.
- The "Flood" Difficulty: Models performed significantly better on earthquake data than on flood data. The authors suggest that the language used during floods might be more ambiguous or less distinct than the formulaic reports often seen during earthquakes.
- Few-Shot Diminishing Returns: Surprisingly, increasing the number of examples (shots) did not consistently improve performance. For some models, more examples actually introduced confusion, likely because the high variability of social media language makes a few "representative" examples less effective than in cleaner datasets.
- Linguistic Fragility:
- Hashtags: Placing hashtags in the middle of a sentence significantly degrades performance, particularly for proprietary models, as it disrupts the syntactic structure.
- Noise: Features like typos and special characters were negatively correlated with model performance, highlighting a lack of robustness to common social media "noise."
- Semantic Confusion: Models frequently confused "Requests or urgent needs" (RUN) with "Rescue volunteering or donation efforts" (RVDE). This is a critical failure, as misclassifying a plea for help as a general donation report could delay emergency response.
Key Results
- Overall Performance: GPT-4o achieved the highest zero-shot F1-score of 0.76, coming close to the state-of-the-art supervised baseline (RoBERTa at 0.78) without requiring task-specific training.
- Disaster Type Variance: As shown in the radar charts below, earthquakes were the "easiest" for models to handle, while floods consistently yielded the lowest scores.
- Information Type Challenges: Most models struggled to identify "Requests or urgent needs," with even the best GPT models often falling below an F1-score of 0.60 for this category.
- Language Gap: Performance dropped notably when processing data from non-native English-speaking countries, even though the tweets themselves were in English.
Limitations and Future Work
The authors acknowledge that their study is limited to text-only analysis. Real-world social media is increasingly multimodal, containing images and videos that provide crucial context during crises.
Future directions include: * Evaluating Large Vision-Language Models (LVLMs) to incorporate visual data. * Conducting deeper qualitative analyses to understand why specific disaster types like floods are harder to classify. * Developing methods to improve LLM robustness against mid-sentence hashtags and other informal linguistic structures common in microblogs.
Summary updated