Abstract
The widespread use of microblogging platforms like X (formerly Twitter) during disasters provides real-time information to governments and response authorities. However, the data from these platforms is often noisy, requiring automated methods to filter relevant information. Traditionally, supervised machine learning models have been used, but they lack generalizability. In contrast, Large Language Models (LLMs) show better capabilities in understanding and processing natural language out of the box. This paper provides a detailed analysis of the performance of six well-known LLMs in processing disaster-related social media data from a large-set of real-world events. Our findings indicate that while LLMs, particularly GPT-4o and GPT-4, offer better generalizability across different disasters and information types, most LLMs face challenges in processing flood-related data, show minimal improvement despite the provision of examples (i.e., shots), and struggle to identify critical information categories like urgent requests and needs. Additionally, we examine how various linguistic features affect model performance and highlight LLMs' vulnerabilities against certain features like typos. Lastly, we provide benchmarking results for all events across both zero- and few-shot settings and observe that proprietary models outperform open-source ones in all tasks.
Introduction
During a crisis, social media platforms become vital conduits for real-time information, ranging from infrastructure damage reports to urgent pleas for help. However, the sheer volume of "chatter" makes manual filtering by human responders impossible. While traditional supervised models like BERT have been used for classification, they often suffer from "domain shift," failing when applied to disaster types or regions outside their training data. Large Language Models (LLMs) offer a potential solution through their zero-shot capabilities, but their robustness in high-stakes, noisy, and linguistically diverse crisis environments requires rigorous quantification.
Dataset and Methodology
This study utilizes the HumAID dataset, a massive collection of 77,196 human-annotated tweets from 19 major disasters between 2016 and 2019. The data spans 11 countries and includes both native and non-native English-speaking regions. Tweets are classified into 10 critical categories, such as Caution and Advice, Urgent Needs, and Infrastructure Damage.
The authors conducted a comprehensive benchmarking study comparing three proprietary models (GPT-3.5, GPT-4, GPT-4o) and three open-source models (Llama-2 13B, Llama-3 8B, Mistral 7B). The experimental design focused on zero-shot versus few-shot performance (1, 3, 5, and 10 shots), linguistic analysis of features like typos and hashtags, and cross-event generalization across disaster types.
Performance and Robustness Analysis
The Proprietary Advantage and Disaster Scaling
Proprietary models, specifically GPT-4o and GPT-4, significantly outperformed open-source alternatives across almost every metric. However, performance was not uniform across disaster types. Models consistently achieved the highest accuracy on earthquakes but struggled significantly with floods. Flood-related tweets appear more linguistically ambiguous, often overlapping with general weather chatter, which complicates classification.
The "Few-Shot" Paradox
Interestingly, the study revealed that adding examples (shots) often provided minimal or even negative gains. For instance, GPT-4's performance degraded at the 5-shot mark before recovering. This suggests that the high variability and noise in social media content can confuse a model's decision boundaries when only a few, potentially unrepresentative, examples are provided.
Critical Category Gaps
The most vital category for life-saving operations—Requests or Urgent Needs—proved to be one of the most difficult for LLMs to identify accurately. Models frequently confused "calls for help" with "reports of help being given" (volunteering), a distinction that is critical for emergency responders.
Linguistic Vulnerabilities
Linguistic features significantly impacted model reliability. The study found that:
- Hashtag Placement: Tweets with hashtags buried in the middle of a sentence caused higher error rates than those with hashtags at the beginning or end, likely disrupting the syntactic flow.
- Noise Sensitivity: Typos and special characters were statistically significant predictors of model failure.
- Geographic Bias: All models performed better on tweets from native English-speaking countries. In non-native regions, GPT-4o’s F1-score dropped from 0.76 to 0.62, highlighting a gap in handling diverse linguistic patterns.
Conclusion
While LLMs like GPT-4o show impressive generalizability, they have not yet surpassed specialized, supervised models like RoBERTa (which holds a top F1 of 0.78 compared to GPT-4o's 0.76). The vulnerabilities to linguistic noise and the difficulty in identifying urgent needs suggest that while LLMs are powerful tools, they require further refinement—perhaps through Vision-Language Models or advanced prompt engineering—before they can be fully relied upon as digital first responders.
Summary updated