CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informatics

Kai Yin, Bo Li, Chengkai Liu, Ali Mostafavi, Xia Hu

TL;DR

CrisisSense-LLM is a specialized framework that fine-tunes Large Language Models (LLMs) to simultaneously classify multiple dimensions of disaster-related social media posts—such as event type, informativeness, and specific humanitarian needs. By moving from traditional single-label models to a unified multi-label instruction-tuned approach, the researchers achieved a 15x improvement in accuracy over zero-shot baselines, reaching 63.8% across complex, overlapping categories.

What Problem Does This Paper Address?

In the chaos of a natural disaster, social media platforms like X (formerly Twitter) become vital lifelines for real-time situational awareness. However, the sheer volume of data is overwhelming for human responders.

Existing AI tools for "disaster informatics" typically rely on single-label classification. For example, one model might decide if a tweet is "informative," while a completely separate model identifies the "event type" (e.g., flood vs. earthquake). This approach is inefficient and ignores the multifaceted nature of crisis communication—a single tweet might report a flood, confirm infrastructure damage, and request urgent medical aid all at once. The authors address this gap by developing a model that understands these inter-related insights in a single pass.

Methods

The researchers developed a three-stage pipeline to transform a general-purpose LLM into a disaster-specialist tool:

  1. Instruction Dataset Construction: They converted over 60,000 human-annotated tweets into (Instruction, Input, Output) triplets. This "instructionalizes" the data, teaching the model not just to label, but to follow a specific task format.
  2. Instruction Fine-Tuning: They tested two primary paths using Llama 2 (both the base and chat-optimized versions):
  3. LoRA (Low-Rank Adaptation): A parameter-efficient method that freezes the main model weights and only trains a small set of external matrices.
  4. Full-Parameter Tuning: Updating all model weights using 3D parallelism (model, tensor, and data parallelism) across 16 A100 GPUs.
  5. Inference & Ensemble: To boost reliability, they implemented a majority-vote mechanism across multiple model checkpoints to arrive at a final classification.
Figure 2. The proposed overall framework for Disaster LLM: multi-label disaster

The study also highlights where fine-tuning fits into the broader landscape of LLM optimization, as seen in Figure 1:

Figure 1. Three approaches to improve the power LLM (figure adapted from Grootendorst,

Datasets

The study utilizes CrisisBench, a comprehensive repository of human-annotated disaster tweets.

  • Scale: 60,000+ samples.
  • Labels: The researchers processed the data to include three concurrent labels for every instance:
  • Event Type: 14 categories (e.g., Hurricane, Flood, Earthquake).
  • Informativeness: Binary (Informative vs. Not Informative).
  • Humanitarian Aid Type: 16 categories (e.g., Requests/Needs, Infrastructure Damage, Missing People).
Distribution of labels across the three categories in the CrisisBench dataset.

Key Insights and Ideas

  • The Synergy of Multi-task Learning: The authors found that training the model on multiple labels simultaneously actually helps performance. Understanding the "human aid type" (e.g., medical request) provides contextual clues that help the model identify the "event type" (e.g., earthquake) and vice versa.
  • Chat Models vs. Base Models: In LoRA (parameter-efficient) settings, the Llama 2-Chat model outperformed the base model because it was already "pre-aligned" to follow instructions. However, with full-parameter tuning, the base model eventually reached higher peaks, likely because it had more "unstructured" capacity to absorb domain-specific knowledge.
  • Extreme Prompt Sensitivity: A critical finding was that the model learns the structure of the prompt as much as the task itself. If the prompt used during inference differed from the one used during training, performance dropped by over 50%.
  • Catastrophic Forgetting: The researchers observed that as the model learned to be a disaster expert, it began to "forget" how to explain its reasoning, eventually outputting only the required JSON labels.

Key Results

  • Massive Accuracy Gains: The fine-tuned model achieved an overall accuracy of 63.8% (meaning all three labels were exactly correct). For comparison, the zero-shot Llama 2-Chat model achieved only 4.27%.
  • Category Performance: Accuracy was highest for "Informativeness" (~80-90% in some checkpoints) and lowest for "Humanitarian Aid Type," which is the most complex category with 16 distinct labels.
  • Efficiency: LoRA with a rank of 32 was identified as a "sweet spot," providing near-state-of-the-art performance with significantly fewer computational resources than full-parameter tuning.

Limitations and Future Work

  • Text-Only Focus: Currently, CrisisSense-LLM only processes text. Real-world social media is multimodal; future iterations should incorporate image analysis (e.g., photos of flood damage) to complement the text.
  • Prompt Rigidity: The model's high sensitivity to prompt formatting makes it less "plug-and-play" for users who might use different phrasing.
  • Real-time Deployment: While the accuracy is high, the paper notes that the risk of "hallucinations" or invalid responses (returning "None") still requires robust parsing algorithms before these models can be fully automated in emergency operations centers.

Summary updated

Abstract

In the field of crisis/disaster informatics, social media is increasingly being used for improving situational awareness to inform response and relief efforts. Efficient and accurate text classification tools have been a focal area of investigation in crisis informatics. However, current methods mostly rely on single-label text classification models, which fails to capture different insights embedded in dynamic and multifaceted disaster-related social media data. This study introduces a novel approach to disaster text classification by enhancing a pre-trained Large Language Model (LLM) through instruction fine-tuning targeted for multi-label classification of disaster-related tweets. Our methodology involves creating a comprehensive instruction dataset from disaster-related tweets, which is then used to fine-tune an open-source LLM, thereby embedding it with disaster-specific knowledge. This fine-tuned model can classify multiple aspects of disaster-related information simultaneously, such as the type of event, informativeness, and involvement of human aid, significantly improving the utility of social media data for situational awareness in disasters. The results demonstrate that this approach enhances the categorization of critical information from social media posts, thereby facilitating a more effective deployment for situational awareness during emergencies. This research paves the way for more advanced, adaptable, and robust disaster management tools, leveraging the capabilities of LLMs to improve real-time situational awareness and response strategies in disaster scenarios.
BibTeX
@article{yin2026crisissense,
  author = {Kai Yin and Bo Li and Chengkai Liu and Ali Mostafavi and Xia Hu},
  title = {{CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informatics}},
  journal = {Adv. Eng. Informatics},
  year = {2026},
  doi = {10.1016/j.aei.2026.104836},
}
Figure