On Identifying Hashtags in Disaster Twitter Data

Jishnu Ray Chowdhury, Cornelia Caragea, Doina Caragea

TL;DR

This paper tackles the challenge of extracting useful hashtags from disaster-related tweets, which are often missing or too general. They introduce a novel, large-scale annotated dataset for disaster hashtag extraction and propose a Multi-Task Learning (MTL) framework based on LSTMs, achieving an impressive F1-score of 92.22%. This work is crucial for improving disaster response by enabling better filtering and retrieval of critical information from social media.

What Problem Does This Paper Address?

During disasters, social media platforms like Twitter become vital sources of real-time information. However, the sheer volume and unstructured nature of these posts make it difficult for emergency responders to quickly find actionable insights. A key challenge is the unreliability of user-provided hashtags: many disaster tweets lack them entirely, and those that do often use hashtags that are too broad (e.g., just the disaster name) or too specific, failing to capture crucial details like damage reports, requests for help, or situational awareness. This makes traditional keyword or hashtag-based search methods ineffective, leading to missed critical information and delayed response efforts. The paper aims to bridge this gap by developing methods to automatically identify and extract relevant and topically informative hashtags directly from the tweet content.

Methods

The core of the paper's methodology revolves around two main components: dataset creation and model development.

Dataset Creation: The authors address the lack of a suitable benchmark dataset by constructing a large, unique dataset of disaster-related tweets. This dataset was created by:

  1. Collecting existing disaster tweet datasets: They aggregated data from various sources covering a wide range of disaster types (hurricanes, earthquakes, floods, wildfires, etc.).
  2. Filtering and cleaning: They removed irrelevant or uninformative tweets and non-English content.
  3. Semi-automated annotation: They developed a disaster lexicon and used it to automatically annotate tweets with relevant hashtags. This was followed by a quality assessment involving human annotators to validate the annotations.

Model Development: The authors propose and evaluate a Long Short-Term Memory (LSTM)-based Multi-Task Learning (MTL) framework for hashtag extraction. This approach is framed as a sequence labeling task, similar to keyphrase extraction, allowing the model to discover new hashtags not seen during training. They explore several variants of this model:

  • LSTM-MTL: A baseline LSTM-MTL model that stacks two Bi-LSTM layers, one for word-level identification and another for phrase-level labeling.
  • LSTM-MTL+ELMO: Incorporates contextualized ELMO word embeddings to better capture tweet context.
  • LSTM-MTL+IPA,POS: This variant aims to handle informal writing by incorporating phonetic (IPA) and part-of-speech (POS) information, using character-level CNNs for phonetic feature extraction.
  • LSTM-MTL+ELMO,IPA,POS: The best-performing model, which combines ELMO embeddings with IPA/POS features.

Datasets

  • Newly Introduced Benchmark Dataset: The paper introduces a large, unique dataset of 67,288 disaster-related tweets annotated with relevant and topically informative hashtags. This dataset is a significant contribution as it specifically targets disaster scenarios, unlike previous datasets that often used user-provided hashtags as gold-standard.
  • Existing Datasets: The authors leveraged numerous publicly available disaster tweet datasets from various sources (e.g., Imran et al., Olteanu et al., Alam et al.) as the foundation for their new benchmark.
  • CrisisLex lexicon: Used as a source for initial lexicon-based annotation.

Key Insights and Ideas

  • User-provided hashtags are insufficient: The paper strongly demonstrates that relying solely on user-generated hashtags for disaster information retrieval is highly problematic due to their generality, specificity, or complete absence.
  • Extractive approach is key: Unlike classification-based hashtag recommendation, an extractive approach is more suitable for disaster scenarios because it can identify novel, emerging entities and concepts as hashtags, which is crucial given the dynamic nature of disasters.
  • Informal writing matters: Incorporating features that capture the nuances of informal language, such as phonetic spellings (IPA) and part-of-speech tags, significantly improves the model's ability to extract relevant hashtags from noisy social media text.
  • MTL framework is powerful: The Multi-Task Learning approach, by jointly training related tasks, allows the model to learn richer representations and perform better on the primary task of hashtag extraction.

Key Results

  • Dataset Quality: The semi-automated annotation process, validated by human annotators, achieved high quality, with approximately 89% of keyphrases deemed "Appropriate" and only about 3% "Not Appropriate" by majority vote.
  • Model Performance: The best performing model, LSTM-MTL+ELMO,IPA,POS, achieved an impressive F1-score of 92.22% on the "Multiple disasters" test set.
  • Generalization: The models demonstrated good generalization capabilities, performing well on unseen disaster types like Maria Hurricane, California Fire, and Philippines Floods.
  • Informal writing benefits: Variants that incorporated informal writing features (IPA, POS) showed significant improvements over baseline LSTM-MTL models, with the LSTM-MTL+ELMO,IPA,POS model outperforming others.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while their model can predict certain named entities beyond the lexicon, it can also miss others (e.g., "valenzuela city", "UST Hospital").
  • The current work focuses on English tweets, and the models might not directly transfer to other languages.
  • Future Work:
    • Multilingual setup: Extending the approach to handle multiple languages is a key future direction.
  • Hierarchical classification: Utilizing the dataset for hierarchical classification with fine and coarse-grained labels.
  • Domain knowledge integration: Incorporating more domain-specific knowledge could further enhance performance.
  • Real-time systems: Integrating these models into systems that provide real-time situational awareness for disaster response organizations.

Summary updated

Abstract

Tweet hashtags have the potential to improve the search for information during disaster events. However, there is a large number of disaster-related tweets that do not have any user-provided hashtags. Moreover, only a small number of tweets that contain actionable hashtags are useful for disaster response. To facilitate progress on automatic identification (or extraction) of disaster hashtags for Twitter data, we construct a unique dataset of disaster-related tweets annotated with hashtags useful for filtering actionable information. Using this dataset, we further investigate Long Short Term Memory-based models within a Multi-Task Learning framework. The best performing model achieves an F1-score as high as 92.22%. The dataset, code, and other resources are available on Github.
BibTeX
@inproceedings{chowdhury2020identifying,
  author = {Jishnu Ray Chowdhury and Cornelia Caragea and Doina Caragea},
  title = {{On Identifying Hashtags in Disaster Twitter Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i01.5387},
}
Figure