Weakly-Supervised Fine-Grained Event Recognition on Social Media Texts for Disaster Management

Wenlin Yao, Cheng Zhang, Shiva Saravanan, Ruihong Huang, Ali Mostafavi

Abstract

People increasingly use social media to report emergencies, seek help or share information during disasters, which makes social networks an important tool for disaster management. To meet these time-critical needs, we present a weakly supervised approach for rapidly building high-quality classifiers that label each individual Twitter message with fine-grained event categories. Most importantly, we propose a novel method to create high-quality labeled data in a timely manner that automatically clusters tweets containing an event keyword and asks a domain expert to disambiguate event word senses and label clusters quickly. In addition, to process extremely noisy and often rather short user-generated messages, we enrich tweet representations using preceding context tweets and reply tweets in building event recognition classifiers. The evaluation on two hurricanes, Harvey and Florence, shows that using only 1-2 person-hours of human supervision, the rapidly trained weakly supervised classifiers outperform supervised classifiers trained using more than ten thousand annotated tweets created in over 50 person-hours.

Introduction

During natural disasters, social media platforms like Twitter become critical conduits for real-time information, including rescue requests and infrastructure damage reports. However, extracting actionable, fine-grained events from this "firehose" of data presents two significant challenges. First, the labeling bottleneck: traditional supervised learning requires thousands of manually annotated examples, a process too slow for emergency response. Second, noise and ambiguity: tweets are short and context-dependent. A word like "dead" might signify a casualty, a depleted phone battery, or a pop-culture reference.

This research bridges the gap between fast but inaccurate unsupervised methods and accurate but slow supervised methods by introducing a weakly-supervised framework that drastically reduces human effort while maintaining high performance.

Methodology

The proposed framework operates in three distinct phases designed to maximize efficiency and contextual understanding.

The three-phase framework for weakly-supervised event recognition, including data labeling and model training.

Phase One: Rapid Data Labeling via Clustering

Instead of individual tweet annotation, the system extracts tweets containing event keywords and groups them using the Speaker-Listener Label Propagation Algorithm (SLPA). A domain expert reviews a small sample from the largest clusters to disambiguate word senses (e.g., filtering out irrelevant uses of keywords). This "Clustering-assisted Manual Word Sense Disambiguation (WSD)" allows for the labeling of thousands of tweets in minutes.

Phase Two: Multi-channel BiLSTM Classifier

To address the brevity of tweets, the model utilizes a Multi-channel Bi-directional Long Short-Term Memory (BiLSTM) architecture. This model enriches the target tweet by encoding two additional channels:

  • Context Tweets: The five preceding tweets from the same user to capture situational background.
  • Reply Tweets: Responses from other users that often clarify the original tweet's intent.

Phase Three: Bootstrapping

The model is iteratively retrained by identifying high-confidence predictions in the unlabeled pool. This expands the training set with event descriptions that may not have contained the initial seed keywords.

Evaluation and Results

The system was evaluated on datasets from Hurricane Harvey (2017) and Hurricane Florence (2018), tracking nine fine-grained categories such as Help and Rescue (RES), Utilities (UTI), and Flood Control Infrastructure (FCI).

Performance comparison of various methods across fine-grained event categories.

As shown in the results table, the weakly-supervised system (Row 12) achieved a macro-average F1-score of 65.5%, notably outperforming the fully supervised classifier (Row 13) which achieved 61.9%. Key insights from the evaluation include:

  • Efficiency: The system achieved state-of-the-art results with a 25x to 50x reduction in human supervision time (1-2 hours vs. 50+ hours).
  • Precision Boost: Clustering-assisted cleaning improved the precision of the basic classifier by 18% compared to using raw, noisy keyword-matched data.
  • The Power of Context: The multi-channel approach proved vital for disambiguating vague posts, such as identifying an evacuation request by looking at a user's previous mentions of rising water levels.

Conclusion

This framework demonstrates that for noisy social media data, intelligent, cluster-based labeling is far more efficient than brute-force manual annotation. While the approach relies on the availability of conversational context and a brief window of expert time, it offers a scalable solution for rapid deployment during diverse disaster scenarios. Future work will explore adapting this framework to other emergency types, such as wildfires or earthquakes, where event ontologies and vocabularies differ.

Summary updated

Abstract

People increasingly use social media to report emergencies, seek help or share information during disasters, which makes social networks an important tool for disaster management. To meet these time-critical needs, we present a weakly supervised approach for rapidly building high-quality classifiers that label each individual Twitter message with fine-grained event categories. Most importantly, we propose a novel method to create high-quality labeled data in a timely manner that automatically clusters tweets containing an event keyword and asks a domain expert to disambiguate event word senses and label clusters quickly. In addition, to process extremely noisy and often rather short user-generated messages, we enrich tweet representations using preceding context tweets and reply tweets in building event recognition classifiers. The evaluation on two hurricanes, Harvey and Florence, shows that using only 1-2 person-hours of human supervision, the rapidly trained weakly supervised classifiers outperform supervised classifiers trained using more than ten thousand annotated tweets created in over 50 person-hours.
BibTeX
@inproceedings{yao2020weakly,
  author = {Wenlin Yao and Cheng Zhang and Shiva Saravanan and Ruihong Huang and Ali Mostafavi},
  title = {{Weakly-Supervised Fine-Grained Event Recognition on Social Media Texts for Disaster Management}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=x4zVDXaArub},
}
Figure