Low-supervision urgency detection and transfer in short crisis messages

Mayank Kejriwal, Peilin Zhou

During humanitarian disasters, social media platforms like Twitter serve as vital "social sensors," providing real-time data on unfolding events. However, identifying actionable, urgent needs—such as individuals trapped or in need of medical attention—remains a significant challenge. Traditional machine learning models require massive amounts of labeled data, which are typically unavailable in the critical first hours of a new crisis. This research, presented at ASONAM '19, introduces a low-supervision ensemble system designed to detect urgency using as few as 400 labeled examples.

The Challenge of Crisis Data

Extracting information during a disaster is complicated by several factors:

  1. Data Sparsity: Labeled examples of "urgent" vs. "non-urgent" messages are scarce in the immediate aftermath.
  2. Domain Shift: Models trained on one type of disaster (e.g., an earthquake) often fail to generalize to another (e.g., a flood).
  3. Noise and Sentiment: Social media is rife with slang and emotional expressions that may be negative but do not indicate an actionable urgent need.

Proposed Methodology

The researchers developed a robust ensemble framework that integrates three distinct feature sets to capture different semantic nuances:

  • Manual Features: An 11-dimensional vector capturing expert-selected keywords (e.g., hit, help, kill, strand) and numerical counts, which often signify the number of people in distress.
  • Local Embeddings: Utilizing fastText, the system trains skip-gram models on the specific unlabeled background corpus of the current disaster. fastText’s sub-word character n-grams are particularly effective at handling the typos and out-of-vocabulary (OOV) words common in tweets.
  • Global Embeddings: Pre-trained fastText embeddings from English Wikipedia provide a stable semantic baseline, acting as a regularizer when local data is sparse.
The system architecture combines local word embeddings, manual features, and global Wikipedia embeddings into a weighted ensemble.

Each feature set is processed by its own Linear Regression model, and the final urgency score is determined by a weighted average of these three outputs.

Transfer Learning via Up-sampling

To adapt the system to a new "target" crisis using data from a "source" crisis, the authors implemented an up-sampling strategy. By taking the limited labeled samples from the new crisis and up-sampling them (by a factor of 6) before mixing them with the source data, the model is forced to prioritize the specific characteristics of the current event over the historical source data.

Experimental Results

The system was evaluated on three real-world datasets: the 2015 Nepal earthquake, the 2018 Kerala floods, and a 2018 disaster simulation in Macedonia.

Performance metrics for the Nepal dataset demonstrate that the ensemble approach significantly outperforms individual feature sets.

Key findings include:

  • Superior Performance: On the Nepal dataset, the ensemble achieved a 69.44% F-Measure, significantly outperforming the local embedding baseline (63.93%).
  • The Role of Wikipedia: Global embeddings proved to be a critical "semantic safety net," improving generalization when local disaster data was too sparse to build a coherent word-space.
  • Precision vs. Recall: While manual keyword features provided high precision, the embedding models were essential for maintaining high recall by identifying urgent messages that did not contain specific keywords.

Conclusion

By combining manual expertise with both local and global semantic models, this approach provides a viable path for urgency detection in low-resource environments. While F-measures in the 60-70% range indicate that further refinement is needed before full autonomy, the system's ability to transfer knowledge across disaster types marks a significant step forward for automated humanitarian response.

Summary updated

BibTeX
@inproceedings{kejriwal2019low,
  author = {Mayank Kejriwal and Peilin Zhou},
  title = {{Low-supervision urgency detection and transfer in short crisis messages}},
  booktitle = {Asonam},
  year = {2019},
  doi = {10.1145/3341161.3342936},
}
Figure