During humanitarian disasters, social media platforms like Twitter serve as vital "social sensors," providing real-time data on unfolding events. However, identifying actionable, urgent needs—such as individuals trapped or in need of medical attention—remains a significant challenge. Traditional machine learning models require massive amounts of labeled data, which are typically unavailable in the critical first hours of a new crisis. This research, presented at ASONAM '19, introduces a low-supervision ensemble system designed to detect urgency using as few as 400 labeled examples.
The Challenge of Crisis Data
Extracting information during a disaster is complicated by several factors:
- Data Sparsity: Labeled examples of "urgent" vs. "non-urgent" messages are scarce in the immediate aftermath.
- Domain Shift: Models trained on one type of disaster (e.g., an earthquake) often fail to generalize to another (e.g., a flood).
- Noise and Sentiment: Social media is rife with slang and emotional expressions that may be negative but do not indicate an actionable urgent need.
Proposed Methodology
The researchers developed a robust ensemble framework that integrates three distinct feature sets to capture different semantic nuances:
- Manual Features: An 11-dimensional vector capturing expert-selected keywords (e.g., hit, help, kill, strand) and numerical counts, which often signify the number of people in distress.
- Local Embeddings: Utilizing fastText, the system trains skip-gram models on the specific unlabeled background corpus of the current disaster. fastText’s sub-word character n-grams are particularly effective at handling the typos and out-of-vocabulary (OOV) words common in tweets.
- Global Embeddings: Pre-trained fastText embeddings from English Wikipedia provide a stable semantic baseline, acting as a regularizer when local data is sparse.
Each feature set is processed by its own Linear Regression model, and the final urgency score is determined by a weighted average of these three outputs.
Transfer Learning via Up-sampling
To adapt the system to a new "target" crisis using data from a "source" crisis, the authors implemented an up-sampling strategy. By taking the limited labeled samples from the new crisis and up-sampling them (by a factor of 6) before mixing them with the source data, the model is forced to prioritize the specific characteristics of the current event over the historical source data.
Experimental Results
The system was evaluated on three real-world datasets: the 2015 Nepal earthquake, the 2018 Kerala floods, and a 2018 disaster simulation in Macedonia.
Key findings include:
- Superior Performance: On the Nepal dataset, the ensemble achieved a 69.44% F-Measure, significantly outperforming the local embedding baseline (63.93%).
- The Role of Wikipedia: Global embeddings proved to be a critical "semantic safety net," improving generalization when local disaster data was too sparse to build a coherent word-space.
- Precision vs. Recall: While manual keyword features provided high precision, the embedding models were essential for maintaining high recall by identifying urgent messages that did not contain specific keywords.
Conclusion
By combining manual expertise with both local and global semantic models, this approach provides a viable path for urgency detection in low-resource environments. While F-measures in the 60-70% range indicate that further refinement is needed before full autonomy, the system's ability to transfer knowledge across disaster types marks a significant step forward for automated humanitarian response.
Summary updated