Abstract
HumSet is a large-scale, expert-annotated multilingual dataset designed to facilitate Natural Language Processing (NLP) in humanitarian crisis response. Comprising approximately 17,000 documents and 148,000 tagged entries in English, French, and Spanish, HumSet addresses the critical need for structured, reliable information extraction from secondary data sources like official reports and news. We benchmark several transformer-based and heuristic models, identifying key challenges such as non-sentential snippets and significant class imbalance across the humanitarian taxonomy.
Introduction
In the immediate aftermath of a humanitarian disaster, the first 72 hours are vital for effective aid deployment. Analysts must synthesize vast amounts of "secondary data"—including situation reports and news articles—to gain situational awareness. This process is currently manual and labor-intensive. While NLP has been applied to crisis data in the past, existing datasets frequently rely on noisy social media streams that lack the structured frameworks required by international organizations. HumSet provides a professional-grade alternative, grounded in the analysis frameworks used by the UN and Red Cross.
The HumSet Dataset
HumSet was curated by professional humanitarian analysts using the Data Entry and Exploration Platform (DEEP). The dataset spans 2018 to 2021 and covers 46 global projects. It is inherently multilingual, with English (61.3%), French (20.4%), and Spanish (18.3%) representation.
The data is organized according to a "Humanitarian Analysis Framework" consisting of 11 sectors (e.g., Health, Education) and a complex hierarchy of 62 unique pillar and subpillar tags.
Methodology
The researchers framed the humanitarian analysis problem as two distinct NLP tasks:
- Entry Extraction: Identifying specific snippets within long documents that contain relevant information. This is treated as a token classification task because only 38.8% of relevant entries align with full sentence boundaries; the remainder are fragments or multi-sentence spans.
- Multi-label Entry Classification: Categorizing extracted snippets into the hierarchical taxonomy.
Benchmarks were established using XLM-RoBERTa (Base), XtremeDistil, and FastText, compared against heuristic baselines like LEAD4.
Key Results and Insights
XLM-RoBERTa (Base) emerged as the top-performing model, achieving a ROUGE-L F1 score of 0.41 for extraction and a macro F1 of 0.73 for broad Sector classification. However, performance dropped significantly on more granular tasks, such as the 33-tag "Subpillars 1D" category (F1 of 0.38).
A primary challenge identified is the "long tail" of the dataset. As shown in the distribution of tag occurrences, over half of the tags (33 out of 62) appear in less than 2% of the entries. This creates a significant few-shot learning challenge for automated systems.
Conclusion and Future Work
HumSet represents a significant step toward professional-grade NLP for humanitarian aid. However, limitations remain, including the noise introduced when extracting text from PDF layouts and the inherent subjectivity of expert annotations across different global projects. Future work will focus on layout-aware models and improving classification performance on rare, long-tail tags.
Summary updated