Introduction
Humanitarian organizations operate in increasingly volatile environments where violent incidents—ranging from the looting of aid convoys to the destruction of health facilities—directly threaten life-saving operations. To manage security and optimize program delivery, these organizations require real-time, granular data on conflict events. However, existing automated monitoring tools often fall short due to a lack of domain-specific labels and a heavy reliance on English-language sources.
To bridge this gap, researchers have introduced HUMVI, a comprehensive multilingual dataset designed to detect and categorize violent incidents that impact humanitarian aid. By providing over 17,000 expert-labeled news articles in English, French, and Arabic, HUMVI enables the development of more precise risk assessment models for the humanitarian sector.
The Need for Domain-Specific Data
Most current conflict datasets are designed for general political science research rather than operational humanitarian response. While a general dataset might flag a "bombing," a humanitarian planner needs to know if that bombing targeted a school or a food warehouse. Furthermore, many conflict-affected regions communicate primarily in French, Arabic, or local dialects, yet most NLP tools remain English-centric.
As shown in the comparison above, HUMVI is the first dataset to simultaneously address the news domain, provide specific humanitarian usecase tags, offer multilingual support, and maintain high-quality manual labels from sector experts.
The HUMVI Dataset and Task Definition
The dataset is divided into two primary variants:
- HUMVI-core: 15,631 English articles categorized into four sectors: Aid Security, Education, Health, and Protection.
- HUMVI-expansion: A broader set of 17,497 articles adding French and Arabic coverage, along with a fifth category: Food Security (aligned with UN Sustainable Development Goal 2).
The core task involves a two-step classification process. First, a model must determine if an article describes a conflict event impacting humanitarian operations (relevance). Second, it must categorize the event into specific aid sectors.
Methodology and Modeling
The researchers developed a pipeline combining automated scraping via NewsAPI and GDELT with expert validation from Insecurity Insight. To evaluate performance, the team benchmarked two distinct architectures:
- Fine-tuned Transformers: Monolingual and multilingual versions of BERT, RoBERTa, and DistilBERT.
- Large Language Models (LLMs): Zero-shot evaluations of GPT-4, GPT-4o, Llama-3-70B, and Mistral-Large.
To address data scarcity in French and Arabic, the team employed Translation Augmentation—synthetically generating training data by translating English samples. They also utilized Masking Loss to train models on partially labeled datasets without penalizing them for missing category tags.
Key Results and Insights
- Fine-Tuning Outperforms Zero-Shot LLMs: Smaller, fine-tuned models like XLM-RoBERTa consistently outperformed massive LLMs. While GPT-4o was the strongest LLM, its F1 scores for specific categories were often 10–20 points lower than fine-tuned BERT models. This suggests that the highly specific definitions required for humanitarian aid classification can overburden LLMs in zero-shot settings.
- The Power of Augmentation: Translation augmentation proved highly effective for low-resource scenarios. In Arabic relevance classification, this technique boosted the F1 score from 0.69 to 0.79.
- Sector Challenges: Models often struggled with the "Victim-Sector" connection. For instance, a model might identify violence against a bakery but fail to apply a "Food Security" tag without an inherent understanding of essential infrastructure.
Conclusion
HUMVI represents a significant step forward in "AI for Social Good." By providing a high-quality, multilingual, and expert-labeled dataset, the authors have laid the groundwork for automated systems that can help humanitarian organizations navigate the frontlines of global conflict more safely and effectively.
Summary updated