Safety Information Mining - What can NLP do in a disaster -

Graham Neubig, Yuichiroh Matsubayashi, Masato Hagiwara, Koji MurakamiIJCNLP 2011 · OpenReview

In the immediate aftermath of a massive natural disaster, information is as critical as physical aid. However, official channels often lag behind the chaotic, real-time surge of social media. This paper, presented at IJCNLP 2011, chronicles how a group of NLP researchers mobilized during the 2011 East Japan Earthquake to turn the "noise" of Twitter into actionable safety data.

The Challenge: Mining Safety Data in a Crisis

When the Magnitude 9.0 earthquake struck Japan in March 2011, communication infrastructure failed, and local governments were overwhelmed. While survivors turned to Twitter to post safety updates (e.g., "I am safe at X shelter" or "Has anyone seen Person Y?"), this information was buried under millions of unrelated tweets.

The researchers identified three primary challenges:

  1. Filtering: Separating rare safety-related tweets from the massive volume of general disaster commentary.
  2. Extraction: Identifying specific person names and locations from informal, noisy, and often ambiguous Japanese text.
  3. Speed: Traditional research cycles take months; in a disaster, information loses its value within days or even hours.

Methods: A Rapid NLP Pipeline

The authors formed a spontaneous working group called "ANPI NLP" and developed a pipeline focused on domain adaptation and rapid classification.

The NLP pipeline for extracting person and location entities from Japanese text.

The pipeline utilized KyTea, an open-source tool for Japanese word segmentation and Part-of-Speech (POS) tagging. Because general-purpose models failed on local Tohoku place names, they rapidly integrated domain-specific dictionaries containing postal codes, local surnames, and station names.

To refine the system, they employed Active Learning, where volunteers corrected the 100 most "uncertain" words identified by the model across four rounds. For classification, a multi-class SVM (using LIBLINEAR) categorized tweets into nine labels, such as "Alive," "Missing," or "Information Request."

Datasets and Annotation

The team created the "ANPI" Tweet Corpus, consisting of 61,376 disaster-related tweets. Over 65 volunteers annotated 33,242 of these tweets with safety labels and Named Entity (NE) tags in just two days. This was supplemented by general resources like the Balanced Corpus of Contemporary Written Japanese (BCCWJ) and custom-built lists of surnames and locations specific to the affected prefectures (Iwate, Miyagi, Fukushima, and Ibaraki).

Key Results and Insights

The inclusion of domain-specific dictionaries (+DICT) and active learning (+ACTIVE) provided a massive boost in performance, particularly for Named Entity Recognition.

Performance improvements in word segmentation (WS) and named entity recognition (NE) using domain dictionaries and active learning.

As shown in the results, adding local dictionaries increased labeled NE precision by nearly 16%. The best classification model achieved a micro-averaged F-score of 86.8% for identifying safety categories.

Beyond metrics, the system had real-world impact. The team manually verified and uploaded safety information for over 100 individuals to the Google Person Finder repository, providing closure to families when official registries were still weeks behind.

Lessons for Digital Humanitarians

The project highlighted several critical takeaways for future disaster NLP efforts:

  • Domain Adaptation is Non-Negotiable: "Out-of-the-box" models perform poorly because disaster-stricken areas have unique geographic and naming conventions.
  • The Power of "Good Enough": In a crisis, a robust, simple system that can be deployed in hours is superior to a state-of-the-art model that requires weeks of training.
  • Ambiguity in Brevity: Tweets often lack context (e.g., missing surnames). Without specific details, extracted information is often unactionable, marking a fundamental limit of microblogging data.
  • Preparedness: Future efforts need "pre-baked" annotation frameworks and task roadmaps ready to activate instantly when a crisis hits.

Summary updated

BibTeX
@inproceedings{neubig2011safety,
  author = {Graham Neubig and Yuichiroh Matsubayashi and Masato Hagiwara and Koji Murakami},
  title = {{Safety Information Mining - What can NLP do in a disaster -}},
  booktitle = {IJCNLP},
  year = {2011},
  url = {https://openreview.net/forum?id=Knqzd8t0koj},
}
Figure