Practical, Efficient, and Customizable Active Learning for Named Entity Recognition in the Digital Humanities

Alexander Erdmann, David Joseph Wrisley, Benjamin Allen, Christopher Brown, Sophie Cohen-Bodénès, Micha Elsner, Yukun Feng, Brian Joseph, Béatrice Joyeux-Prunel, Marie-Catherine de MarneffeNAACL 2019 · DOI

Introduction

Named Entity Recognition (NER)—the process of identifying names, places, and organizations in text—is a mature technology for modern news English. However, it often fails Digital Humanities (DH) scholars working with historical languages, non-standard orthography, or noisy data from OCR-scanned books. To bridge this gap, researchers have developed the Humanities Entity Recognizer (HER), a toolkit designed to build high-quality custom NER models with minimal human effort.

The HER Framework and PTDL Algorithm

HER is a "whitebox" toolkit that implements a human-in-the-loop active learning workflow. Instead of requiring massive pre-labeled datasets, the system identifies the most informative sentences for a human to label next, iteratively improving the model.

The HER active learning workflow, illustrating the cycle between annotation, ranking, and tagging.

The core technical innovation is the Pre-Tag DeLex (PTDL) ranking algorithm. PTDL optimizes the "cold-start" phase of annotation through four distinct steps:

  1. Pre-tagging: Uses a growing gazetteer (dictionary) to find easy matches in unlabeled data.
  2. Delexicalized Tagging: Trains a model using contextual features (the words around a target) rather than the word itself. This prevents the model from simply memorizing known words and helps it discover new entities.
  3. Vocabulary Weighting: Scores Out-of-Vocabulary (OOV) words based on their likelihood of being entities.
  4. Sentence Ranking: Prioritizes sentences for human review based on a combination of model uncertainty, representativeness of patterns, and vocabulary diversity.

Evaluation: Inclusive vs. Exclusive

A key insight of this research is the distinction between evaluation metrics. While standard NLP focuses on Exclusive Evaluation (generalization to unseen test sets), DH scholars often prioritize Inclusive Evaluation—the accuracy of the entire project corpus, combining human labels and model predictions.

Comparison of Inclusive and Exclusive evaluation metrics across different model architectures and sampling strategies.

Key Results and Insights

The authors evaluated HER across diverse datasets, including historical French (FranText), Latin, Arabic, Spanish, and German. The results demonstrate significant efficiency gains:

  • Reduced Workload: PTDL reduced manual annotation requirements by 20–60% compared to random sampling. In the FranText corpus, PTDL achieved target performance with only 12% of the corpus labeled, whereas random sampling required nearly double that effort.
  • Shallow vs. Deep Models: In the "inclusive" setting, shallow models like Conditional Random Fields (CRFs) often outperform deep learning models (BiLSTMs) during the early stages of a project because they require far less data to become effective.
  • Language Agnostic: The system proved robust across typologically diverse languages, including those without capitalization (Arabic) or with non-standard capitalization (German and Latin).

Conclusion and Future Work

By prioritizing context through delexicalization and focusing on inclusive accuracy, the HER toolkit provides a practical path for researchers to annotate specialized historical corpora. Future developments will focus on handling nested entities (e.g., a person's name within a building name) and better managing inter-annotator disagreement in subjective humanities contexts.

Summary updated

BibTeX
@inproceedings{erdmann2019practical,
  author = {Alexander Erdmann and David Joseph Wrisley and Benjamin Allen and Christopher Brown and Sophie Cohen-Bodénès and Micha Elsner and Yukun Feng and Brian Joseph and Béatrice Joyeux-Prunel and Marie-Catherine de Marneffe},
  title = {{Practical, Efficient, and Customizable Active Learning for Named Entity Recognition in the Digital Humanities}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1231},
}
Figure