Introduction
Named Entity Recognition (NER)—the process of identifying names, places, and organizations in text—is a mature technology for modern news English. However, it often fails Digital Humanities (DH) scholars working with historical languages, non-standard orthography, or noisy data from OCR-scanned books. To bridge this gap, researchers have developed the Humanities Entity Recognizer (HER), a toolkit designed to build high-quality custom NER models with minimal human effort.
The HER Framework and PTDL Algorithm
HER is a "whitebox" toolkit that implements a human-in-the-loop active learning workflow. Instead of requiring massive pre-labeled datasets, the system identifies the most informative sentences for a human to label next, iteratively improving the model.
The core technical innovation is the Pre-Tag DeLex (PTDL) ranking algorithm. PTDL optimizes the "cold-start" phase of annotation through four distinct steps:
- Pre-tagging: Uses a growing gazetteer (dictionary) to find easy matches in unlabeled data.
- Delexicalized Tagging: Trains a model using contextual features (the words around a target) rather than the word itself. This prevents the model from simply memorizing known words and helps it discover new entities.
- Vocabulary Weighting: Scores Out-of-Vocabulary (OOV) words based on their likelihood of being entities.
- Sentence Ranking: Prioritizes sentences for human review based on a combination of model uncertainty, representativeness of patterns, and vocabulary diversity.
Evaluation: Inclusive vs. Exclusive
A key insight of this research is the distinction between evaluation metrics. While standard NLP focuses on Exclusive Evaluation (generalization to unseen test sets), DH scholars often prioritize Inclusive Evaluation—the accuracy of the entire project corpus, combining human labels and model predictions.
Key Results and Insights
The authors evaluated HER across diverse datasets, including historical French (FranText), Latin, Arabic, Spanish, and German. The results demonstrate significant efficiency gains:
- Reduced Workload: PTDL reduced manual annotation requirements by 20–60% compared to random sampling. In the FranText corpus, PTDL achieved target performance with only 12% of the corpus labeled, whereas random sampling required nearly double that effort.
- Shallow vs. Deep Models: In the "inclusive" setting, shallow models like Conditional Random Fields (CRFs) often outperform deep learning models (BiLSTMs) during the early stages of a project because they require far less data to become effective.
- Language Agnostic: The system proved robust across typologically diverse languages, including those without capitalization (Arabic) or with non-standard capitalization (German and Latin).
Conclusion and Future Work
By prioritizing context through delexicalization and focusing on inclusive accuracy, the HER toolkit provides a practical path for researchers to annotate specialized historical corpora. Future developments will focus on handling nested entities (e.g., a person's name within a building name) and better managing inter-annotator disagreement in subjective humanities contexts.
Summary updated