Too Young to NER: Improving Entity Recognition on Dutch Historical Documents

Vera Provatorova, Marieke van Erp, Evangelos KanoulasCOLING 2024

TL;DR

This paper tackles the challenge of Named Entity Recognition (NER) on historical Dutch documents, a task that lags behind modern language processing. By training and evaluating three BERT-based models on a new historical dataset, the authors demonstrate that specialized models can significantly outperform general-purpose ones, paving the way for better searchability of historical archives.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is crucial for making digital archives searchable by identifying people, places, and organizations. However, applying NER to historical documents is significantly harder than to modern text due to challenges like OCR noise, historical spelling variations, and evolving language use. While NER for languages like English and French has seen progress, Dutch historical NER remains an underexplored area, despite the availability of historical Dutch texts. This paper aims to bridge this gap by investigating the performance of modern NER models on historical Dutch documents and identifying key challenges.

Methods

The authors fine-tuned three BERT-based models for NER, treating it as a token classification problem:

  • BERTje: A Dutch model trained on a mix of modern and historical texts, with modern texts dominating.
  • GysBERT: A Dutch model specifically designed and pre-trained on historical data.
  • mBERT: A multilingual model that includes Dutch.

These models were trained on a GPU for 15 epochs with early stopping. For evaluation, they compared these models against WikiNEuRal, a strong baseline that is not specifically optimized for historical data. The evaluation involved analyzing precision, recall, and F1 scores across different centuries and creating confusion matrices to understand error patterns.

Datasets

The study utilized the following datasets:

  • Dutch Language Institute (2022) Dataset: This is the primary dataset used for training and evaluation. It comprises historical notarial deeds from various Dutch archives, spanning the 17th to 19th centuries. It includes training, validation, and several historical test splits (SA, VOC, RHC, NHA).
  • Tjong Kim Sang (2002) Dataset: This dataset of modern texts was used as a contemporary benchmark to compare the models' performance on current language versus historical language.

Key Insights and Ideas

  • Historical Data is a Different Beast: Modern NER models, even those trained on Dutch, struggle with historical texts. The unique linguistic characteristics of older documents pose significant challenges.
  • Specialization Matters: Models pre-trained on historical data (like GysBERT) show promise, but in this study, they didn't consistently outperform BERTje, suggesting that the amount of historical data in pre-training might be less critical than the quality and task-specific fine-tuning.
  • Error Analysis is Crucial: Simply looking at overall scores isn't enough. The paper's detailed error analysis revealed common issues like misidentified mention boundaries, confusion between people and places (especially when family names incorporate locations), and errors stemming from inconsistencies in the historical datasets themselves.
  • Ensembling Potential: The low overlap in errors between the three fine-tuned models suggests that an ensemble approach could further improve performance on Dutch historical NER.

Key Results

  • All three fine-tuned BERT-based models (BERTje, GysBERT, mBERT) significantly outperformed the contemporary baseline (WikiNEuRal) on historical Dutch test data across all centuries.
  • On contemporary data, WikiNEuRal performed best, highlighting the gap between modern and historical language processing.
  • GysBERT, despite being pre-trained on historical data, did not consistently outperform BERTje or mBERT on historical test sets, indicating that fine-tuning on the specific historical dataset was highly effective.
  • The models achieved approximately the same results on historical data, suggesting a plateau or that the current models are reaching their limits with the given data.

Limitations and Future Work

  • Dataset Limitations: The paper notes that some errors in the test sets were due to incorrect annotations, and certain entities were split incorrectly.
  • Limited Entity Types: The study focused on PER and LOC entities, excluding ORG and MISC due to dataset limitations.
  • Future Work: The authors suggest implementing and testing an ensemble of the three models to leverage their complementary error patterns. They also propose experimenting with more diverse entity types and applying these methods to additional historical domains.

Summary updated

BibTeX
@inproceedings{provatorova2024too,
  author = {Vera Provatorova and Marieke van Erp and Evangelos Kanoulas},
  title = {{Too Young to NER: Improving Entity Recognition on Dutch Historical Documents}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure