A Data-driven Approach to Named Entity Recognition for Early Modern French

Pedro Ortiz Suarez, Simon GabayCOLING 2022

TL;DR

This paper tackles the challenge of Named Entity Recognition (NER) for Early Modern French, a historically rich but linguistically complex text domain. By creating a large, fine-grained annotated corpus and fine-tuning existing Transformer models, they achieve state-of-the-art results, demonstrating the power of a data-driven approach over specialized architectures.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is crucial for digital humanities research, enabling the analysis of large text collections. However, historical texts, particularly Early Modern French (16th-18th centuries), present significant hurdles for NER systems. These challenges include inconsistent spelling, evolving place names, and the disappearance of older linguistic forms, making it difficult to apply tools developed for contemporary languages. Existing NER datasets and models are heavily biased towards modern languages, leaving a gap for historical text analysis. This paper aims to bridge this gap by developing both a comprehensive dataset and effective NER models for Early Modern French.

Methods

The authors adopt a data-driven approach, prioritizing the creation of a large, annotated corpus over developing novel, specialized NER architectures. They leverage existing state-of-the-art Transformer models:

  • BiLSTM-CRF: A classical neural architecture serving as a baseline, combining pre-trained word embeddings with character embeddings and a bidirectional LSTM encoder with a CRF decoder.
  • CamemBERT: A contemporary French language model based on the RoBERTa architecture, using SentencePiece for tokenization.
  • D'AlemBERT: An Early Modern French language model, sharing the same architecture as CamemBERT but using the original BPE tokenizer and pre-trained on a corpus of Early Modern French texts.

All models were fine-tuned using the flair framework. For CamemBERT and D'AlemBERT, a linear layer was appended to the last hidden representation of the <s> token and the mean of the subword units for each token, employing a "mean" subword pooling strategy.

Datasets

The core contribution is a newly created, fine-grained NER annotated corpus for Early Modern French. This corpus is derived from a subpart of the "core corpus" of the Presto project, encompassing approximately 5 million annotated tokens across texts from the 15th to 18th centuries. This corpus is significantly larger than existing popular NER evaluation corpora for contemporary English and French.

The authors also mention using:

  • Contemporary English and French corpora for comparison and context.
  • Pre-trained embeddings like Common Crawl-based FastText.
  • Existing French language models (CamemBERT) and a newly introduced one for Early Modern French (D'AlemBERT).

Key Insights and Ideas

  • Data-driven approach efficacy: The paper strongly advocates for the power of a large, high-quality dataset. Their results suggest that with sufficient data, even standard Transformer models can achieve state-of-the-art performance on challenging historical language tasks, outperforming specialized architectures.
  • "Forgetting" and "Re-learning": The authors hypothesize that the large size of their Early Modern French corpus allows Transformer models (CamemBERT and D'AlemBERT) to "forget" some of their contemporary pre-training data and effectively "re-learn" the nuances of the historical language, leading to strong performance.
  • Challenges of historical language: The paper highlights specific difficulties in Early Modern French NER, such as inconsistent spelling and evolving place names, underscoring the need for specialized resources.
  • Corpus size matters: The sheer size of their corpus (5 million tokens) is presented as a key factor in achieving their impressive results, especially for CamemBERT, which struggled with smaller datasets in previous work.

Key Results

  • The fine-tuned models achieved state-of-the-art results for Early Modern French NER.
  • Performance was on par with current state-of-the-art NER systems for Contemporary English.
  • F1-scores above 0.93 were achieved by both CamemBERT and D'AlemBERT on the test set.
  • The BiLSTM-CRF baseline achieved a respectable F1-score of 0.8586, demonstrating the inherent difficulty of the task and the quality of the data.
  • The newly created corpus is approximately 18 times larger than the French treebank and 23 times larger than the CoNLL 2003 corpus.

Limitations and Future Work

  • Annotation Imperfection: The authors acknowledge that due to the corpus size, not every token was exhaustively checked, and the annotation is not claimed to be perfect.
  • Single Annotator Consistency: While a single annotator was used to ensure consistency, this also means the annotation reflects one person's interpretation.
  • Coarse-Grained Annotation Focus: For model training, only the coarse level of annotation was used due to computational costs.
  • Future Work:
    • Investigating the impact of fine-tuning data size by iteratively training with subsets of the corpus.
  • Developing specialized architectures capable of predicting all annotation layers simultaneously.
  • Further research into the "forgetting" and "re-learning" phenomenon in Transformer models.
  • Leveraging the full annotation layers (beyond coarse NER) for more complex tasks.

Summary updated

BibTeX
@inproceedings{suarez2022data,
  author = {Pedro Ortiz Suarez and Simon Gabay},
  title = {{A Data-driven Approach to Named Entity Recognition for Early Modern French}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure