“Gotta catch ‘em all!”: Retrieving people in Ancient Greek texts combining transformer models and domain knowledge

Marijke Beersmans, Alek Keersmaekers, Evelien de Graaf, Tim Van de Cruys, Mark Depauw, Margherita FantoliACL 2024

TL;DR

This paper tackles the challenge of automatically identifying people's names (Named Entity Recognition or NER) in Ancient Greek texts, a task crucial for historical research. By comparing various transformer models and exploring the integration of domain knowledge like gazetteers and syntactic information, the authors demonstrate significant improvements, particularly for identifying individuals.

What Problem Does This Paper Address?

Accurately identifying mentions of people in historical texts is a fundamental yet challenging task for Named Entity Recognition (NER). For scholars studying Ancient Greek, precisely locating and categorizing individuals is vital for large-scale analyses of historical representation. While transformer models have shown promise in improving NER performance, applying them to classical languages like Ancient Greek presents unique difficulties due to factors like data scarcity, linguistic evolution, and noisy historical sources. This paper aims to bridge this gap by evaluating existing transformer models and proposing methods to enhance their performance, specifically for the "people" entity type.

Methods

The researchers employed and compared four transformer-based models for NER on Ancient Greek. Initially, they assessed general NER performance across categories like people, locations, and groups. To focus on the core problem, they then simplified the task to a binary "person" vs. "miscellaneous" classification for capitalized tokens. Crucially, they explored two methods to integrate domain knowledge:

  • Rule-based Gazetteer Approach: This involved using external lists of names (Trismegistos NamVar) as a post-processing step to correct predictions.
  • Machine Learning Gazetteer Approach: Here, gazetteer information was incorporated as a binary mask input to the transformer model during training.
  • Syntactic Dependency Approach: This method leveraged grammatical relationships within sentences to identify and reconstruct multi-token person entities.

Datasets

The study utilized a combination of existing and newly annotated datasets:

  • Harmonized Corpus: Data from four projects was combined and harmonized:
  • Odyssey (OD): Ancient Greek epic poetry.
  • Deipnosophistae (DEIPN): Encyclopedic dialogue.
  • Stepbible Corpus (SB): Ancient Greek New Testament.
  • Pausanias' Periegesis Hellados (PH): Travelogue.
  • GLAux TEST: A manually annotated subset of the larger GLAUx corpus, representing a broad range of Ancient Greek literature from the 8th century BC to the 4th century CE. This dataset was specifically created for this project to evaluate model generalizability.

Key Insights and Ideas

  • Transformer models show promise but have limitations: While transformers significantly improve NER over previous methods, they struggle with out-of-domain data and can be misled by capitalization patterns alone.
  • Domain knowledge is key for improvement: Integrating external knowledge sources like gazetteers and syntactic information demonstrably enhances the performance of NER models, especially for the challenging "person" entity.
  • Simplifying the task helps: Focusing on a binary "person" vs. "miscellaneous" classification for capitalized tokens, rather than full NER, leads to better performance and is a practical step for targeted analysis.
  • Multi-token entities are a persistent challenge: Identifying and correctly segmenting names that span multiple words remains difficult, even with advanced techniques.
  • Data harmonization is critical: The diverse nature of historical text sources necessitates careful harmonization of annotations to create a cohesive dataset for training and evaluation.

Key Results

  • Model Performance: AG_BERT, a model specifically trained on Ancient Greek, generally outperformed other models, especially on the out-of-domain GLAUx TEST set.
  • Gazetteer Impact: The machine learning approach incorporating gazetteer information (AG_BERT_mask) showed a notable improvement in precision on the GLAUx TEST set (0.84 to 0.90), though with a slight drop in recall. The rule-based approach (AG_BERT_rule) increased recall but decreased precision.
  • Syntax for Multi-token Entities: The AG_BERT_syntax approach significantly improved the recognition of multi-word person entities, particularly for sequences up to three tokens long.
  • Error Analysis: Common errors included misclassifying nicknames, tokens that appear as both common nouns and names (e.g., "Lychnos" for a deity vs. "lychnos" for candle), and mathematical notations.

Limitations and Future Work

The authors acknowledge several limitations:

  • Reliance on Capitalization: The approach heavily depends on the capitalization choices made in digital editions, which can be inconsistent and limit transferability to other languages or editions.
  • Gazetteer Coverage: Gazetteers, while useful, cannot guarantee complete coverage of all name variations.
  • Exact Matching: The strict exact matching between text lemmas and gazetteer forms missed variations in language.
  • Automatic Parsing Errors: Reliance on automatic syntactic parsing introduced errors that impacted the performance of the syntax-based approach.

Future work suggested includes integrating more gazetteers, incorporating attestation counts as weights, and improving rule-based extraction for multi-token entities through more detailed dependency tree analysis.

Summary updated

BibTeX
@inproceedings{beersmans2024gotta,
  author = {Marijke Beersmans and Alek Keersmaekers and Evelien de Graaf and Tim Van de Cruys and Mark Depauw and Margherita Fantoli},
  title = {{“Gotta catch ‘em all!”: Retrieving people in Ancient Greek texts combining transformer models and domain knowledge}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure