Latin Treebanks in Review: An Evaluation of Morphological Tagging Across Time

Marisa Hudspeth, Brendan O’Connor, Laure ThompsonACL 2024

TL;DR

This paper tackles the challenge of reliably performing morphological tagging on Latin texts across different historical periods and genres. By meticulously curating and harmonizing existing Latin treebanks, the authors demonstrate that modern BERT-based models significantly outperform previous approaches, showing improved robustness and accuracy, especially on rarer linguistic features.

What Problem Does This Paper Address?

Morphological tagging, the process of identifying grammatical features like case, gender, and tense for each word, is crucial for computational analysis of historical languages like Latin. While large digitized Latin archives exist, they are spread across various "treebanks" (annotated datasets) with inconsistent annotation schemes, time periods, and genres. This heterogeneity makes it difficult to train and evaluate robust morphological taggers that can generalize across the vast span of Latin literature. Previous work has often focused on specific subsets of Latin or used limited harmonization, leaving a gap in understanding how taggers perform across the full historical and stylistic spectrum of the language.

Methods

The authors' approach involves several key steps:

  • Metadata Curation: They manually compiled detailed metadata (time period, genre, author, etc.) for over 500 texts across five Universal Dependencies (UD) Latin treebanks.
  • Harmonization and Standardization: They addressed inconsistencies between UD treebanks and the LASLA treebank. This involved harmonizing feature values to ensure consistency and standardizing the tagset to align with traditional Latin grammar conventions, rather than solely relying on UD's cross-linguistic approach.
  • Custom Data Splits: They created new, time-period-specific training, development, and testing datasets to enable robust cross-time evaluation, ensuring that individual works remained within a single split to avoid data leakage.
  • BERT-based Tagger: They finetuned a LatinBERT model, a transformer-based architecture, using separate classification heads for each morphological feature. This model was trained on the harmonized and standardized datasets.
  • Comparative Evaluation: They compared their BERT-based tagger against previous state-of-the-art models (LatinCy and Stanza) using both standard UD splits and their custom time-period splits.

Datasets

The paper utilizes and analyzes several existing Latin treebanks:

  • Universal Dependencies (UD) Treebanks: Perseus, PROIEL, LLCT, ITTB, and UDante.
  • LASLA: A large, non-UD treebank for Classical Latin.
  • Harmonized UD Treebanks: A version of the UD treebanks harmonized by Gamba and Zeman (2023a).
  • Custom Data Splits: The authors introduce new train/dev/test splits categorized by time period (Classical, Bible, Post-Classical) and by combining UD and LASLA data.

Key Insights and Ideas

  • The Importance of Harmonization: Simply using existing treebanks without careful harmonization leads to unreliable evaluations. Differences in annotation schemes, even for seemingly minor features, can significantly impact model performance.
  • Traditional Grammar vs. UD: While UD offers a valuable cross-linguistic framework, aligning with traditional Latin grammar conventions can be more intuitive and potentially more effective for Latin-specific NLP tasks.
  • BERT's Superiority: Transformer-based models like BERT, when trained on harmonized and standardized data, significantly outperform older, non-transformer models and even other transformer models trained on less curated data.
  • Robustness Across Time and Genre: The BERT-based tagger demonstrates remarkable robustness, performing well even when trained on data from one time period and tested on another, a crucial capability for historical linguistics.
  • Rarity of Features: The paper highlights that rare feature values (e.g., certain verb forms, specific cases) are particularly challenging and benefit most from improved harmonization and powerful models.

Key Results

  • The BERT-based tagger achieved significantly higher whole-string morphological accuracy compared to previous methods across various UD test sets. For example, on the "All UD" test set, BERT achieved 0.929 accuracy, while LatinCy achieved 0.726.
  • The BERT tagger also showed superior performance on individual features, with smaller differences between in-domain and out-of-domain performance, indicating better generalization.
  • Adding LASLA data to the training set boosted performance for some rare features (like Case=Dat and Mood=Ger) but slightly decreased it for others (like Degree and UPOS), likely due to noisier training labels from annotation differences.
  • The authors found that training a single model on all time periods generally performed as well as, or even slightly better than, period-specific models, especially for morphological accuracy.

Limitations and Future Work

  • Remaining Inconsistencies: Despite extensive harmonization, some annotation differences between treebanks (e.g., regarding deponent verbs, or the classification of "esse" as a noun) could not be fully resolved and are left for future work.
  • Tokenization: The paper notes that words segmented into multiple subtokens by the tokenizer exhibit higher error rates, suggesting that a morphologically-aware tokenizer could further improve performance.
  • Acontextual Ambiguity: A significant portion of errors stemmed from genuine acontextual ambiguity, where the model struggled to pick the correct interpretation in context, even though its prediction was lexically valid. This points to the ongoing challenge of contextual understanding.
  • Future Directions: The authors suggest exploring conditioning morphological feature prediction on UPOS (and vice versa), enforcing grammatical constraints through modeling, and developing a morphologically-aware tokenizer as promising avenues for future research.

Summary updated

BibTeX
@inproceedings{hudspeth2024latin,
  author = {Marisa Hudspeth and Brendan O’Connor and Laure Thompson},
  title = {{Latin Treebanks in Review: An Evaluation of Morphological Tagging Across Time}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure