ÚFAL LatinPipe at EvaLatin 2024: Morphosyntactic Analysis of Latin

Milan Straka, Jana Straková, Federica GambaCOLING 2024

TL;DR

The ÚFAL LatinPipe system won the EvaLatin 2024 Dependency Parsing shared task by leveraging a fine-tuned, concatenated pre-trained language model with a novel architecture for joint dependency parsing and morphological analysis. This approach achieved state-of-the-art results by harmonizing diverse Latin treebanks and ensembling multiple models.

What Problem Does This Paper Address?

Accurately parsing and analyzing the morphology of Latin, an ancient language with a rich and complex grammatical structure, presents a significant challenge. Existing methods often struggle with the nuances of Latin's inflectional system and the variability in annotation styles across different historical corpora. The EvaLatin 2024 Dependency Parsing shared task aimed to push the boundaries of Latin NLP by evaluating systems on a diverse set of Latin treebanks, highlighting the need for robust models that can handle linguistic variation and achieve high accuracy in both dependency parsing and morphological analysis.

Methods

The core of the ÚFAL LatinPipe system is a deep learning architecture built upon pre-trained language models (PLMs). Specifically, they experimented with fine-tuning and concatenating base and large versions of Latin PLMs like LaBerta, PhilBerta, and XLM-R. The architecture features a dot-product attention head for dependency parsing and separate softmax classification heads for morphological analysis, enabling joint learning of these tasks.

Key methodological innovations include:

  • Frozen Pretraining: Initially freezing the PLM weights and training only the downstream layers to stabilize the learning process.
  • Stacked BiLSTMs: Adding two bidirectional LSTM layers on top of the Transformer encoder to enhance the modeling of local contextual information and UPOS tags.
  • Gold UPOS Input: Incorporating gold UPOS (Universal Part-of-Speech) tags as an additional input feature to the model, which significantly improved performance.
  • Multi-treebank Training: Training on a combination of seven publicly available Latin corpora, sampling instances proportionally to the square root of their sentence counts.
  • Annotation Harmonization: A crucial step involved harmonizing the annotation styles across different treebanks, particularly addressing inconsistencies in the PROIEL corpus, which led to substantial performance gains.
  • Ensembling: For the final submission, they ensembled the probability distributions of seven randomly instantiated networks to further boost accuracy.
  • Punctuation Handling: A specific strategy was developed to handle punctuation mismatches between training data (which sometimes included punctuation) and test data (which did not), by artificially adding punctuation to the test data and then removing it post-prediction.

Datasets

The system was trained on seven publicly available Latin corpora:

  • ITTB of UD 2.13
  • LLCT of UD 2.13
  • PROIEL (both original and harmonized versions)
  • UDante of UD 2.13
  • Perseus of UD 2.13
  • Sab (UD-style annotated text)
  • Archimedes Latinus UD-style treebank

These corpora were used in various combinations for single-treebank and multi-treebank training. Evaluation was performed on the EvaLatin 2024 test data, which included both Poetry and Prose sections, as well as on UD 2.13 test sets.

Key Insights and Ideas

  • Harmonization is Key: The most significant insight is the critical role of annotation harmonization. Addressing inconsistencies, especially in the PROIEL treebank, led to substantial improvements, underscoring the need for standardized data in NLP tasks.
  • Joint Modeling Pays Off: Jointly learning dependency parsing and morphological analysis within a single neural architecture proved highly effective, allowing the model to leverage shared linguistic information.
  • PLMs are Powerful, but Context Matters: While fine-tuned PLMs form a strong baseline, adding architectural components like BiLSTMs and using gold UPOS as input significantly enhances their ability to capture nuanced linguistic features.
  • Ensembling for Robustness: Ensembling multiple models, even those with similar architectures, consistently improves performance, providing a more robust and accurate final prediction.
  • Multi-treebank Training Benefits Smaller Datasets: Multi-treebank training was most beneficial for smaller datasets, helping them achieve performance closer to larger corpora, while larger datasets sometimes saw marginal gains or even slight decreases.

Key Results

The ÚFAL LatinPipe system achieved state-of-the-art results, winning the EvaLatin 2024 Dependency Parsing shared task.

  • LAS Macro Average: The best entry (ÚFAL LatinPipe 1) achieved a LAS macro average of 76.58 on the EvaLatin 2024 test data (Table 3.D).
  • UPOS and UFeats: The system also surpassed previous systems in UPOS and UFeats tagging accuracy, demonstrating its comprehensive morphosyntactic analysis capabilities (Tables 4 and 5).
  • Impact of Components:
    • Harmonizing PROIEL improved LAS by +1.2 on Poetry and +1.3 on Prose on the EvaLatin test set.
  • Ensembling 7 models added +1.26 percent points on the EvaLatin test set.
  • Using gold UPOS as input improved results by +0.64 percent points on UD 2.13 macro average and +1.2/+1.3 on EvaLatin test data.
  • Comparison to Previous Work: The system outperformed previous methods like UDPipe 2, MaChAmp, and Nehrdich and Hellwig's parser on UD 2.13 test sets.

Limitations and Future Work

  • Overtraining on Perseus: The authors noted that XLM-R large might have overtrained on the Perseus corpus due to its small size.
  • Punctuation Mismatch: A workaround was needed to handle punctuation differences between training and test data when PROIEL was excluded.
  • Future Work: The authors plan to draw insights from LatinPipe's methodologies for the development of UDPipe 3, suggesting a continued focus on improving universal dependency parsing tools.

Summary updated

BibTeX
@inproceedings{straka2024ufal,
  author = {Milan Straka and Jana Straková and Federica Gamba},
  title = {{ÚFAL LatinPipe at EvaLatin 2024: Morphosyntactic Analysis of Latin}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure