Ye Olde French: Effect of Old and Middle French on SIGMORPHON-UniMorph Shared Task Data

William Kezerian, Lam An Wyner, Sandro Ansari, Kristine M. YuNAACL 2024

TL;DR

This paper investigates why French consistently underperforms in the SIGMORPHON-UniMorph shared tasks for morphological inflection. The authors discover that the inclusion of Old and Middle French forms in the training data is the primary culprit, leading to significant accuracy drops. By removing these archaic forms, they demonstrate a substantial improvement in model performance, highlighting the importance of clean, historically accurate datasets for NLP tasks.

What Problem Does This Paper Address?

The SIGMORPHON-UniMorph shared task is a benchmark for evaluating computational morphology systems, specifically their ability to predict inflected word forms given a lemma and a set of features. For years, French has been a puzzling outlier, showing mysteriously lower performance compared to other languages, even high-resource ones. While neural models typically outperform non-neural baselines in this task, this trend was reversed for French in recent years. This paper aims to explain this persistent underperformance and the unusual behavior of neural models on French by examining the quality of the French training data itself.

Methods

The researchers replicated the 2023 SIGMORPHON-UniMorph task for French using both neural and non-neural baseline models. Their core methodology involved:

  • Replication Experiments: Running the standard task with the original French dataset.
  • Data Cleaning: Creating new training/testing splits by systematically removing Old and Middle French lemmas and their associated forms. This was done in two ways:
  • Lemma-sampled (Seg): Removing entire lemmas identified as Old French.
  • Form-sampled (Seg-Minimal): Removing specific forms derived from Old French lemmas.
  • Error Analysis: Applying a detailed taxonomy (based on Gorman et al., 2019) to categorize the types of errors made by the models. This allowed them to pinpoint why errors were occurring, particularly focusing on errors attributable to archaic French forms.
  • Wiktionary-based Lemma Identification: Developing a script to identify Old French lemmas by cross-referencing entries in the English edition of Wiktionary.

Datasets

The primary datasets used were:

  • UniMorph 4.0 French data (fra, fra.args, fra.segmentations): This is the standard dataset for the SIGMORPHON-UniMorph shared task on French. The paper specifically investigates the fra.args file as the source for the 2023 task splits.
  • Past SIGMORPHON-UniMorph shared task splits (2017, 2018, 2019): Used to confirm the historical prevalence of the problem.
  • Flexique: A French lexical database based on post-1950s texts and subtitles, used in previous studies and noted for its high accuracy.
  • Silfverberg and Hulden (2018) datasets: Including a random sample of 1000 lemmas and a filtered sample of high-frequency lemmas from French UniMorph.

The paper also references the English edition of Wiktionary as a source for identifying Old French lemmas.

Key Insights and Ideas

  • The "Ye Olde French" Problem: The most significant finding is that a substantial portion (around 20%) of lemmas in the standard French UniMorph datasets for SIGMORPHON tasks are actually from Old or Middle French. This inclusion is directly correlated with poor model performance.
  • Historical Data vs. Modern Task: Modern morphological inflection tasks are designed for contemporary language. Including archaic forms creates a mismatch, confusing models that are trained to predict modern French.
  • Error Attribution: The error analysis clearly showed that "Old French Lemma errors" and "Old French affix overapplication errors" were dominant when using the original, uncleaned data. Removing these forms eliminated these specific error types.
  • Wiktionary's Role: While Wiktionary is a valuable resource, its English edition's classification of Old French may not always be perfectly aligned with modern linguistic usage, sometimes classifying words as Old French when they have been repurposed idiomatically in modern French.
  • Non-Neural vs. Neural Performance: Even after removing Old French forms, the non-neural baseline continued to outperform the neural model on French. This suggests that the Old French issue, while significant, isn't the sole reason for the neural model's relative underperformance on French.

Key Results

  • Accuracy Improvement: Removing Old French lemmas from the 2023 task splits led to a significant accuracy increase:
  • Non-neural baseline ("RU"): Improved by 10.32-11.72%.
  • Neural model ("NN"): Improved by 12.32-13.34%.
  • Error Reduction: In the original splits, Old French errors accounted for 56.6% (non-neural) and 37.8% (neural) of all errors. After removing Old French forms, these error types were eliminated entirely.
  • Prevalence of Old French: Old French lemmas constituted approximately 20% of the lemmas in the 2017, 2018, and 2023 SIGMORPHON-UniMorph shared task splits.
  • Contrast with Clean Data: Datasets derived from modern French sources (like Flexique) showed near-perfect accuracy, highlighting the impact of data quality.

Limitations and Future Work

  • Wiktionary Nuance: The authors acknowledge that relying solely on the English Wiktionary for Old French identification might not be perfectly accurate, as some words have evolved into modern idiomatic expressions. Future work could involve more nuanced linguistic expertise or cross-referencing with specialized French historical lexicons.
  • Data Inconsistencies Beyond Old French: The paper also briefly touches upon other data inconsistencies, such as the handling of reflexive verbs and multiple grammatical inflections for certain verb classes, which could also affect performance.
  • Future Data Curation: The authors strongly recommend using cleaner datasets like fra.segmentations for future SIGMORPHON tasks involving French. They also suggest that future shared tasks should incorporate more detailed, language-specific data quality analyses and error analyses to better understand and address performance issues.
  • Beyond French: The paper implies that similar language-specific data quality issues might exist for other languages in UniMorph and SIGMORPHON, advocating for more in-depth linguistic studies across the board.

Summary updated

BibTeX
@inproceedings{kezerian2024ye,
  author = {William Kezerian and Lam An Wyner and Sandro Ansari and Kristine M. Yu},
  title = {{Ye Olde French: Effect of Old and Middle French on SIGMORPHON-UniMorph Shared Task Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure