Resisting the Lure of the Skyline: Grounding Practices in Active Learning for Morphological Inflection

Saliha Muradoglu, Michael Ginn, Miikka Silfverberg, Mans HuldenACL 2024

TL;DR

This paper investigates the impact of data sampling strategies in active learning for morphological inflection, a crucial task in low-resource language documentation. They demonstrate that "skyline" sampling (uniform sampling from a large, balanced corpus like UniMorph) is unrealistic and often outperformed by methods that consider naturalistic word frequencies or model confidence, especially when the unannotated pool is designed to mimic real-world language data.

What Problem Does This Paper Address?

Active learning (AL) aims to reduce the manual annotation effort required to build machine learning models by intelligently selecting the most informative unlabeled data points. This is particularly relevant for low-resource languages, where data is scarce and annotation is costly. However, much AL research relies on simulated scenarios where a "pool" of unlabeled data is drawn from existing, often curated, large datasets. The authors argue that these simulated pools, particularly those derived from resources like UniMorph (a collection of morphological paradigms), can be unrealistic. They often exhibit a uniform distribution of linguistic phenomena and lexical diversity, which doesn't reflect the "Zipfian" nature of natural language (where a few words are very frequent and many are rare) or the inherent biases and imbalances found in real-world language data. This mismatch can lead to misleading conclusions about AL strategy performance. The paper specifically addresses this by exploring how different sampling strategies perform when the unannotated pool is designed to be more linguistically realistic, focusing on the task of morphological inflection.

Methods

The core of the paper's methodology lies in simulating active learning experiments for morphological inflection. They train Transformer-based models on a small seed set and then iteratively select 500 additional examples from a larger pool to augment the training data. The key innovation is in how this "unannotated pool" is constructed and sampled from.

They propose two benchmarks for the unannotated pool:

  • Baseline (Frequency Stratified): This pool is sampled from the intersection of Wikipedia word frequencies and UniMorph data. It aims to mimic naturalistic lexical diversity and the distribution of inflected forms found in real speech, accounting for Zipf's law.
  • Skyline (Uniform Sampling): This is a more traditional approach, sampling uniformly from the UniMorph database (excluding the intersection with Wikipedia). This represents a large, balanced, but potentially unrealistic pool.

They compare these pool constructions and sampling strategies against several AL selection criteria:

  • Oracle: Samples forms the model gets wrong (a gold standard for comparison).
  • Transformer Model Confidence (TMC): Selects examples the model is least confident about.
  • Language Model Confidence (LMC): Uses character-level language models (trained on lemma+MSD or wordform+MSD) to select examples with high negative log-likelihood.

Datasets

The experiments are conducted on the UniMorph database of inflection tables, covering eight typologically diverse languages: Arapaho (arp), Finnish (fin), Georgian (kat), Quechua (que), Sakha (sah), Turkish (tur), Tuvan (tyv), and Zulu (zul).

To create their "realistic" baseline, they leverage Wikipedia texts for each language to derive word frequencies. The unannotated pool is constructed by either sampling from the intersection of UniMorph and Wikipedia data (U ∩ W) or from UniMorph data not present in Wikipedia (U \ W).

Key Insights and Ideas

  • The "Skyline" is Unrealistic: The paper strongly argues that uniform sampling from a large, balanced corpus like UniMorph (the "skyline") is an unrealistic benchmark for active learning in language documentation. While it often yields high accuracy, it's practically impossible to replicate in a real-world scenario due to the curated nature of such datasets.
  • Context Matters: Pool Design is Crucial: The composition of the unannotated sample pool significantly influences active learning performance. A pool designed to mimic naturalistic word frequencies and lexical diversity (the "baseline") leads to more meaningful and often better results compared to a uniform, balanced pool.
  • Frequency-Based Sampling is Key: Strategies that incorporate naturalistic word frequencies (like their "baseline" approach) are shown to be more effective than uniform sampling when the pool is designed realistically.
  • Model Confidence Outperforms Uniformity: When the unannotated pool is linguistically realistic, active learning strategies based on model confidence (TMC and LMC) consistently outperform uniform random sampling and even the "oracle" in some cases. This suggests that focusing on uncertain examples is more beneficial than randomly picking from a diverse set.
  • Levenshtein Distance Limitations: The paper highlights that simple edit distance metrics (like Levenshtein) can be misleading for evaluating morphological inflection, as they don't account for linguistic factors like vocabulary diversity or the semantic role of different parts of speech.

Key Results

  • The "skyline" approach (uniform sampling from UniMorph) achieved the highest average accuracy gains across languages, but this is attributed to its unrealistic sampling.
  • The "baseline" approach (frequency-stratified sampling from Wikipedia intersection) underperformed on average, but its results were more consistent across languages.
  • Targeted sampling strategies, specifically Transformer Model Confidence (TMC) and Language Model Confidence (LMC(Wordform, MSD)), yielded the greatest improvements in accuracy, outperforming the baseline and often the skyline when the pool was realistically constructed. For example, TMC showed an average accuracy gain of 0.247 and LMC(Wordform, MSD) 0.230, significantly higher than the baseline's 0.067.
  • The paper also found a correlation between lower cross-entropy (meaning the sampled set is more similar to the test set) and better performance, suggesting that sampling data that closely matches the target distribution is beneficial.

Limitations and Future Work

  • Language Family Bias: The study used eight languages, but three were from the Turkic language family. The authors acknowledge that it was challenging to find languages with sufficient Wikipedia data and UniMorph coverage that also mapped well orthographically. This limits the generalizability of findings across a wider range of typological features.
  • Wikipedia as a Proxy: While Wikipedia is used to simulate naturalistic text, it has its own limitations in terms of style, register, and genre diversity, which may not fully represent all aspects of natural speech or all genres relevant to language documentation.
  • Future Work: The authors suggest that future work should focus on developing more sophisticated methods for constructing realistic unannotated pools, potentially incorporating more diverse data sources and genres. Further research into how to best approximate the test distribution when it's unknown is also a key direction. They also note the need for future work to address the limitations of the current study regarding language diversity.

Summary updated

BibTeX
@inproceedings{muradoglu2024resisting,
  author = {Saliha Muradoglu and Michael Ginn and Miikka Silfverberg and Mans Hulden},
  title = {{Resisting the Lure of the Skyline: Grounding Practices in Active Learning for Morphological Inflection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure