TL;DR
This paper explores what linguistic knowledge is transferred when training models to inflect words across languages. Using an analogical, memory-based framework, they demonstrate that models can learn generalizable representations of morphological processes like suffixation, which aids in one-shot learning for unseen languages. This approach offers a new lens for understanding cross-lingual transfer beyond simple relatedness.
What Problem Does This Paper Address?
Morphological inflection, the process of changing a word's form to indicate grammatical function (e.g., adding "-ed" to "walk" to form "walked"), is a fundamental aspect of language. Transfer learning, where knowledge gained from one task or language is applied to another, has shown promise in improving inflection models, especially for low-resource languages. However, it's not always clear what is being transferred. Previous work suggests that language relatedness or typological similarity plays a role, but there are inconsistencies, with some closely related languages showing poor transfer and dissimilar ones showing surprising success. This paper aims to disentangle these effects by investigating what abstract representational concepts inflection models acquire and how these are shared across languages, moving beyond traditional tag-based approaches.
Methods
The core of this work is an analogical, memory-based framework for morphological inflection. Instead of relying on explicit linguistic tags (like "past tense"), the model learns by completing four-part analogies: lemma : target :: exemplar lemma : exemplar target. The "exemplar" is a known inflection of a different word in the same language. This setup allows the model to potentially "copy" the inflectional mechanism from the exemplar, rather than needing to learn language-specific rules from scratch.
The authors experiment with two exemplar selection strategies:
- Random selection: The exemplar is chosen arbitrarily from the training data. This makes the task harder but forces the model to learn more robust representations.
- Similarity-based selection: The exemplar is chosen to have a similar inflectional pattern (edit rule) to the target lemma. This makes the task easier and is primarily used for model training.
The underlying learning model is a character-based transformer, a standard architecture for sequence-to-sequence tasks, tuned for morphological challenges.
Datasets
The paper utilizes two main types of data:
- Sigmorphon 2020 development languages: A set of 45 languages across 5 families (Austronesian, Germanic, Niger-Congo, Oto-Manguean, Uralic) used for training and fine-tuning models.
- Sigmorphon 2020 surprise languages: An additional 45 languages used for evaluating one-shot transfer performance.
- Synthetic probe data: Artificially generated data designed to test specific morphological processes like prefixing, suffixing, reduplication, and gemination in isolation, allowing for a controlled investigation of transferability.
Key Insights and Ideas
- Analogical learning can capture generalizable processes: The memory-based framework, by focusing on analogy rather than explicit tags, shows that models can learn abstract representations of processes like suffixation that are useful across languages.
- One-shot transfer can be surprisingly effective: Even without fine-tuning, the analogical model can achieve substantial accuracy on unseen languages, sometimes outperforming baselines on specific difficult cases.
- Language typology matters more than just relatedness: The synthetic experiments suggest that the inherent structure of a language (e.g., whether it's agglutinative or has complex inflection classes) plays a significant role in how well morphological processes transfer, potentially more so than genetic relatedness.
- The "why" behind successful source languages: The paper offers a hypothesis for why languages like Hungarian are good source languages for unrelated targets: their agglutinative nature might provide a "cleaner" bias for concatenative affixation, less polluted by complex lexical or phonological alternations.
- Transfer is not perfect: While general processes are learned, their application can be disrupted by language-specific morpho-phonological rules, and transfer across different scripts (e.g., Latin vs. Cyrillic) is more challenging.
Key Results
- Supervised Fine-tuning: The random-exemplar model achieved an average accuracy of 89% across development languages, closely matching the baseline (90%), especially in families like Austronesian and Uralic. The similarity-based model performed worse (57%), highlighting the importance of learning to cope with varied inputs.
- One-Shot Learning:
- The similar-exemplar model generally outperformed the random-exemplar model in one-shot settings, achieving an overall accuracy of 30% vs. 14%.
- High scores were observed for languages like Shona (Niger-Congo, 88% for similar-exemplar) and Tajik (Iranian, 79% for similar-exemplar), sometimes surpassing baseline performance on those specific languages.
- Romance languages showed a significant difference, with the similar-exemplar model achieving 53% accuracy compared to only 5% for the random-exemplar model, indicating the importance of matching exemplars for this family.
- Synthetic Experiments: The models showed varying success in transferring specific processes. Suffixation was relatively successful, but prefixation and reduplication were more language-specific. Gemination transfer was particularly challenging, with only the similar-exemplar model showing some success, primarily in Uralic.
Limitations and Future Work
- Exemplar Selection: The performance of the memory-based model is highly sensitive to exemplar selection. While similarity-based selection is good for training, random selection is necessary for testing but can lead to errors when exemplars don't closely match the target.
- Handling Unseen Cells: The current model cannot predict inflections for cells where no exemplar is available. Future work could explore adaptive source selection to address this.
- Data Sparsity and Zipfian Distribution: Real-world datasets are often Zipfian, with sparse coverage of inflectional cells. The model needs to be extended to handle this.
- Typological Features: Replacing language family features with more granular typological indicators (like WALs properties) could potentially improve learning speed and transfer in low-resource settings.
- Cognitive Model Adaptation: The memory-based architecture's cognitive inspiration suggests it could be adapted as a model for language learning in contact situations, particularly for understanding how speakers learn new exponents versus new processes.
Summary updated