TL;DR
This paper presents the SIGMORPHON 2021 Shared Task on unsupervised morphological paradigm clustering, where systems group word forms into inflectional paradigms without any prior linguistic knowledge. Grammar-based systems outperformed others, but even the best systems fell short of a supervised lemmatizer, highlighting significant room for improvement in unsupervised learning for morphology.
What Problem Does This Paper Address?
Computational morphology research has largely focused on supervised learning for word inflection, where systems learn to generate inflected forms given a lemma and specific grammatical features. However, this approach doesn't mirror how humans acquire language, particularly children, who learn to segment speech, decode meaning, and infer grammatical patterns from noisy, unlabeled input. This paper tackles one crucial step in this natural acquisition process: unsupervised morphological paradigm clustering. The challenge is to group word forms that belong to the same lexeme (e.g., "walk," "walked," "walking") into inflectional paradigms, without knowing what a "lemma" is, what grammatical features exist, or how many forms constitute a complete paradigm. This requires systems to identify true inflectional relationships while distinguishing them from accidental similarities and derivational patterns.
Methods
The SIGMORPHON 2021 Shared Task invited participants to develop systems for unsupervised paradigm clustering. Submissions generally fell into two categories:
- Similarity-based systems: These systems used various orthographic and embedding-based similarity metrics to represent words, followed by clustering algorithms like k-means or agglomerative clustering.
- Grammar-based systems: These systems aimed to learn grammatical rules or patterns directly from the data. They either applied these learned grammars to clustering or first segmented words into stems and affixes, then clustered forms sharing a stem.
The paper also describes a simple baseline system that groups words sharing a common substring of a certain length (tuned to k=5).
Datasets
The task utilized corpora from the Johns Hopkins University Bible Corpus (JHUBC).
- Development Languages: Maltese, Persian, Portuguese, Russian, and Swedish.
- Test Languages: Basque, Bulgarian, English, Finnish, German, Kannada, Navajo, Spanish, and Turkish.
For evaluation, gold partial paradigms were created for the development languages. These were derived from the UniMorph dataset, filtered to include only word types attested in the Bible corpus. Gold data for test languages were withheld until after the task.
Key Insights and Ideas
- Grammar-based approaches excel: The paper highlights that grammar-based systems significantly outperformed similarity-based methods and the simple substring baseline, suggesting that learning explicit linguistic structures is more effective for this task.
- The supervised ceiling is high: Even the best unsupervised systems were outperformed by a supervised lemmatizer. This is a critical finding, indicating that while progress has been made, unsupervised methods still have a long way to go to match supervised performance, even on a simplified task.
- Language typology matters: The paper notes that languages with complex morphology (like Basque and Navajo, which use both prefixes and suffixes) and agglutinative structures pose greater challenges. This is partly due to the large number of unique word forms and the difficulty in distinguishing inflection from derivation.
- Overgeneralization vs. Underspecification: Systems often exhibit a trade-off between precision and recall. Overgeneralization (high recall, low precision) occurs when systems incorrectly group unrelated words, while underspecification (low recall, high precision) happens when true inflectional variants are missed.
- Singleton paradigms are a challenge: Many systems tend to overgenerate singleton paradigms (paradigms with only one word), especially for agglutinative languages or when corpora are sparse. This suggests difficulty in identifying true inflectional relationships from limited data.
Key Results
- The Edinburgh system, a grammar-based approach, achieved the best overall performance with an average best-match F1 score of 67.96% on the test sets.
- Grammar-based systems (including CU-UBC and Edinburgh) outperformed the baseline by at least 23.06% F1.
- A supervised lemmatizer from the Stanza toolkit outperformed all unsupervised systems on most languages, demonstrating the significant gap between supervised and unsupervised performance.
- Performance varied greatly across languages, with English and Spanish yielding the highest scores, while Basque and Navajo were the most difficult.
Limitations and Future Work
- Data Sparsity: The JHUBC, while large, is still considered sparse for many languages, making it difficult to learn robust morphological patterns.
- Evaluation on Partial Paradigms: The evaluation was based on gold partial paradigms, meaning not all words in the corpus were covered. This limits the scope of evaluation and could potentially misrepresent system performance if the gold subset is particularly challenging.
- Distinguishing Inflection from Derivation: Accurately separating true inflectional relationships from derivational ones remains a significant challenge.
- Future Work: The authors envision future tasks expanding on this work to include:
- Paradigm slot categorization: Assigning grammatical labels (e.g., past tense, singular) to the forms within a paradigm.
- Lemma identification: Identifying a canonical form for each paradigm.
- Paradigm completion: Generating missing forms for a paradigm.
- Expansion to more under-resourced languages: Developing methods that are robust to limited data.
Summary updated