TL;DR
This paper explores the trade-offs between human linguist expertise and machine learning for building morphological grammars. They found that while neural models are powerful, hand-crafted grammars can match or surpass them, especially for complex languages, but require significant linguistic effort.
What Problem Does This Paper Address?
The paper tackles the question of whether hand-crafted morphological grammars, traditionally considered the "gold standard" for linguistic accuracy, can still compete with modern sequence-to-sequence (seq2seq) neural models. While seq2seq models have shown impressive success in learning morphological patterns from data, it's often assumed that a linguist could, in principle, create a more accurate system with enough effort. This research investigates the practical cost-benefit of this human-driven approach, particularly in the context of low-resource languages where rapid development of NLP tools is crucial. The paper also highlights the need for linguistically informed grammars for normative and prescriptive language applications, where neural models lack verifiable guarantees.
Methods
The core of the research involved a group of 19 students with linguistic training developing finite-state morphological grammars for various languages as part of a shared task. They utilized the foma finite-state tool. The development process included:
- Triage: Rapid analysis of training data to identify difficult inflectional classes and select languages for modeling.
- Computational Tools: Using tools to reconstruct partial paradigms, extract alphabets, canonicalize tag orders, and provide a rapid development environment with instant feedback.
- Template Grammar: Starting with a template that allowed for either morphophonology-only grammars or those requiring lemma classification.
- Inflection Table Clustering: A novel approach to group lexemes into potential inflectional classes based on a similarity metric derived from string transformations between word forms. This metric quantifies the overlap of transformation rules between lexemes.
- Lexeme Similarity Measure: A two-step process involving alignment of word forms and then generating a regular expression to capture the transformation, which is compiled into an FST.
- Paradigm Completion Model: A non-neural system that uses learned transformation rules from other lexemes to hypothesize missing forms in paradigms, employing a majority vote strategy with similarity-based weighting.
Datasets
The study used data from the 2020 SIGMORPHON shared task, which included training, development, and test sets for numerous languages. The authors selected languages from this task for their experiment.
Key Insights and Ideas
- Human Effort vs. Neural Power: Achieving accuracy comparable to state-of-the-art neural models with hand-crafted grammars requires substantial linguistic development effort, especially for languages with complex morphophonology.
- Linguist Advantage in Complexity: For highly complex languages (like Ingrian and Tagalog), linguists can indeed surpass neural models, but this comes at a significant cost in terms of development time.
- The "Guesser" Component: Integrating a "guesser" component into FST grammars is crucial for handling unseen lemmas, a common challenge in morphological analysis.
- Clustering for Insight: The developed inflection table clustering method provides a valuable tool for linguists to quickly hypothesize inflectional classes and analyze language structure.
- Overfitting is Less of a Concern: The linguist-driven development process, with continuous evaluation, showed a tendency for test set accuracy to be higher than training set accuracy, suggesting less susceptibility to overfitting compared to some machine learning approaches.
Key Results
- The hand-written grammars (CU-7565-01) matched or surpassed the strongest neural participants on 13 languages in the SIGMORPHON shared task.
- For complex languages like Ingrian (izh) and Tagalog (tgl), the linguist-developed grammars significantly improved upon other models.
- The non-neural paradigm completion model (CU-7565-02) performed reasonably well across nearly all 90 languages and is considered a strong improvement over baseline non-neural models.
- An estimate of at least 40 person-hours of work (excluding infrastructure and training) is suggested for developing a hand-written morphological analyzer and generator on par with neural models, though this varies greatly by language complexity.
Limitations and Future Work
- Data Quality: The authors observed that some languages had skewed data or inconsistencies that made it difficult for hand-written grammars to reach neural model performance. They hope to re-evaluate with cleaner test data.
- Complex Lemma Structures: Handling lemmas listed with multiple alternate forms (e.g., in Hiligaynon) was challenging and not attempted in this study.
- Ambiguous Data: Cases with identical lemma/MSD but different outputs, or target forms with alternatives separated by slashes, presented difficulties that were excluded from modeling.
- Future Work: The paper suggests that cleaning data for languages with inconsistencies would be beneficial. Further exploration of the developed tools and their application to more languages is also implied. The authors also note that their estimates do not include the time spent on student training in linguistics and computational modeling.
Summary updated