TL;DR
This paper explores how to make neural networks better at predicting word inflections by combining them with linguistic "analogical patterns." They found that a model focusing on the differences between input and output word forms (Model 2) significantly outperformed a more standard encoder-decoder model, suggesting that focusing on the inflectional change itself is key to mimicking human linguistic intuition.
What Problem Does This Paper Address?
Predicting how words change form (like "walk" to "walked" or "sing" to "sang") is a fundamental aspect of language. While neural networks, particularly encoder-decoder (ED) models, have shown promise in this area, it's unclear if they truly capture the cognitive processes humans use. Specifically, the authors question whether ED models, which often learn mappings from raw input to raw output, are sufficient to replicate human judgments on novel word inflections. They aim to improve ED models by incorporating pre-compiled linguistic knowledge in the form of "analogical patterns" to see if this leads to more human-like predictions.
Methods
The core of the paper is the development and evaluation of four models for the SIGMORPHON 2021 Shared Task 0.
- Encoder-Decoder (ED) Architecture: The foundation for models M1 and M2 is a standard ED architecture using bidirectional LSTMs. This architecture takes a sequence (like a word's lemma) and transforms it into another sequence (the inflected form).
- Analogical Patterns (APs): The authors leverage two types of analogical patterns:
- Broad Alternation Patterns (BAPs): These capture commonalities and differences between two word forms through pairwise alignment, essentially identifying shared substrings and variations.
- Fine Alternation Patterns (FAPs): These are more sophisticated, identifying recurrent partials across an entire paradigm of word forms, aiming to capture more stable exponents (like suffixes). FAPs are derived by considering the overall system of word forms, not just isolated pairs.
- Model Variations:
- M1: Integrates both BAPs and FAPs directly into the input of the ED network alongside the lemma and UniMorph tag.
- M2: Focuses on FAPs, specifically the second part of the FAP (FAP2), which represents the inflectional change. The model is trained to predict this FAP2 rather than the full inflected form.
- M3: A baseline model that estimates word-likeness effects based on phonological similarity to attested inflected forms with the same UniMorph tag. It uses an LSTM-based binary classifier.
- M4: A simple baseline that uses the raw type frequency of the BAP relating the wug lemma and the wug form as a score.
Datasets
The study uses data from the SIGMORPHON 2021 Shared Task 0, which includes training, development, and test datasets for English (ENG), German (DEU), and Dutch (NLD). These datasets consist of lemma/form pairs annotated with UniMorph tags (UTs). The test sets contain "wug forms" (novel words) for which human judgments on their inflected forms are available for evaluation.
Key Insights and Ideas
- Focusing on the Change Matters: The most striking insight is the superior performance of M2, which predicts the inflectional pattern (FAP2) rather than the entire inflected word. This suggests that human-like inflection prediction might hinge more on understanding the transformation itself, rather than just mapping an input to an output.
- Analogical Patterns are Powerful: The inclusion of analogical patterns, particularly FAPs, proved crucial. M1, which included both BAPs and FAPs, performed poorly, but M2, which specifically leveraged FAPs, achieved excellent results.
- Beyond Simple Similarity: The paper challenges the idea that simple phonological similarity (as in M3) or raw frequency (as in M4) are sufficient for cognitively plausible inflection. While M4 performed surprisingly well on some languages, M2's consistent high performance across languages points to a more nuanced understanding of linguistic structure.
- Reconsidering "Islands of Reliability": The authors' M2 model achieved performance comparable to, and in some cases better than, established models like the Minimal Generalization Learner (MGL) on human judgments of English past tense. This suggests that ED models, when augmented with appropriate linguistic features like FAPs, might offer a more parsimonious explanation for human inflectional behavior than previously thought.
Key Results
- M2 Outperforms: Model M2 consistently achieved the best performance across all three languages (English, German, Dutch) based on AIC scores.
- M2 Ranked Second Overall: M2 secured the second-highest rank among all systems submitted to the SIGMORPHON Shared Task 0.
- M1 Underperforms: Despite a similar architecture to M2, Model M1 performed poorly, even worse than the simple baseline M4 on some languages, highlighting the importance of how analogical patterns are integrated.
- M4's Surprising Success: Model M4, a simple frequency-based baseline, performed surprisingly well on English and German data, though poorly on Dutch.
Limitations and Future Work
- Model M1's Poor Performance: The authors acknowledge the poor performance of M1 and conjecture that its architecture might not be optimal for integrating analogical patterns.
- Need for More Experiments: While M2's performance is promising, the authors suggest more experiments are needed to draw definitive conclusions.
- Future Directions:
- Develop specialized versions of M1 and M2 to assess the importance of specific inflectional series.
- Test models on more comprehensive datasets, potentially predicting inflected forms from other forms besides the lemma, or jointly from multiple forms within a lexeme.
- Explore the role of stems more deeply, as M2's success suggests focusing on the inflectional change might be more critical than accurately predicting the stem.
Summary updated