TL;DR
This paper presents a novel approach to modeling morphological inflection that mimics child language acquisition, focusing on producing plausible errors rather than perfect forms. By adapting a character-based transformer model with techniques like upsampling and incorporating phonological neighborhood density, the BGU-MeLeL system achieved strong correlations with child-like error patterns, outperforming a baseline that aimed for perfect inflections.
What Problem Does This Paper Address?
The SIGMORPHON 2023 Shared Task on Morphological Inflection introduces a unique challenge: instead of building systems that generate perfectly inflected words, the goal is to create models that mimic the errors children make when learning morphology. This shifts the focus from maximizing accuracy to understanding and replicating the developmental process of language acquisition, specifically how children learn to inflect nouns in languages like Polish, Estonian, and Finnish. Previous research often focused on single languages, but this task requires a cross-linguistic approach to identify universal aspects of acquisition.
Methods
The BGU-MeLeL team adapted a state-of-the-art character-based transformer model, originally designed for tasks like machine translation and grapheme-to-phoneme conversion. Key modifications for this task include:
- Childlike Error Modeling: The primary goal is to produce plausible errors, not perfect inflections. This is achieved by incorporating features that correlate with children's learning patterns.
- Surface-Form Frequency: The model leverages the frequency of specific inflected forms in the input data. This is implemented through upsampling, where rarer forms are artificially increased in the training data to better reflect their occurrence in a child's linguistic environment. Additionally, surface-form frequency embeddings are used to feed this information directly into the model.
- Phonological Neighborhood Density (PND): This concept, which measures how many similar-sounding words exist for a given form, is theorized to influence learning. The authors suggest that the transformer's ability to process phonetic representations (IPA) inherently allows it to capture some aspects of PND, as similar pronunciations lead to similar hidden states.
- Multilingual Training: To generalize across languages, the model was trained multilingually by adding language tags to the input.
- Smaller Model Architecture: Experiments suggested that a smaller transformer model (fewer layers, heads, and smaller hidden dimensions) better approximated childlike performance.
Datasets
The paper focuses on data for three languages: Polish, Estonian, and Finnish. The task uses a dataset where instances are limited to single-feature inflection of lemmas into various grammatical cases. Both orthographic and phonetic (IPA) forms are provided, along with the surface-form frequency of the lemma and the error rate of children's inflections. A key characteristic of the dataset is that lemmas in the training set do not appear in the test set, preventing simple memorization.
Key Insights and Ideas
- Mimicking Errors is Key: The most significant insight is that focusing on replicating the process of learning, including its imperfections, is crucial for modeling child language acquisition. A system aiming for perfect output is inherently misaligned with this goal.
- Frequency Matters: Surface-form frequency is a strong predictor of how accurately children inflect words. Upsampling and embedding this information proved effective.
- Phonology's Role: While not explicitly engineered with a PND module, the transformer's ability to process phonetic information suggests it can implicitly capture some of the effects of phonological similarity on learning.
- Complementary Methods: Combining upsampling, frequency embeddings, and multilingual training yielded the best results, indicating that these approaches address different facets of the acquisition process.
Key Results
The BGU-MeLeL system demonstrated substantial improvement in correlation with children's error rates compared to a baseline that aimed for perfect inflections. The best performing model, which incorporated all three methods (smaller model, upsampling, and multilingual training), showed the strongest correlation. Specifically, the paper highlights that while the baseline achieved perfect accuracy (1.0000) and zero edit distance, its correlations were poor (-0.029 Pearson, -0.061 Spearman). In contrast, the submitted model achieved better correlations, with the best overall performance reported in Table 1. Table 2 provides per-language results, showing varying levels of "in top-10" predictions and correlation scores across Polish, Estonian, and Finnish.
Limitations and Future Work
- Implicit PND: While the authors suggest the transformer can implicitly capture PND, this aspect is not directly engineered or explicitly controlled. Future work could explore more direct methods for incorporating PND.
- "Forgetfulness" Tuning: The paper mentions modifying hyperparameters to add "forgetfulness," but the specifics and impact of this tuning are not detailed.
- Limited Scope: The task focuses on noun inflection and specific grammatical cases. Expanding to other word classes and inflectional phenomena would be a natural next step.
- Error Analysis: While the system correlates well with child errors, a deeper qualitative analysis of the types of errors produced by the model compared to children would be valuable.
Summary updated