SIGMORPHON–UniMorph 2022 Shared Task 0: Modeling Inflection in Language Acquisition

Jordan Kodner, Salam KhalifaNAACL 2022

TL;DR

This paper presents the results of a shared task that challenged AI models to mimic human language acquisition by learning morphological inflection from child-directed speech. The study reveals that while current models achieve impressive accuracy, they often exhibit "inhuman" over-regularization and over-irregularization, highlighting the complexities of truly human-like language learning.

What Problem Does This Paper Address?

The paper tackles the challenge of building AI systems that don't just learn language rules but do so in a way that mirrors how children acquire language. Specifically, it focuses on morphological inflection – how words change form (like adding "-ed" to make a past tense) – and aims to understand how well current AI models replicate the learning process and error patterns observed in children. This is crucial for developing more robust and human-like natural language processing systems. The "Past Tense Debate" in linguistics, concerning how the brain represents and processes regular vs. irregular forms, provides a historical backdrop for this investigation.

Methods

The core of the methodology involved a shared task where participants submitted AI systems designed to predict inflected word forms given a lemma (base word) and its morphological features. Key aspects include:

  • Child-Directed Speech (CDS) Data: Training data was extracted from corpora of child-directed speech, mimicking the linguistic input children receive. This approach aims to approximate natural learning trajectories.
  • Nested Training Sets: Systems were trained on progressively larger datasets to observe how performance scales with exposure, similar to a child's increasing vocabulary.
  • Focus on Generalization and Errors: The evaluation prioritized not just accuracy but also the types of errors made, comparing them to known human developmental patterns.
  • Three Languages: The task focused on English past tense, German noun plurals, and Arabic noun plurals, chosen for their varying complexities and extensive study in developmental linguistics.
  • Neural and Non-Neural Baselines: The submitted systems were compared against established neural (character-level transformer) and non-neural baselines.

Datasets

  • English: Data extracted from the Brown and Brent corpora (part of CHILDES), weighted by CHILDES frequencies.
  • German: Data sourced from the CHILDES Leo corpus and matched with German UniMorph, augmented with gender tags.
  • Arabic: Data from the Penn Arabic Treebank (PATB), enriched with additional features.

The paper describes the creation of nested training, development, and test sets for each language, with specific sampling strategies to reflect realistic learning scenarios.

Key Insights and Ideas

  • Superhuman Speed on Simple Patterns: Models learned the productive English "-ed" past tense pattern extremely quickly (after only 100 training items), far outpacing human children.
  • The "Inhuman" Over-Irregularization Problem: A major finding is the persistent tendency of models to over-apply irregular patterns to regular words (e.g., bring-brang) or apply complex irregular patterns where they don't belong. This is a stark contrast to human learners, who tend to under-apply these rarer patterns.
  • Frequency Matching vs. Deep Understanding: While models perform well, it's unclear if their success stems from a deep understanding of morphological rules or simply from sophisticated frequency matching of the training data, especially for simpler patterns.
  • Single-Route Models' Hallmarks: The systems, largely based on neural single-route models, consistently showed a tendency towards over-irregularization, suggesting this might be an inherent characteristic of such architectures compared to dual-route models.
  • Arabic as the Toughest Challenge: The complex Arabic noun pluralization system proved to be the most difficult for all systems, with significantly lower accuracy compared to English and German.

Key Results

  • Overall Accuracy: Performance was generally good, with English past tense being the easiest, followed by German noun plurals, and then Arabic noun plurals.
  • English Past Tense: Systems achieved high accuracy, but none outperformed a simple baseline that always predicts "-ed," indicating a struggle with truly irregular forms. Over-regularization was the dominant error type.
  • German Noun Plurals: CLUZH achieved the highest accuracies. Systems showed a tendency to under-apply rarer plural suffixes (-er, -s) and over-apply the common -e and -(e)n suffixes.
  • Arabic Noun Plurals: Accuracy was significantly lower across all systems. A major issue was the over-production of sound-to-broken and broken-to-broken errors, which are rare in child productions.

Limitations and Future Work

  • Human-like vs. Accurate: The paper highlights that high accuracy doesn't necessarily equate to human-like learning. The models' error patterns, particularly over-irregularization, diverge significantly from children's.
  • Single-Route Models: The lack of dual-route model submissions means the findings are specific to single-route architectures. Future work could explore how dual-route models perform on this task.
  • Data Limitations: While CDS data was used, it's still a simplified input compared to the rich multimodal and social context children learn from.
  • Future Directions: The authors suggest further work is needed to understand the mechanisms behind over-irregularization and to develop models that exhibit more human-like error patterns, potentially by exploring different architectures or training paradigms. The goal is to move beyond mere prediction accuracy towards models that offer genuine insights into cognitive processes.

Summary updated

BibTeX
@inproceedings{kodner2022sigmorphonunimorph,
  author = {Jordan Kodner and Salam Khalifa},
  title = {{SIGMORPHON–UniMorph 2022 Shared Task 0: Modeling Inflection in Language Acquisition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure