TL;DR
Large Language Models (LLMs) can generalize their morphological knowledge to new words, but their success is more influenced by a language's irregularity (integrative complexity) than its sheer number of forms (enumerative complexity) or training data volume. This suggests that for LLMs to truly master diverse languages, especially low-resource ones, we need to consider more than just data quantity.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown remarkable abilities in understanding and generating human language. However, a key question remains: can they truly generalize their linguistic knowledge beyond the data they were trained on, especially for complex linguistic phenomena like morphology? Morphology, the study of word formation and structure, is crucial for grammatical correctness. This paper investigates whether LLMs, like humans, can apply morphological rules to novel, unfamiliar words (nonce words) across different languages. Specifically, it probes how cross-linguistic differences in morphological complexity and the amount of training data affect LLMs' ability to generalize these rules.
Methods
The researchers adapted the classic "Wug Test," a well-established tool for assessing children's morphological knowledge, into a multilingual version. This involved translating 23 Wug Test questions into six languages: Vietnamese, French, Spanish, Romanian, Portuguese, and German. Language experts then validated these translations and adjusted the nonce words to fit each language's phonotactic rules.
Two LLMs, GPT-3.5 and GPT-4, were then prompted to complete these Wug Test questions in a zero-shot manner, meaning they were given the task without explicit examples of how to perform it. The models' generated inflected and derived forms were then evaluated by native speakers of each language, who rated the correctness on a scale of 1 to 5. These ratings were binarized into "correct" or "wrong" for analysis.
The study also quantified morphological complexity for each language using two measures:
- E-complexity (Enumerative Complexity): The number of inflectional paradigms and their degree of irregularity.
- I-complexity (Integrative Complexity): The predictability of inflected forms from context, essentially measuring irregularity.
Statistical models were used to analyze how these complexity measures and the proportion of training data for each language correlated with LLM accuracy.
Datasets
The primary "dataset" for this study was a newly created multilingual Wug Test, adapted from the original English Wug Test. The LLMs themselves were trained on massive, unspecified datasets. For estimating the proportion of training data per language, the authors referred to GPT-3 dataset statistics.
Key Insights and Ideas
- Morphology truly matters: LLMs demonstrate an ability to generalize morphological rules to novel words, but their success is not uniform across languages.
- Irregularity is the key challenge: Integrative complexity (I-complexity), which captures the irregularity of morphological forms, was found to be a significant predictor of LLM performance. Languages with more irregular morphology were harder for LLMs to generalize to.
- Number of forms is less critical: Enumerative complexity (E-complexity), representing the sheer number of possible word forms, did not significantly predict LLM performance. This suggests LLMs are not simply overwhelmed by a large number of potential inflections.
- Training data volume is surprisingly less influential: The amount of training data for a language had no significant impact on LLM accuracy in this study. This is a surprising finding, challenging the conventional wisdom that more data always leads to better performance.
- English bias is real: A significant error type observed was the "English fall-back," where LLMs applied English morphological rules to nonce words in other languages, likely due to English's dominance in training data.
Key Results
- Overall Accuracy: Both GPT-4 and GPT-3.5 generally performed well, with mean accuracies of 0.69 (SD = .46). GPT-4 scored slightly higher than GPT-3.5, but the difference was not statistically significant.
- Complexity vs. Accuracy:
- I-complexity (irregularity) had a significant, weak negative effect on accuracy (β = -.67, p = .0187).
- E-complexity (number of forms) did not predict accuracy (β = .10, p = .7463).
- Training Data vs. Accuracy: The amount of training data was not predictive of Wug test performance in either model (β = .10, p = .5853 and β = -.02, p = .9137).
- Error Types: Beyond "real-word bias," four main error types were identified: inflection ignorance, inflection mismatch, English fall-back, and real-word bias.
Limitations and Future Work
- Limited Models: The study only analyzed two LLMs (GPT-3.5 and GPT-4), prioritizing expert evaluation over broader model coverage.
- Language Expert Reliance: For most languages, only one language expert provided ratings, though interrater agreement was qualitatively checked for Vietnamese.
- Nonce Word Irregularity: The inherent irregularity of nonce words could vary across languages, potentially influencing results.
- Isolating Language Focus: Only one isolating language (Vietnamese) was included, limiting conclusions about this language type.
Future work could explore:
- The impact of tokenization strategies on LLM performance.
- Investigating the "English fall-back" error more deeply.
- Examining the role of generation attempts (k-value) in LLM performance, as seen in comparisons with previous studies.
- Expanding the analysis to more languages and LLMs.
Summary updated