TL;DR
This paper proposes a novel three-stage framework to evaluate language models (LMs) based on human language acquisition stages. They found that while newer LMs outperform older ones, their development doesn't mirror human learning, suggesting data characteristics (register theory) are more influential than developmental stages.
What Problem Does This Paper Address?
The rapid advancement of large language models (LMs) has led to impressive capabilities in understanding and generating human language. However, current evaluation methods often focus on specific tasks or general benchmarks, failing to capture how LMs develop their language abilities in a way that parallels human language acquisition. This paper aims to bridge this gap by evaluating LMs through the lens of established psycholinguistic theories of human language development, seeking to understand if LMs follow a similar developmental trajectory and what factors influence their progress.
Methods
The authors developed a three-stage framework inspired by classical human language development theories:
- Stage I (0-6 years): Focuses on basic vocabulary understanding, simple syntax, and preliminary reasoning.
- Stage II (6-12 years): Emphasizes mastering complex grammar, semantics, and logical relationships.
- Stage III (12+ years): Assesses advanced language abilities, including complex reasoning and abstract thinking.
They then designed specific datasets and tasks to evaluate LMs across these stages, including:
- one-word understanding: Multiple-choice questions to test vocabulary recognition.
- agent-action-object (AAO): Distinguishing grammatically correct from incorrect sentences.
- bc-if-why: Testing understanding of conditionality and causality.
- grammar-comp & BLiMP-comp: Evaluating complex grammatical structures.
- COLA: Grammatical correctness judgment.
- WiC: Word sense disambiguation.
- ReClor: Complex logical reasoning.
Additionally, they analyzed LM generation abilities using five linguistic dimensions: noun usage (NN), average word length (AWL), clause usage, type-token ratio (TTR), and auxiliary verb usage. They also considered register theory, which posits that language use varies based on context and situation.
Datasets
The paper utilizes a variety of existing datasets for evaluation, including:
- Oxford Learner's Dictionary: For one-word understanding.
- BLiMP (Warstadt et al., 2023): For AAO and BLiMP-comp tasks.
- Multi-Genre Natural Language Inference (MNLI) dataset (Williams et al., 2018): For bc-if-why and grammar-comp tasks.
- COLA (Warstadt et al., 2018): For grammatical correctness.
- WiC (Pilehvar and Camacho-Collados, 2019): For word sense disambiguation.
- ReClor (Yu et al., 2020): For logical reasoning.
No new datasets were introduced in this paper.
Key Insights and Ideas
- LMs don't mirror human acquisition: Despite advancements, LMs don't follow the same developmental stages as humans. They might perform well in later stages but struggle with earlier, foundational aspects.
- Data characteristics matter more than developmental stages: Register theory provides a better explanation for LM performance differences than human developmental trajectories. The linguistic features and biases present in training data significantly shape LM capabilities.
- "Easy" linguistic features show little progress: LMs exhibit minimal improvement over time in areas like average word length, clauses, and auxiliary verbs, likely because these features are highly represented and less varied in training corpora.
- Architecture and scale have nuanced effects: While larger models generally perform better, the paper notes that architecture (encoder vs. decoder) and scale don't always lead to significant differences across all tasks. Encoder models show promise in classification, while decoder models excel in certain comparison tasks.
- Generation abilities vary: LMs show more variation from human performance in noun usage and type-token ratio, suggesting they may rely on more frequent patterns rather than flexible, context-dependent expression like humans.
Key Results
- Overall performance: Newer LMs generally outperform older ones, but performance varies by task.
- Stage I performance: Most models performed well in Stage I tasks, though some, like T5 and RoBERTa, lagged in one-word understanding due to issues with contextual information and collocation.
- Stage II performance: LMs generally performed well in Stage II, indicating mastery of complex grammar, though specific grammatical phenomena within BLiMP-comp showed significant differences.
- Stage III performance: Performance differences became more pronounced in Stage III, particularly in the ReClor task, where larger models showed an advantage.
- Generation metrics: Humans exhibit more variation in noun usage and type-token ratio compared to LMs. LMs showed smaller differences from humans in average word length, clause usage, and auxiliary verbs.
- Data scale impact: A trend was observed where larger pretraining data scales generally led to better performance across stages, though not always linearly.
Limitations and Future Work
- Dataset limitations: The evaluation was limited to text-based datasets, excluding multimodal information (images, speech) crucial for certain aspects of human language acquisition.
- Accessibility of datasets: Some datasets used were not commercially available, posing challenges for future research.
- Rapid LM advancements: The fast pace of LM development necessitates continuous evaluation.
- Individual differences: The paper acknowledges that human language abilities show individual differences, which LMs currently do not model.
Future work could involve creating multimodal datasets, developing open-source evaluation benchmarks, and exploring how LMs can better model individual variations in language abilities.
Summary updated