Abstract
The "Critical Period" (CP) hypothesis in linguistics posits a developmental window during which language acquisition is most efficient. This study investigates whether CP effects emerge naturally from statistical learning or require biological maturation. By training Transformer models (GPT-2 and RoBERTa) under various bilingual schedules, we demonstrate that standard language models (LMs) do not inherently exhibit critical periods; they remain highly plastic but suffer from catastrophic forgetting of their first language (L1). However, by introducing Elastic Weight Consolidation (EWC) to simulate a loss of plasticity, we successfully replicate human-like CP effects, suggesting that the human CP likely stems from innate biological mechanisms rather than statistical entrenchment alone.
Introduction
In human development, the Critical Period (CP) represents a phase where acquiring a first language (L1) is effortless, and subsequent second language (L2) acquisition becomes increasingly difficult after this window closes. A long-standing debate centers on whether this is due to innate biological maturation (nature) or statistical entrenchment from L1 learning (nurture). This research utilizes neural language models as a control group to determine if a pure statistical learner naturally develops a critical period through data exposure.
Methodology
The researchers evaluated two model architectures—GPT-2 (autoregressive) and RoBERTa (masked)—trained from scratch using five distinct schedules to simulate different linguistic experiences:
- Monolingual: Baseline training on a single language.
- Interleaved: Simultaneous exposure to L1 and L2 from the start.
- Sequential: Full L1 training followed by a complete switch to L2.
- Sequential + Interleaved: L1 training followed by joint L1 and L2 exposure (immersion).
- Sequential + EWC: L1 training followed by L2 learning with Elastic Weight Consolidation (EWC), a regularizer that penalizes changes to weights critical for L1, simulating a maturational loss of plasticity.
Models were trained on a balanced multilingual dataset comprising OpenSubtitles, Project Gutenberg, and Wikipedia, with English as the primary L2 and various languages (e.g., German, Finnish) as L1.
Key Results
The study yielded several critical insights into the nature of learning in Transformers compared to humans:
Absence of a Natural Critical Period
Standard LMs proved to be "too plastic." In the sequential condition without regularization, L2 proficiency was comparable to or better than the interleaved baseline, regardless of the "age" at which the model was exposed to the new language. This suggests that statistical learning alone does not produce a critical period.
Catastrophic Forgetting vs. Attrition
Unlike humans, who retain their native language even after years of L2 immersion, standard LMs in the sequential condition experienced "catastrophic forgetting," where L1 knowledge was almost entirely overwritten by L2. This is evidenced by a sharp spike in L1 perplexity when the training switched to L2.
Inducing Human-like Effects with EWC
By applying EWC, the researchers "reverse-engineered" the critical period. The regularizer prevented the loss of L1 knowledge (reducing attrition) but significantly hindered the model's ability to achieve native-level proficiency in L2. For instance, GPT-2's English perplexity was notably higher when it had to protect its L1 weights.
The Role of Language Similarity
L2 acquisition was facilitated by structural similarity between L1 and L2. Interestingly, pre-training on Java (a programming language) improved performance on English grammatical tasks (BLiMP) more than some natural languages, likely due to the strict structural requirements of code.
Discussion and Conclusion
The findings suggest that the human critical period is likely not an inevitable byproduct of statistical learning. Instead, the results point toward the necessity of innate biological mechanisms—simulated here by EWC—that reduce neural plasticity over time. While LMs are powerful learners, their inherent plasticity makes them prone to forgetting, a stark contrast to the stable linguistic foundations seen in humans. Future work may explore multimodal models and more biologically plausible regularizers, such as synaptic pruning or myelination, to further bridge the gap between artificial and biological intelligence.
Summary updated