Word Acquisition in Neural Language Models

Tyler A. Chang, Benjamin K. Bergen

TL;DR

This paper investigates how neural language models (LMs) like BERT and GPT-2 learn individual words during training, comparing their "word acquisition" to that of human children. They find that while LMs heavily rely on word frequency, they differ significantly from children in how they learn based on word length, concreteness, and lexical class, highlighting the importance of real-world interaction for human language development.

What Problem Does This Paper Address?

Understanding how neural language models learn is crucial for building better AI. While many studies evaluate fully trained models, this paper focuses on the process of learning, specifically how individual words are acquired during training. This is analogous to how developmental psychologists study child language acquisition. The key gap is the lack of direct comparison between the learning trajectories of LMs and children, which could reveal fundamental differences in their learning mechanisms and inform the development of more human-like AI.

Methods

The researchers trained four different language models: unidirectional and bidirectional LSTMs, and Transformer-based models (BERT and GPT-2). For each model, they tracked the "surprisal" (a measure of how unexpected a word is given its context) for over 600 words from the MacArthur-Bates Communicative Development Inventory (CDI) across different stages of training. This allowed them to generate "learning curves" for each word, showing how the model's prediction accuracy improved over time. They then defined an "age of acquisition" for each word in the LM as the training step at which it was considered "learned" (analogous to 50% of children producing the word). Finally, they used linear regression to identify which linguistic features (like word frequency, length, concreteness, and lexical class) best predicted these ages of acquisition in both LMs and children.

Datasets

  • Training Data: A combined corpus of BookCorpus and WikiText-103.
  • Evaluation Data: The MacArthur-Bates Communicative Development Inventory (CDI) for word acquisition data.
  • Child Language Data: The CHILDES corpus (North American English) for child-directed speech, used to derive predictors for child word acquisition.

Key Insights and Ideas

  • Frequency is King (for LMs): Language models are overwhelmingly driven by word frequency. More frequent words are learned much earlier than in children, and this effect is far more pronounced in LMs.
  • Children Learn Differently: The effects of word length, concreteness, and lexical class on word acquisition are significantly different between LMs and children. This suggests that LMs, relying solely on distributional statistics, miss crucial aspects of human learning.
  • Sensorimotor Experience Matters: The lack of a concreteness effect in LMs, despite its importance for children, highlights how real-world, sensorimotor experience is vital for human language acquisition, something LMs lack.
  • Early LM Training is Simple: LMs initially approximate simple unigram (single word frequency) and then bigram (two-word sequence) probabilities before developing more nuanced understanding. This pattern holds across different architectures.

Key Results

  • Log-frequency was a strong predictor of age of acquisition in both LMs (adjusted R² = 0.91-0.94) and children (adjusted R² = 0.01), but the effect was vastly stronger in LMs.
  • Word length (n-chars) showed a negative effect in LMs (longer words learned earlier), contrasting with children (shorter words learned earlier). This might be due to polysemy in shorter words.
  • Concreteness had no significant effect on LM word acquisition, unlike in children where concrete words are learned earlier.
  • Lexical class effects differed: unidirectional LMs learned nouns and function words later than adjectives and verbs, whereas children learning English tend to learn nouns earlier. Bidirectional models (BERT, BiLSTM) showed no significant lexical class effects, suggesting other predictors accounted for this.
  • MLU (mean utterance length) had a positive effect in most LMs (longer utterances led to later word learning), mirroring findings in children.
  • First words learned by LMs were often function words or pronouns due to their high frequency, while children's first words were typically single-word expressions.

Limitations and Future Work

  • Dataset Differences: The language model training corpus was vastly larger than the CHILDES corpus used for child data, potentially confounding direct comparisons.
  • "Learned" Definition: The definition of a word being "learned" (a specific surprisal cutoff) is an approximation and alternative definitions were explored with similar results.
  • Future Work: The authors suggest investigating the acquisition of syntactic structures and semantic information in language models. They also propose that future work integrating multimodal information and world knowledge (e.g., visual and linguistic inputs) could lead to more human-like language acquisition in AI.

Summary updated

Abstract

We investigate how neural language models acquire individual words during training, extracting learning curves and ages of acquisition for over 600 words on the MacArthur-Bates Communicative Development Inventory (Fenson et al., 2007). Drawing on studies of word acquisition in children, we evaluate multiple predictors for words’ ages of acquisition in LSTMs, BERT, and GPT-2. We find that the effects of concreteness, word length, and lexical class are pointedly different in children and language models, reinforcing the importance of interaction and sensorimotor experience in child language acquisition. Language models rely far more on word frequency than children, but, like children, they exhibit slower learning of words in longer utterances. Interestingly, models follow consistent patterns during training for both unidirectional and bidirectional models, and for both LSTM and Transformer architectures. Models predict based on unigram token frequencies early in training, before transitioning loosely to bigram probabilities, eventually converging on more nuanced predictions. These results shed light on the role of distributional learning mechanisms in children, while also providing insights for more human-like language acquisition in language models.
BibTeX
@article{chang2022word,
  author = {Tyler A. Chang and Benjamin K. Bergen},
  title = {{Word Acquisition in Neural Language Models}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=TBK4I8W24B},
}
Figure