Development of Cognitive Intelligence in Pre-trained Language Models

Raj Sanjay Shah, Khushi Bhardwaj, Sashank VarmaEMNLP 2024 · DOI

Abstract

This research investigates whether the developmental trajectories of Large Language Models (LLMs) during pre-training mirror the cognitive growth observed in human children. By evaluating ten popular LLM families across four core domains of human intelligence—numerical, linguistic, conceptual, and fluid reasoning—we identify a "window of maximal alignment" (typically between 100M and 20B tokens). During this phase, models most closely track human developmental stages before diverging toward engineering-driven optimization.

Introduction

While Large Language Models (LLMs) frequently achieve human-level performance on various benchmarks, the path they take to reach this proficiency remains fundamentally different from human biological development. Humans learn from a few million words, whereas LLMs require trillions of tokens. This study addresses the "developmental alignment" gap: do Pre-trained Language Models (PLMs) acquire skills in the same order and manner as humans? By treating PLMs as subjects of psychometric testing throughout their training, we explore whether these models can serve as scientifically plausible models of human cognition or if they are primarily "stochastic parrots" optimized for a loss function unrelated to biological intelligence.

Methodology

The evaluation is grounded in the Cattell-Horn-Carroll (CHC) theory, a prominent psychometric framework of human intelligence. The researchers focused heavily on the Pythia suite, which provides 154 intermediate checkpoints for eight different model sizes, allowing for a granular analysis of training trajectories.

Overview of the four cognitive intelligence domains evaluated: numeric abilities, linguistic abilities, concept understanding, and fluid reasoning.

The four intelligence domains were operationalized as follows:

  • Numeric Abilities: Testing for the "Mental Number Line" using latent representations to observe the "distance effect" (the relative ease of distinguishing widely separated numbers versus adjacent ones).
  • Linguistic Abilities: Utilizing the Benchmark of Linguistic Minimal Pairs (BLiMP) to assess morphological, syntactic, and semantic acceptability.
  • Concept Understanding: Measuring "typicality effects" (e.g., the recognition that a sparrow is a more "typical" bird than a penguin) using surprisal values and prompting.
  • Fluid Reasoning: Employing a textual adaptation of Raven’s Progressive Matrices (I-Raven), requiring models to identify abstract visual patterns described as numerical tuples.

Results and Developmental Insights

The study reveals several key phases and patterns in the cognitive development of LLMs:

The Developmental Window

A critical "sweet spot" in training exists between approximately 100 million and 20 billion tokens. In this window, cognitive alignment with human profiles is most monotonic and rapid. Beyond this point, while further training continues to reduce the model's loss (the engineering objective), alignment with human cognitive stages often plateaus or becomes unstable.

Training trajectories across the four domains, highlighting the developmental window (shaded blue) where alignment with human cognitive profiles is most rapid.

Domain-Specific Findings

  • Linguistic Hierarchy: Across all evaluated models, accuracy followed a consistent hierarchy: Morphology > Syntax > Semantics. Basic linguistic structures are mastered early in the training process.
  • Numerical Consistency: All models exhibited a human-like "distance effect" in their latent spaces. Interestingly, scaling the model size did not significantly enhance this specific alignment, suggesting it is a fundamental property of the training objective or data.
  • Fluid Reasoning and Scaling: Unlike basic numerical or conceptual tasks, Fluid Reasoning (RPM) showed significant gains with both increased model size and longer pre-training. This mirrors human development, where fluid reasoning develops through adolescence and benefits from increased cognitive capacity.
  • Commercial Performance: Advanced models like GPT-4 and Gemini-1-Pro significantly outperformed open-source models in conceptual typicality, likely due to massive scale and Reinforcement Learning from Human Feedback (RLHF).

Discussion and Limitations

The findings suggest a weak parallelism between LLM training and human growth. Models master linguistic abilities early, while complex reasoning emerges much later. However, several limitations remain. The trajectory analysis relied heavily on the Pythia suite; different architectures might exhibit unique developmental paths. Furthermore, even the "developmental window" of 100M+ tokens represents an input scale orders of magnitude larger than what a human child receives. Future research should expand this psychometric battery to include spatial and commonsense reasoning to further probe the limits of developmental alignment.

This paper was presented at EMNLP 2024.

Summary updated

BibTeX
@inproceedings{shah2024development,
  author = {Raj Sanjay Shah and Khushi Bhardwaj and Sashank Varma},
  title = {{Development of Cognitive Intelligence in Pre-trained Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.539},
}
Figure