Developmental Trajectories of Neural Representations

This direction is split into other directions: longitudinal checkpoint analysis. Its last version stays readable and citable here.

Version 1 of 1166 papers169 conceptsfrozen 2026-10-04

In machine learning, we typically evaluate models as static artifacts—finished products defined by their final weights. However, a growing body of research suggests that the most profound insights into artificial intelligence come not from the final state, but from the longitudinal trajectory of the training process. By treating training epochs as developmental stages, researchers are discovering that models follow a predictable, often biological-like sequence of "maturation."

The Chronology of Competence

When a model begins training, it doesn't learn everything at once. Instead, it follows a strict hierarchy of acquisition. Early in the pre-training of models like RoBERTa, linguistic skills such as syntax and part-of-speech tagging are mastered almost instantly. As Yizhong Wang and Noah A. Smith demonstrate in Probing Across Time, while surface-level grammar stabilizes early, factual knowledge trickles in slowly, and complex reasoning remains elusive even after millions of steps.

This "syntax-first" phenomenon is a recurring theme. Using Singular Vector Canonical Correlation Analysis (SVCCA), Naomi Saphra and Adam Lopez found that models prioritize local structural patterns before moving on to global semantic topics (Understanding Learning Dynamics Of Language Models with SVCCA). This suggests that "learning to speak" (grammar) is a computational prerequisite for "learning to know" (facts).

Convergent Evolution with the Human Brain

Perhaps the most striking finding in this field is that AI models often "evolve" toward human-like internal representations, even without explicit biological constraints. Guobin Shen and Yi Zeng show that as models become more capable, their internal activations naturally align with human brain activity patterns (Convergent Evolution across Modalities, Scales and Training Trajectories). Crucially, this alignment often emerges before the model shows major performance gains, suggesting that brain-like strategies are a necessary stepping stone for high-level intelligence.

This developmental mirroring extends to specific cognitive domains. Sashank Varma and Zekun Wang found that computer vision models develop geometric sensitivities and a "mental number line" in stages that mirror human child development (Computer Vision Modeling of the Development of Geometric and Numerical Concepts in Humans).

The Point of Divergence

While models start by mimicking human-like learning, they eventually reach a "tipping point" where they outgrow biological constraints. Byung-Doh Oh and William Schuler identified a "sweet spot" at approximately two billion training tokens where model surprisal best predicts human reading times; beyond this, the model becomes "superhuman" and its internal logic begins to diverge from natural human processing (Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens).

Badr AlKhamissi argues that this divergence happens because models move from formal linguistic competence (which humans share) to a type of functional competence that relies on statistical scales humans cannot access (From Language to Cognition: How LLMs Outgrow the Human Language Network).

Mechanistic Emergence and Stability

By looking under the hood during training, researchers can see the literal "wiring" of these capabilities. Advey Nandan and colleagues discovered "universal neurons"—units that perform the same function across different training runs—which emerge early and remain stable throughout the model's life (Universal Neurons in GPT-2: Emergence, Persistence, and Functional Impact).

In bilingual models, this process involves a transition from language-specific features to shared semantic concepts. Tatsuro Inaba and Masaru Isonuma used Sparse Autoencoders to show that models initially learn languages independently before gradually aligning them into a unified conceptual space (How LLMs Learn: Tracing Internal Representations with Sparse Autoencoders).

Why the Trajectory Matters

Understanding the when of training allows for practical interventions. For instance, Tanya Goyal and Greg Durrett found that summarization models learn to copy text early but learn to hallucinate late. By identifying this window, they developed "Loss Truncation" to stop the model from internalizing noise before it becomes a stable part of the representation (Training Dynamics for Text Summarization Models).

Ultimately, longitudinal analysis shifts our view of AI from a "black box" to a "growing organism." It reveals that the path to intelligence is not a straight line of optimization, but a complex journey of emergence, consolidation, and eventual divergence from its human creators.

Papers in this version

Showing 60 of 166 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_developmental_trajectories_of_neural_rep_b8904093,
  title = {Developmental Trajectories of Neural Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/developmental-trajectories-of-neural-representations/rdr_b8904093a7f24a49b277691fa1ea8b7e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_developmental_trajectories_of_neural_rep_b8904093_v1,
  title = {Developmental Trajectories of Neural Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/developmental-trajectories-of-neural-representations/rdr_b8904093a7f24a49b277691fa1ea8b7e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}