Developmental Interpretability in Deep Learning

123 concepts102 papers1 proposal

To understand how a complex biological organism functions, a biologist does not just examine a fully grown adult; they study its embryology and development. For years, the field of mechanistic interpretability has taken the opposite approach with artificial neural networks. It has largely operated as a post-hoc "autopsy," probing frozen, fully trained checkpoints to locate where specific features or capabilities reside.

Developmental interpretability flips this paradigm. Instead of treating a model as a static artifact, it investigates the training trajectory itself—examining how internal structures, circuits, and geometric representations self-organize, mature, and stabilize from the very first gradient step to final convergence. As argued by Ryan Teehan et al., shifting our focus to these training dynamics fills a critical gap, moving us from merely cataloging what a model knows to explaining the developmental process behind how it learns. Martina G. Vilas et al. reinforce this by drawing on cognitive neuroscience, noting that a truly mechanistic theory of AI must bridge the gap between static network architectures and the transient learning dynamics that shape them.

The Non-Linear Path of Learning: Phase Transitions

When observing a model's training loss, the curve typically looks like a smooth, continuous decay. However, looking under the hood reveals that the acquisition of internal capabilities is often jagged and non-linear, punctuated by sudden macroscopic shifts.

For example, language models do not master grammar incrementally. Angelica Chen et al. demonstrated that Masked Language Models acquire syntactic structures during abrupt "phase transitions," which correlate with sudden drops in training loss.

Smooth Loss Curve:      ════════════════════════════════════>
Internal Circuits:     [Subcircuit A] ──┐
                       [Subcircuit B] ──┼─> [Sudden Phase Transition: Capability Emerges]
                       [Subcircuit C] ──┘

How do these sudden capabilities materialize out of chaotic weights? To find out, Aaditya K Singh et al. tracked the formation of "induction heads"—the fundamental circuits responsible for in-context learning in Transformers. By causally clamping activations during training, they discovered that the seemingly instantaneous emergence of an induction head is actually driven by the continuous, smooth maturation of three separate, underlying subcircuits. When these subcircuits reach a critical threshold of readiness, they interact and lock into place, creating a sudden macro-level shift in capability.

This pattern is supported by theoretical work in simpler architectures. Scott Pesme and Nicolas Flammarion proved that even in two-layer diagonal linear networks, optimization is characterized by "saddle-to-saddle" dynamics, where the network's weights jump abruptly between different sparse solutions rather than climbing a smooth gradient.

Geometric Crystallization and Dimensional Cycles

As these circuits mature, the geometry of the network's representation space undergoes a profound structural evolution.

In the initial stages of training, models often undergo an "expansion-compression" cycle. As observed in transformer models, the network first unfolds raw input data by expanding it into a high-dimensional space, separating entangled features. Once these features are isolated, the model compresses them into highly refined, low-dimensional representations.

This structural reorganization happens in distinct qualitative phases. Tracking pre-training snapshots via sparse crosscoders, Xuyang Ge et al. identified a clear two-phase process:

  1. The Statistical Phase: Early in training, the model primarily internalizes superficial data properties, such as raw word frequencies.
  2. The Feature Learning Phase: Later, the model reorganizes its internal parameters to form sparse, abstract concepts that directly drive downstream reasoning capabilities.

As the model transitions into this second phase, its internal logic begins to "crystallize." Elena Terzieva et al. tracked feature attributions over time and found that as optimization progresses, the network's explanations become significantly more intense and less chaotic, stabilizing into a coherent decision-making strategy regardless of whether the model is generalizing or overfitting.

Transient Scaffolding: The Role of Vestigial Structures

Perhaps the most surprising insight from developmental interpretability is that the internal structures present during early training are not just immature versions of the final features. Some act as temporary "scaffolding"—essential for building representations early on, but discarded or rendered redundant by the end of training.

Consider class-selective neurons, individual units that fire exclusively in response to specific categories (like "dogs" or "cars"). In fully trained networks, these neurons present a paradox: you can delete them with virtually no impact on final accuracy, leading many researchers to dismiss them as accidental byproducts of training.

However, Omkar Ranadive resolved this paradox by analyzing their developmental trajectory. The author showed that class-selective neurons play a vital, transient role during the early phases of training, serving as the primary mechanism for driving initial representation formation. Once the network establishes a more distributed, robust geometry, these individual neurons become obsolete. Evaluating a model solely on its final state mistakes these crucial historical scaffolds for useless, vestigial structures.

Methodologies for Tracking Evolution

To observe these hidden dynamics without disrupting the learning process, researchers have developed specialized diagnostic tools that track representations across both training time (epochs) and network depth (layers).

Training Time (Epochs) ──>
┌───────────┐      ┌───────────┐      ┌───────────┐
│ Snapshot  │ ───> │ Snapshot  │ ───> │ Snapshot  │  ==> Tracked via Probes,
│  Epoch 1  │      │  Epoch 10 │      │ Epoch 100 │      RTD, and DSA
└───────────┘      └───────────┘      └───────────┘
  • Linear Classifier Probes: Introduced by Yoshua Bengio, these non-intrusive diagnostics are attached to intermediate layers across different training iterations. By measuring how easily a simple linear model can extract features at a given layer, researchers can map the spatial and temporal progression of learning, identifying exactly when and where specific features become "ready."
  • Representation Topology Divergence (RTD): Developed by Serguei Barannikov, RTD uses Topological Data Analysis (TDA) to quantify the multi-scale geometric structure of data manifolds. This allows researchers to mathematically measure how far a representation has structurally diverged from its initialization or from other training runs.
  • Dynamical Similarity Analysis (DSA): Because recurrent networks compute through time-evolving states, traditional spatial metrics often fail to capture their behavior. Mitchell Ostrow and Adam Eisen introduced DSA, which uses Koopman operator theory to compare the underlying "flow" (vector fields) of networks. They demonstrated that the specific learning rules used during optimization leave unique, identifiable signatures in how these latent dynamics evolve over time.

By treating neural networks as dynamic, evolving systems rather than static black boxes, developmental interpretability provides the tools and conceptual framework needed to understand not just the final state of AI, but the fundamental principles governing how machines learn.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_developmental_interpretability_62d12277,
  title = {developmental interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/developmental-interpretability/rdr_62d1227760b2487f82c910bc670d02ff},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_developmental_interpretability_62d12277_v1,
  title = {developmental interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/developmental-interpretability/rdr_62d1227760b2487f82c910bc670d02ff/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake