TL;DR
This paper introduces the concept of "feature life history" in language models, revealing a stable "carrier scaffold" of features that emerges early in training and forms the backbone of the model's representations. This scaffold is crucial for model performance, predictable from early feature function, and organizes the majority of later-emerging features.
What Problem Does This Paper Address?
Understanding how language models learn and represent information is a major challenge in AI interpretability. While we can analyze model states at specific points in training (snapshots), the dynamic process of how features emerge, evolve, and contribute to the model's overall function over time remains largely unexplored. This "life history" of features is crucial because it might explain why certain features are more important than others and how the model's complex structure develops. The authors argue that a static snapshot misses critical information about feature importance and development, leading to an incomplete understanding of model mechanics.
Methods
The core of the methodology involves training Sparse Autoencoders (SAEs) at multiple checkpoints throughout the training of Pythia language models. SAEs are used to decompose the model's internal activations into a set of interpretable features. The authors then track these features across checkpoints, analyzing their "firing identity" (what they respond to) and "decoder direction" (how they influence predictions). Key techniques include:
- Feature Tracking: Matching features across checkpoints using Hungarian assignment to build "life histories."
- Cross-Layer DAG Construction: Building a directed acyclic graph (DAG) to represent hierarchical relationships between features across different layers.
- Joint Cross-Layer Ablation: A novel technique where multiple features across different layers are simultaneously ablated to measure their collective impact on model performance (measured by cross-entropy cost). This is contrasted with standard per-firing single-feature ablation.
- Predictive Modeling: Training a logistic regression model on early-stage feature properties (firing patterns) to predict which features will become part of the stable "carrier scaffold" later in training.
- Transplantation Control: Replacing decoder columns with earlier versions to isolate the contribution of decoder direction calibration.
Datasets
- Pythia Models: The study uses Pythia-160M and Pythia-410M models.
- The Pile: A large, diverse dataset used for training the SAEs and for evaluating model performance. A held-out slice of The Pile was used for SAE training and ablation experiments, and a canonical validation sample (S*) was used for feature-level statistics.
Key Insights and Ideas
- The Carrier Scaffold: A small, stable population of features emerges early in training and forms a "carrier scaffold" around which the rest of the model's representational structure organizes.
- Life History Matters: A feature's importance and role are not solely determined by its final state but by its entire developmental trajectory.
- Two-Phase Training: Training appears to have two distinct phases: an early "selection" phase (first ~1% of training) that establishes the scaffold, and a later "calibration" phase (remaining 99%) where decoder directions are refined around this fixed substrate.
- Function Precedes Direction: A feature's "function" (what it responds to) is largely determined early on, and this early function predicts its eventual role in the scaffold, even before its "direction" (how it influences outputs) has stabilized.
- Load-Bearing Scaffold: The carrier scaffold features are disproportionately load-bearing, meaning their removal has a much larger impact on model performance than removing an equivalent number of non-scaffold features. This load is invisible to standard per-feature analysis.
- Hierarchical Seeding: The carrier scaffold actively recruits and organizes a large majority of other features into its hierarchy as training progresses.
Key Results
- Early Assembly: Feature emergence and death rates collapse by ~40x in the first ~1% of training, with the scaffold largely fixed by then.
- Load-Bearing Impact: Jointly ablating persistent carriers incurs ~91x greater cross-entropy cost compared to ablating matched non-scaffold features.
- Predictive Power: A predictor trained on step-1k firing patterns can identify future scaffold members with 9.3x enrichment over the base rate.
- Organizational Reach: By the end of training, scaffold carriers have recruited 64% of active features (Pythia-160M) into their hierarchy.
- Temporal Split: Feature function stabilizes early (step 1k), while decoder direction calibration continues throughout training, creating a significant temporal gap.
Limitations and Future Work
- Scale and Architecture: The study is conducted on two specific Pythia models and a limited number of layers. Generalizing these findings to other architectures, training corpora, or model scales remains an open question.
- Correlational Evidence: While ablation and prediction strongly suggest the scaffold's importance, the evidence is correlational and doesn't establish a definitive causal mechanism for its formation.
- Layer Coverage: SAEs were trained on a subset of layers, and denser sampling might reveal more about scaffold structure.
- Future Directions: The authors suggest investigating whether fine-tuning preserves scaffold function while rotating decoder directions, mapping published circuits onto the scaffold population, and generalizing scaffold size and load concentration to larger models. They also propose that training-time triage of feature populations is viable.
Summary updated