TL;DR
This position paper argues that AI research must move away from treating models as static artifacts and relying on brittle post-training fixes ("fixing it in post"). Instead, building a true "Science of AI" requires studying training dynamics as time-evolving processes to enable predicting, intervening in, and designing reliable model behaviors from the start.
What Problem Does This Paper Address?
Modern AI engineering heavily relies on a "fix it in post" mentality. When a large language or vision model exhibits undesirable behaviors—such as racial bias, safety violations, or hallucinations—the standard industry response is to apply post-training interventions like Reinforcement Learning from Human Feedback (RLHF), prompt filtering, or fine-tuning.
The authors argue that this approach is fundamentally limited and brittle. Post-hoc safety patches are easily bypassed by simple jailbreaks, and superficial fixes (like secretly appending diversity keywords to prompts) leave underlying representational biases intact. The core problem is that the AI field lacks a predictive, causal scientific framework. By treating models as fixed, static endpoints, researchers miss the opportunity to understand why specific behaviors emerge during the training process itself.
Methods
As a position paper, this work does not introduce a new algorithm or empirical dataset. Instead, it proposes a conceptual framework grounded in the philosophy of science (specifically drawing from Thomas Kuhn’s characteristics of good scientific practice) to reorient AI research.
The authors outline a three-tiered capability ladder to operationalize and measure scientific progress in AI:
- Predicting Training: Forecasting a model's final properties (e.g., capabilities, biases, safety risks) from its initial training setup before the run finishes or begins.
- Intervening on Training: Recognizing when a training trajectory is heading toward an undesirable state and dynamically redirecting it mid-run.
- Designing Training: Constructing training procedures from scratch that reliably guarantee desired properties by understanding which behaviors are stable.
The paper evaluates the current state of the art across four case studies—mechanistic interpretability, fairness, memorization, and simplicity bias—to demonstrate how shifting focus to training dynamics can yield these capabilities.
Datasets
Not discussed in the paper. As a position paper reviewing existing literature, it does not introduce new empirical datasets or benchmarks, though it references existing open-source model suites and benchmarks (such as Pythia, OpenFold, and Global MMLU) to illustrate its arguments.
Key Insights and Ideas
- Models as Dynamical Systems: AI models are not static objects; they are snapshots of time-evolving processes shaped by data mixtures, objectives, and optimization paths. The trajectory contains vital information that endpoint comparisons completely miss.
- The Illusion of Interpretability: Many mechanistic interpretability studies look at a single model checkpoint and declare that a specific neuron performs a specific function. However, because individual neuron participation fluctuates across different random seeds, these are often implementation details rather than stable, invariant mechanisms.
- Non-Monotonic Memorization: Memorization is not a simple linear function of data repetition and recency. Models are actually most "absorbent" and prone to memorization around 10–20% into their training run, reflecting a complex interplay between remaining capacity and developing linguistic competence.
- Sequential Complexity Learning: Research on simplicity bias shows that networks trained with Stochastic Gradient Descent (SGD) naturally learn statistical features in order of increasing complexity (e.g., learning unigram probabilities before bigrams and trigrams).
Key Results
While the paper focuses on synthesizing existing literature rather than presenting new quantitative experiments, it highlights several key paradigm-shifting results from recent training dynamics research:
- Predictive Scaling: While scaling laws routinely predict final loss based on compute budgets, recent work has successfully extended this to predict specific evaluation benchmarks, memorization rates, and social biases using smaller or partially trained models.
- Phase Transitions: Studies show that critical internal structures, like induction heads (responsible for in-context learning), form abruptly during sharp phase transitions early in training, rather than developing smoothly.
- Brittle Fine-Tuning: The authors highlight documented consensus that safety fine-tuning is highly fragile and can be inadvertently undone by fine-tuning the model further on completely benign, unrelated datasets.
Limitations and Future Work
The authors acknowledge that due to the highly complex and stochastic nature of neural network training, any emergent scientific theories of AI may initially be approximate, heuristic, and domain-specific.
To move the field forward, they explicitly define five open problems for the community:
- Differentiating Computation vs. Data Structure: Developing methods to determine which features in hidden representations reflect actual model processing versus surface-level patterns in the input data.
- Designing Social Bias: Creating quantitative training methods that allow developers to precisely tune data mixtures to achieve specific, predetermined demographic output rates.
- Measuring Cultural and Linguistic Erasure: Building systematic infrastructure to predict before deployment which cultures or dialects will suffer representation harms or distortions.
- Predicting Sequence-Specific Memorization: Developing theories to forecast exactly which data sequences a large model will memorize verbatim given a specific dataset order.
- Unifying Simplicity Biases: Reconciling spectral bias (Fourier decomposition) with distributional simplicity bias (statistical moments) to explain sequential learning phenomena observed in models like OpenFold.
Summary updated