TL;DR
This position paper argues that AI research must move away from treating models as static artifacts and relying on brittle post-training patches ("fixing it in post"). Instead, a true science of AI must study training dynamics—treating models as time-evolving processes—to enable researchers to predict, intervene in, and intentionally design model behaviors from the ground up.
What Problem Does This Paper Address?
When modern AI models exhibit undesirable behaviors—such as bias, safety risks, or hallucinations—the dominant industry response is to "fix it in post." This means applying post-training interventions like Reinforcement Learning from Human Feedback (RLHF), fine-tuning, or input/output filtering.
The authors argue that this mentality is fundamentally flawed and short-sighted. Post-hoc safety fixes are notoriously brittle and easily bypassed, and patching symptoms leaves underlying structural flaws intact. The root problem is that the AI field lacks a predictive, causal scientific theory. Current research primarily describes what a finished model does rather than explaining why certain behaviors emerge during the training process. Without understanding models as dynamic systems shaped by data, objectives, and optimization over time, we cannot reliably prevent failures or safely scale AI.
Methods
As a position paper, this work does not introduce a new algorithm or empirical experiment. Instead, it proposes a conceptual framework for a "Science of AI" grounded in Thomas Kuhn’s philosophy of science, evaluating theories based on empirical accuracy, consistency, scope, simplicity, and fruitfulness.
To operationalize progress, the authors outline a roadmap based on three progressively harder capabilities:
- Predicting training: Forecasting a model's final properties from its initial training setup before the run finishes.
- Intervening on training: Recognizing when a training trajectory is heading somewhere undesirable and redirecting it mid-run.
- Designing training: Setting desired properties in advance and constructing a training procedure that reliably guarantees them.
The paper evaluates the current state of the art by reviewing literature across four case studies: mechanistic interpretability, fairness/bias, memorization, and simplicity bias.
Datasets
Not discussed in the paper. Because this is a position paper reviewing the broader field, it does not introduce or train on specific datasets. It conceptually references existing model suites and benchmarks (like Pythia, Hubble, and Global MMLU) to illustrate its points.
Key Insights and Ideas
- The "Band-Aid" Fallacy: Relying on post-training patches (such as secretly appending diversity keywords to user prompts to mask underlying demographic bias) produces superficially diverse outputs while leaving the core bias intact. The authors compare this to pre-germ-theory medicine, which treated symptoms without understanding the underlying cause of disease.
- Models as Time-Evolving Processes: Smooth, aggregate loss curves at the end of training often mask discrete, sudden phase transitions (like the abrupt emergence of induction heads or syntactic structures) that occur early in training.
- Conflating Signal with Noise: AI research frequently ignores dimensions of variation, such as random seeds or data distribution. For example, treating English-centric models as representative of "language models generally" ignores how different languages interact with model architectures and tokenizers.
- Non-Monotonic Memorization: Memorization is not a simple linear function of data repetition. Models are actually most "absorbent" (prone to memorizing) around 10–20% into training, reflecting a complex interplay between remaining capacity, data distribution, and emerging linguistic competence.
Key Results
Because this is a synthesis of existing literature, it highlights qualitative conclusions and open challenges rather than new quantitative benchmarks:
- Brittleness of Safety Fine-Tuning: The paper highlights how easily safety fine-tuning can be undone by fine-tuning on benign data, asking questions in the past tense, or using low-resource languages.
- Intermediate Checkpoints as Predictors: Studies show that intermediate training checkpoints predict final model memorization better than smaller, fully-trained models do, proving that the training trajectory contains vital hidden information.
- Identification of Open Problems: The authors formalize several concrete open problems for the community, including:
- Differentiating Computation from Data Structure: Distinguishing between actual model processing and surface-level patterns in the input data.
- Designing Social Bias: Creating training methods that reliably hit target demographic frequencies (since simply matching the training data ratio to the target ratio is empirically insufficient).
- Predicting Memorization: Forecasting exactly which sequences a large model will memorize given a dataset in a fixed order.
Limitations and Future Work
The authors openly address several counter-perspectives prevalent in the AI community:
- Engineering Sufficiency: The view that brute-force engineering and post-hoc fixes are "good enough" because they work empirically. The authors counter that this only optimizes locally and offers no guidance for scaling safely.
- Anti-Theory Skepticism: The belief that deep learning is too complex or stochastic for meaningful theory. The authors point to thermodynamics and quantum mechanics as proof that non-deterministic processes can be fruitfully analyzed using statistical tools.
- Safety Pragmatism: The argument that immediate harm reduction requires post-training patches. The authors agree short-term mitigations are necessary but argue they must be supplemented with design-level solutions.
Future Work: The ultimate goal is to extend the predictive success of scaling laws—which currently forecast loss quite well—to more complex properties like capabilities, biases, robustness, and safety-relevant behaviors.
Summary updated