An Analytical Model of Compute-limited Multistage Training Pipelines

Nishil Patel, Jin Hwa Lee, Basile Confavreux, Andrew SaxearXiv 2026 · arXiv

TL;DR

This paper introduces the first analytically tractable model for the entire foundation model training pipeline—spanning unsupervised pretraining, supervised fine-tuning (SFT), and reinforcement learning (RL). By deriving low-dimensional "learning dynamics" using statistical physics, the authors provide a theoretical framework to understand how pretraining quality sets a performance ceiling and how to optimally distribute a limited compute budget across training stages.

What Problem Does This Paper Address?

Modern AI models are built in stages: first, they learn general representations from massive unlabeled data (Pretraining); then, they are refined on specific tasks with labels (SFT); and finally, they are optimized for complex reasoning or preferences via rewards (RL).

Despite the success of this pipeline, we lack a mechanistic understanding of how these stages interact. Researchers often rely on expensive empirical "scaling laws" to guess how much compute to spend on each stage. This paper seeks to replace trial-and-error with an analytical model that explains why certain stages complement each other—for instance, why SFT is often necessary to "jump-start" RL.

Methods

The authors model the pipeline using a high-dimensional teacher-student framework:

  • Pretraining: Modeled as a Linear Autoencoder (LAE) that learns to project data into a "signal" subspace. The quality of this representation is governed by the ratio of pretraining samples to data dimensions ($\alpha = N/D$).
  • SFT: Modeled as a perceptron learning from noisy, stepwise labels (dense feedback).
  • RL: Modeled as learning from sparse episodic rewards (bulk feedback), where a reward is only granted if a sequence of decisions is correct.
  • The "Math Magic": Using Random Matrix Theory and an operator-projection closure, the authors reduce the high-dimensional stochastic gradient updates into a small system of Ordinary Differential Equations (ODEs). These ODEs track "order parameters" (like the overlap between the student and the ideal teacher) to predict performance without running full-scale simulations.
Figure 1: (a) A Schematic of multistage training pipeline. Each solid trapezoid indicates a single linear neural network layer. Pretraining is a Linear Autoencoder (LAE) that approximately learns the principal subspace of data x ∼N(0, Σ), where Σ is equal to the identity apart from a noise subspace, with identity covariance scaled by ∆< 1. CPT is the compute allocated to pretraining. In the subsequent finetuning (SFT and RL), a teacher network w∗gives labels y∗from the signal subspace and it is used to train a student network w learning in the subspace of the LAE encoder projection with feedback. The student predicts label y using the effective features ˜x. Importantly, SFT is modelled by every observation at each t updating the student, while RL is modelled by bulk feedback through reward at the end of a ‘correct’ sequence through the entire t = 1, . . . , T episode. We denote the compute allocations to SFT and RL, CSFT and CRL respectively. (b) Optimal generalisation performance against total Compute flops under three protocols: SFT only, RL only, and SFT with RL. Compute is allocated as described in 5.1, parameters are set as in 5d.

Datasets

  • Synthetic Data: High-dimensional Gaussian data with a defined signal-to-noise structure used to validate the mathematical theory.
  • OLMo-2 1B: The authors validated their "RL plateau" findings by fine-tuning the 1-billion parameter OLMo-2 model on the GSM8K (Grade School Math) benchmark.

Key Insights and Ideas

  • The Pretraining Bottleneck: Finite pretraining data imposes a fundamental limit on downstream accuracy. The authors show that the Bayes classification error scales as $\alpha^{-1/2}$. No amount of SFT or RL can fully overcome a poor initial representation.
  • SFT as an RL "Catalyst": RL often suffers from a long "plateau" where the model learns nothing because rewards are too sparse. SFT provides dense feedback that moves the model's weights into a regime where it can start hitting the "correct" sequence of actions, effectively shortening the RL plateau.
  • Non-monotonic Compute Allocation: Interestingly, the optimal share of compute for RL does not always increase with the total budget. Depending on the quality of SFT labels (noise), the optimal strategy might involve spending more on SFT to provide a better starting point for a shorter, more efficient RL phase.
  • Skill Amplification: In a compositional setting, RL doesn't just improve everything; it selectively amplifies task-relevant skills learned during pretraining while largely preserving "junk" skills that might be useful for future, different tasks.

Key Results

  • Theory-Simulation Match: The derived ODEs show an excellent match with actual high-dimensional simulations, proving that the low-dimensional dynamics capture the essence of the training process (see Figure 2 in the paper).
  • Empirical Validation: The "SFT warm start" effect predicted by the model was observed in real LLMs. Models initialized with SFT checkpoints showed much faster reward improvement on GSM8K compared to RL-only baselines.
  • Compute Frontiers: The model allows for the calculation of "optimal performance" curves. For small budgets, SFT-only is often best. As budgets grow, a three-stage pipeline (Pretrain + SFT + RL) becomes the most compute-efficient way to reach high performance.
Figure 2: ODEs accurately capture distinct learning dynamics of SFT and RL. (a) Good agreement of normalised traces of powers of projection-signal overlap matrices for: RMT (solid), PCA (open square), and LAE (filled diamond). Good agreement of ODE theory (dashed) in time (ξ) and simulated evolution (solid) of order parameters for: (b) the SFT model, (c) the RL model. (d) The evolution of probability of correct classification (P) under optimal compute budget allocation for a range of total compute budgets in FLOPs. Parameters: ∆= 0.1, ρ = 0.2. (a) D = 2000. (b) ϵ = 0.05, spherically unconstrained weights, η = 1. (c) spherically constrained weights, locally optimal learning rate. (b), (c) and (d) T = 6, D = 1000. (d) ϵ = 0.2.

Limitations and Future Work

  • Linearity: The model uses linear layers and autoencoders. While this makes the math tractable, real foundation models are deep, non-linear Transformers.
  • Frozen Features: The model assumes that the representation learned during pretraining is fixed (frozen) during fine-tuning. While this aligns with Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA, it doesn't account for "representation drift" in full fine-tuning.
  • Independence Assumption: The RL model assumes independence between observations in an episode, which ignores the temporal dependencies found in real-world sequential decision-making.

The authors suggest that future work should extend this framework to more complex, non-linear architectures and explore the costs of acquiring high-quality human supervision versus synthetic RL feedback.

Summary updated

Abstract

Modern foundation models are trained through multiple stages, yet how these stages interact and how compute should be allocated across them remain poorly understood. We introduce an analytically tractable high-dimensional model combining unsupervised pretraining with a linear autoencoder, supervised fine-tuning (SFT) with noisy stepwise labels, and reinforcement learning (RL) with sparse episodic rewards. Using random matrix theory and an operator-projection closure, we derive approximate low-dimensional learning dynamics that closely match simulations. We show that finite pretraining data imposes a representation-dependent performance limit, with Bayes classification error scaling as the inverse square root of the pretraining sample ratio, and observe the same exponent in asymptotic SFT and RL errors. Better pretraining also reduces sensitivity to noisy supervision. SFT initialization shortens the initial RL plateau induced by sparse rewards, an effect we also observe in OLMo-2 1B on GSM8K. Longer SFT warm starts additionally improve asymptotic RL performance. Optimizing compute across all three stages reveals that the optimal allocation depends on supervision quality and total budget, with a non-monotonic dependence of the RL compute share on budget. Finally, a complementary compositional model shows how RL selectively strengthens task-relevant pretrained skills while largely preserving unused skills. Together, these results identify mechanisms linking representation quality, feedback structure, and compute allocation in multistage training.
BibTeX
@misc{patel2026analytical,
  author = {Nishil Patel and Jin Hwa Lee and Basile Confavreux and Andrew Saxe},
  title = {{An Analytical Model of Compute-limited Multistage Training Pipelines}},
  year = {2026},
  eprint = {2611.13276},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13276},
}
Figure