Traditional neural scaling laws treat the training process as a black box. They are static snapshots designed to predict a model's endpoint—the final loss achieved after a massive compute budget is fully spent. As noted by Binghui Li et al. in Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules, this exclusive focus on endpoints fails to capture the continuous evolution of performance. If a multi-million dollar training run encounters an anomaly or hyperparameter mismatch halfway through, endpoint scaling laws offer no guidance on whether the model will eventually recover or fail.
To open this black box, a new paradigm has emerged: trajectory-based scaling laws. Instead of predicting a single final value, this framework models the entire continuous path of training loss over time. The core insight is that neural networks do not wander arbitrarily through loss space; rather, when properly normalized, models of vastly different scales follow a single, universal optimization trajectory.
The Data Efficiency of Intermediate Checkpoints
The first major advantage of modeling full trajectories is practical: it radically reduces the compute required to build scaling laws.
To fit a traditional scaling law, researchers must train multiple independent, smaller proxy models completely to fruition. This requires significant computational overhead. Trajectory-based scaling avoids this by extracting dense information from intermediate checkpoints within just a few short, ongoing training runs.
Endpoint Scaling: [Run 1] ➔ ⬤ (Final Loss)
[Run 2] ➔ ⬤ (Final Loss) ==> Fit curve using only final points
[Run 3] ➔ ⬤ (Final Loss)
Trajectory Scaling: [Run 1] ➔ ⬤--⬤--⬤--⬤--⬤--⬤ (Dense Checkpoints) ==> Fit curve using full path
Leshem Choshen et al. demonstrate in A Hitchhiker's Guide to Scaling Law Estimation that incorporating these intermediate checkpoints provides a dense set of observations that dramatically improves the precision and stability of scaling law fits. Similarly, Margaret Li et al. in (Mis)Fitting Scaling Laws: A Survey of Scaling Law Fitting Techniques in Deep Learning show that leveraging intermediate metrics allows for reliable extrapolation even when working with highly constrained compute budgets.
By treating a training run as a continuous sequence of informative states rather than a single binary outcome, researchers can accurately forecast the step-by-step performance of massive models using a tiny fraction of the compute traditionally required (Howe Tissue et al. in Scaling Law with Learning Rate Annealing).
The Phenomenon of Loss Curve Collapse
How can the early steps of a small model predict the precise trajectory of a model 100 times its size? The answer lies in a geometric phenomenon known as loss curve collapse.
When raw training loss is plotted against steps or tokens for different model sizes, the result is a family of divergent curves. However, if these curves are rescaled using scale-invariant controls, they align onto a single universal master curve.
Shikai Qiu demonstrates this in Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks by introducing affine trajectory normalization. By mapping training progress to a unit interval and rescaling the loss relative to its irreducible minimum and final value, the trajectories of completely different model sizes and compute budgets collapse onto a single line.
This underlying order is not unique to modern Transformers. Its theoretical roots extend back to non-parametric models; for instance, Dörthe Malzahn et al. in A Variational Approach to Learning Curves used statistical physics techniques to derive universal relations where learning curves from different kernels and dimensions collapse onto a single theoretical line.
In practical large language model (LLM) training, this collapse provides a critical diagnostic benchmark. Shane Bergsma et al. show in Scaling with Collapse: Efficient and Predictable Training of LLM Families that when compute-efficient hyperparameters are matched, normalized loss curves collapse perfectly. This alignment serves as a reference signature: if a massive, expensive training run deviates from this collapsed path, engineers can immediately identify hardware or data issues.
This shared trajectory exists because models follow a highly structured path of complexity growth during optimization. As Francesco D'Amico et al. explain in Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks, models trained on different dataset sizes follow a universal performance trajectory when measured against their "spectral complexity" norm, only diverging when they hit dataset-specific saturation points.
Deconstructing the Schedule: Work Done and Annealing
To accurately predict a continuous trajectory, a scaling law must account for the learning rate (LR) schedule. Standard power laws often fail to capture the rapid loss reduction observed during the final "cool-down" or annealing phase of a cosine schedule (Yiding Jiang et al. in Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws).
To resolve this, schedule-aware scaling laws decompose the training trajectory into two distinct forces. As formulated by Kairong Luo et al. in A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules, the loss curve can be modeled as a combination of:
- Cumulative Learning Rate (Base Progress): The total "work" performed on the model weights is determined by the integral of the learning rate over time. Different training schedules with the same cumulative learning rate budget achieve nearly identical baseline loss levels.
- The Decay Bonus: Reducing the learning rate triggers an immediate, predictable drop in loss. This drop follows a power law relative to the magnitude of the decay and the remaining training horizon.
By modeling these components, the Multi-Power Law framework allows researchers to predict loss curves across arbitrary, non-monotonic, or completely unseen learning rate schedules using parameters fitted from just a few short runs.
Loss
▲
│ \ ◀─── Base Progress (Driven by Cumulative LR)
│ \
│ └───┐ ◀─── Annealing Bonus (Predictable drop from LR reduction)
│ └───►
└──────────────────► Time / Tokens
This continuous-time perspective allows training to be modeled as a predictable dynamical system. For example, Jinbo Wang et al. in Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws model the loss trajectory as a continuous-time stochastic differential equation. This allows them to separate signal extraction from noise accumulation and dynamically optimize batch size schedules on the fly.
Even the steady-state power-law decay seen during training can be traced to fundamental architectural properties. Yizhou Liu et al. in Universal One-third Time Scaling in Learning Peaked Distributions demonstrate that when loss is mapped against a dynamic time variable incorporating the LR schedule, curves collapse to a universal $1/3$ scaling exponent. This indicates that the rate of optimization is an intrinsic property of the softmax and cross-entropy loss architecture when learning low-entropy distributions.
From Loss Trajectories to Downstream Capabilities
Ultimately, predicting pre-training loss trajectories is valuable because pre-training loss serves as a universal predictor for downstream task performance.
Yangyi Chen et al. leverage this in Scaling Laws for Predicting Downstream Performance in LLMs via a two-stage framework: $\text{FLOPs} \rightarrow \text{Loss} \rightarrow \text{Performance}$. By collecting data from intermediate checkpoints of a single run, they map how downstream task performance improves relative to pre-training loss, allowing them to forecast the capabilities of large models well before their training concludes.
This predictability works because downstream performance maps cleanly to pre-training loss, regardless of whether that loss was achieved by a small model trained on more data or a large model trained on less. As shown by Zhengxiao Du et al. in Understanding Emergent Abilities of Language Models from the Loss Perspective, complex, seemingly "emergent" abilities manifest predictably once the continuous pre-training loss trajectory crosses specific, quantifiable tipping points.
By shifting from static endpoint predictions to continuous trajectory modeling, researchers gain a granular, data-efficient, and mathematically rigorous framework to monitor, optimize, and forecast the behavior of foundation models from the very first steps of training.
Key Foundational Work
Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks (2024) — contributed 6 concept elements to this direction.
Understanding Emergent Abilities of Language Models from the Loss Perspective (2024) — contributed 1 concept elements to this direction.
A Variational Approach to Learning Curves (2001) — contributed 1 concept elements to this direction.