When a model stops improving, researchers with small compute budgets usually cannot tell why. The model may be too small to represent the target function, which is a capacity bottleneck. Or its learning rate and other training settings may simply be wrong for that size, which is a tuning failure. The standard way to tell these apart is to rerun training many times with different settings. That is exactly what limited budgets rule out. As a result, papers routinely report architecture comparisons in which the two explanations are mixed together. The authors of Jargon say openly that they cannot separate architectural effects such as linear attention from suboptimal configurations. The research file also lists this separation as an open problem in the capacity-bottleneck literature.
We propose a diagnostic that splits a model's loss gap into a part that retuning could recover and a part that only more capacity could recover. It works from one training run plus inexpensive curvature probes. It builds on a finding from Understanding the Mechanisms of Fast Hyperparameter Transfer. That paper shows that a transformer's training trajectory separates into two components. One is a small set of directions, about 40 in Llama-style models, that stays the same across widths and determines the best hyperparameters. The other is a large residual component whose loss falls as width grows but which is insensitive to hyperparameters. That paper used the split to explain why hyperparameters tuned on small models transfer to larger ones. We use it as a per-run measurement. The part of a loss gap that sits in the hyperparameter-sensitive directions indicates tuning failure. The part that sits in the residual indicates a capacity bottleneck.
Core Claim
Consider a training run with loss $L$. Following the fast-transfer paper, split it into $L = L_{\text{top}} + L_{\text{res}}$. Here $L_{\text{top}}$ is the loss reduction carried by the top-$k$ directions of the exponentially smoothed update trajectory, and $L_{\text{res}}$ is everything else. Now compare two runs, for example a candidate and a reference, or the same model at two sizes. Their gap is $\Delta = \Delta_{\text{top}} + \Delta_{\text{res}}$. We define the tuning share as
$$\tau = \frac{\Delta_{\text{top}}}{\Delta}.$$
We claim three things. First, $\tau$ predicts what fraction of $\Delta$ a full hyperparameter sweep would close. Second, $1-\tau$ predicts what fraction would close only through added capacity. Third, whether this prediction can be trusted is itself measurable. It holds only when the top-$k$ directions carry a concentrated share of the optimization, and it should fail visibly where that structure is known to break.
The expectation comes directly from the source paper's two properties: top-$k$ loss moves with hyperparameters and not with width, while residual loss moves with width and not with hyperparameters. Nobody has tested whether these properties are clean enough to support attributing a gap in a single run. Nobody has tested them outside plain width scaling with Adam either.
Research Program
The first step is to build a testbed with known answers. We train Llama-style transformers on WikiText-103 and two-layer MLPs on CIFAR-10, the two settings where the decomposition has already been validated. At these sizes, full sweeps are affordable, so ground truth can be established. We then create plateaus of three kinds.
- Tuning plateaus. A model of adequate width is trained with learning rate or weight decay moved off-optimum by a controlled factor.
- Capacity plateaus. Width or depth is shrunk while settings are tuned correctly for that size under maximal update parameterization. Maximal update parameterization is the rescaling of initialization and per-layer learning rates that keeps training dynamics stable as width grows.
- Mixed plateaus. Both faults are present at once.
For each case, the ground-truth recoverable fractions come from two measurements: a full sweep at the fixed size, and a correctly tuned larger model.
The second step is to make the measurement cheap. The fast-transfer paper selects the cutoff $k$ with a hand-built proxy objective. We replace this with a rule based on how the spectrum decays. We then measure how the diagnostic's accuracy degrades in two ways: as the probes are computed at fewer checkpoints, and as they are computed from shorter prefixes of the run. The target is a total overhead below one extra training run.
The third step moves into regimes where the theory is not established and where the diagnostic is most needed:
- depth scaling;
- depth grown during training in the style of Curriculum-Guided Layer Scaling, which relied on manually tuned learning-rate schedules at each expansion;
- sparse mixture-of-experts layers, in which each token is routed to a few of many expert sub-networks, with the sparsity ratio varied rather than held fixed as in Hyperparameter Transfer with Mixture-of-Expert Layers;
- the Muon optimizer, which the fast-transfer paper shows destroys the low-rank update structure.
Muon therefore serves as a planned negative control. If the diagnostic is sound, its concentration score should drop under Muon, and its predictions should be flagged as unreliable rather than being wrong without warning.
Baselines
Naive reference. The do-nothing reference is the practice these papers describe: train at default or lightly tuned settings and read the loss gap as purely architectural. This amounts to $\tau = 0$ by assumption. The diagnostic beats it whenever tuning plateaus exist in the testbed, and they exist by construction. The interesting question is how large the error is on realistic cases.
Early-trajectory extrapolation. The strongest cheap competitor is Training Performance Estimation from Neural scaling of deep chemical models. It fits early loss curves to predict converged loss. The paper reports a Spearman rank correlation of 0.92 to 1.0 between predicted and true converged rankings, at over 80% compute savings. We give it the same prefix length and checkpoint budget as our diagnostic. It predicts converged loss under each hyperparameter setting it is shown, so it is allowed a small number of truncated runs matched to our compute. Because it ranks configurations, and ranking is not attribution, we compare the two methods on the same target: predicted fraction of the gap recovered by retuning. Beating it means lower error on that fraction at equal compute.
Transfer-then-compare. This baseline tunes a small proxy under maximal update parameterization and transfers the result to the target model. Whatever gap remains is attributed to capacity. It is strong where transfer is proven, namely width scaling with Adam. It serves as the reference there, and we expect the two methods to tie. It is not validated for depth changes, sparsity changes or Muon, and those regimes are where we expect our method to win.
Gold standard. A full learning-rate and weight-decay sweep at target size, together with a tuned larger model, defines the ground truth. Its cost is the denominator in our compute accounting.
Ablations. Three ablations isolate the key ingredient. The first replaces the top-$k$ subspace with a random $k$-dimensional subspace. The second uses the top-$k$ subspace of a single gradient rather than of the smoothed trajectory. The third sweeps $k$. If the random subspace performs as well, the claimed mechanism is false.
Why not just…?
Why not just run a short learning-rate sweep at the target size? A three-to-five-point sweep costs three to five full runs. It also only probes the one axis swept. The TiRex authors could not afford to tune masking ratios or augmentation probabilities, so settings other than the learning rate are often the untuned ones. Still, where a sweep is affordable it is the correct answer, and we treat it as ground truth rather than as a competitor. The diagnostic earns its place only if it matches the sweep's attribution to within a small error at a fraction of its cost. The testbed measures exactly that.
Why not just train a wider model and see whether loss drops? A drop shows that capacity helps, but a mistuned wider model can plateau just as its smaller counterpart does. The Capacity-Aware Mixture Law results show that what a model absorbs changes non-linearly with size, so a width probe mixes the two effects again. The width probe is also more expensive than the target run. We include it in the mixed-plateau cases to show where it misattributes.
Why not just use hyperparameter transfer and blame whatever gap remains on capacity? This is a real threat in the width-plus-Adam regime, and we expect a tie there. The fast-transfer paper itself shows the method failing under Muon and leaves depth scaling open. The MoE transfer paper's guarantees require a fixed sparsity ratio. Those are the regimes in step three where the two methods are compared directly.
Proposed Experiments
The primary metric is the absolute error between the predicted tuning share $\tau$ and the fraction of the gap a full sweep actually recovers, averaged over the testbed's tuning, capacity and mixed plateaus. It is reported alongside compute as a multiple of one training run. The field has no absolute WikiText-103 perplexity to anchor this. The research file gives no reported result on Lacuna for WikiText-103 perplexity at these widths, so all references come from our own runs.
Two checks are decisive. First, on mixed plateaus, $\tau$ must track the injected learning-rate error monotonically while staying nearly flat as width changes. If it moves with width, the separation claim fails. Second, under Muon, the concentration score must fall and the attribution error must rise together. A diagnostic that stays confident while being wrong is worse than none.
As an application, we rerun growth schedules in the style of Curriculum-Guided Layer Scaling at small scale and ask how much of their early loss disadvantage is recoverable by retuning. That paper reports higher pretraining loss but better downstream reasoning. This is a reminder that our diagnostic attributes pretraining loss only and makes no claim about downstream accuracy. We will report downstream accuracy on ARC-Easy alongside, so that any divergence between loss and accuracy is visible rather than hidden.
References
- Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains
- Understanding the Mechanisms of Fast Hyperparameter Transfer
- Curriculum-Guided Layer Scaling for Language Model Pretraining
- Hyperparameter Transfer with Mixture-of-Expert Layers
- Neural scaling of deep chemical models
- TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
- Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization