Stochastic depth trains a deep network while randomly skipping whole layers on each step. A skipped layer passes its input straight through, so training is cheaper, and at test time the full network runs. The method is known to help small-data vision models. It is also known to hurt modern language-model pre-training unless the skip rate is carefully annealed. Nobody can say in advance how much layer dropping a given model can tolerate. We propose a single explanation: the tolerable drop rate is set by where the training run sits relative to its capacity bottleneck. The bottleneck is the regime in which the data holds more learnable structure than the model can absorb.
Stochastic depth is also a useful instrument for a second, older question: whether a model that plateaus on a huge dataset has run out of expressive capacity or out of effective optimization. A layer-dropped model ends training with exactly the same architecture as its dense twin. Only its training-time capacity and its per-layer update count differ. Comparing the two therefore separates "not enough parameters" from "not enough useful gradient steps," which the capacity-bottleneck literature lists as open. The practical output is a predictive law for the maximum sustainable drop rate, which Don't Drop Dropout names explicitly as missing.
Core Claim
For a network with $N$ non-embedding parameters trained on $D$ tokens, define its equivalent capacity $N_{\text{eq}}$. This is the parameter count of a dense model, fitted on the same dense scaling curve, that reaches the same validation loss on the same $D$ tokens. Write $\kappa = N_{\text{eq}}/N$ for the fraction of nominal capacity the layer-dropped model effectively uses. Write $\bar\rho$ for the time- and layer-averaged survival probability, meaning the fraction of layer computations actually executed.
Three hypotheses compete. The first is $\kappa = \bar\rho$: the model behaves like a small model and is paying for exactly what it skipped. The second is $\kappa = 1$: skipping is free. The third is that $\kappa$ depends on the tokens-per-parameter ratio $D/N$.
We predict the third, with a specific shape. Where $D/N$ is small (the model has capacity to spare), $\kappa$ should sit at or above 1, because layer dropping acts as a regularizer. Where $D/N$ is large (underfitting), $\kappa$ should approach $\bar\rho$ or fall below it. The maximum drop rate that keeps loss within a fixed tolerance, $p_{\max}(D/N)$, should decrease monotonically across that transition.
The claim should also hold per data domain. Following Capacity-Aware Mixture Law, which shows that larger models absorb dense "Knowledge" text much faster than small models while code and math behave differently, we expect knowledge-heavy domains to reveal the capacity cost of layer dropping first.
Why This Is Not Already Done
Several neighbouring papers shape this proposal in different ways.
Don't Drop Dropout supplies the method we reuse and motivates the gap. It finds that constant layer dropout degrades single-epoch LLM pre-training, whereas an increasing-with-depth, decreasing-over-time schedule (ILD+DTS) matches the dense model while saving up to 20% of non-embedding training FLOPs. Its schedule is tuned by sweep and never related to $D/N$, and it lists the missing $p_{\max}$ law as a limitation.
Dropout Reduces Underfitting supplies the tension we must resolve. It shows that element-wise dropout applied only early in training reduces underfitting in large-data vision by lowering gradient-direction variance. That is evidence that some dropping can help even in the underfitting regime. Our experiments test whether the same holds for whole-layer dropping in language models or is specific to element-wise noise early in training.
The original Deep Networks with Stochastic Depth shows gains on CIFAR in a regime we would call low $D/N$: error falls from 6.41% to 5.25% on CIFAR-10 with a 110-layer ResNet. Exploring the Limits of Weakly Supervised Pretraining shows that small fixed models underfit 3.5B images. Together these two sit at opposite ends of the curve we want to trace.
Other work touches the same region without asking our question. Curriculum-Guided Layer Scaling, Apollo and zero/one-layer expansion all vary depth during training, either by discrete stacking or by sampled shallow depths, but treat depth as an acceleration device and never measure effective capacity against data scale. CAMEL treats capacity as a fixed constant.
The idea most obviously sitting at this intersection is to condition drop rates on data difficulty. We deliberately do not propose it. A curriculum that orders data from easy to hard, paired with the existing time-annealed schedule, would plausibly deliver the same effect, since difficulty and training time would then be tied together. Our contribution is a measurement and a law, not another schedule.
Research Program
The core artifact is a grid of decoder-only language models. Parameters range from roughly 50M to 1B, and nominal depth is held fixed within each width family. Each is trained at several tokens-per-parameter ratios spanning about 2 to 200. The low-ratio end is reached by repeating a small corpus for multiple epochs, which puts the model deliberately in the regime where it has capacity to spare.
For every cell of the grid we train the dense model and layer-dropped variants at several average survival levels $\bar\rho$. Each variant uses the ILD+DTS shape from Don't Drop Dropout, with masks sampled per sequence as that paper recommends.
A dense scaling curve fitted at each token budget converts every layer-dropped run's loss into $\kappa$. We fit this per domain using CAMEL's five-domain split (English/Chinese web, Knowledge, Code, Math), so that $\kappa$ becomes a vector rather than a single number.
The second component is the optimization-versus-expressiveness test. Because the schedule anneals to zero drop, the final network is architecturally identical to the dense one. We continue both for an additional dense budget and ask whether the loss gap closes, which would mean the deficit came from optimization, or persists, which would suggest the dropped training shaped representations that full depth cannot later recover.
The third component is fitting and extrapolating $p_{\max}(D/N)$. We fit on the models up to about 300M parameters and predict the tolerated drop rate for the 1B models before training them.
One confound must be handled throughout. Learning-rate tuning interacts with depth: Hyperparameter Transfer Laws derives $\eta \propto L^{-3/2}$ for fixed depth $L$, and a fluctuating active depth may shift that optimum. We therefore tune learning rate independently for each drop level at small scale. As a side result, we record whether the optimum tracks nominal depth or expected active depth $\sum_l p_l$.
Baselines
The do-nothing reference is the dense model with the same parameters, data, token count and tuned learning rate, run with no layer dropping. Every layer-dropped run is compared to it at matched tokens, and separately at matched training FLOPs. Beating it at matched FLOPs is the only meaningful win for efficiency claims. Matching it at matched tokens with fewer FLOPs is the result Don't Drop Dropout already reports.
The compute-matched smaller dense model is a shallower dense network whose training FLOPs equal the layer-dropped run's. It is the natural competitor for the hypothesis $\kappa = \bar\rho$. It gets the same data and its own learning-rate sweep. If the layer-dropped model does no better than this baseline in the high $D/N$ regime, stochastic depth buys only elasticity at inference, not capacity.
The strongest existing method is ILD+DTS as specified in Don't Drop Dropout. The file reports its gain as "up to 20% of non-embedding training FLOPs" saved at matched validation loss and "up to 1.4x" inference speedup via self-speculative decoding. No reported validation-loss or downstream-accuracy numbers at our scales were found on Lacuna, so we reproduce it ourselves under identical tuning budgets. Our proposal does not claim to beat ILD+DTS as a schedule. It claims to predict, from $D/N$ alone, the drop level at which ILD+DTS stops matching dense. Success means the predicted $p_{\max}$ for held-out 1B runs lies within the confidence band of the swept value.
Two ablations isolate the key ingredients. Constant layer dropout at the same $\bar\rho$ isolates the role of annealing. Early element-wise dropout following Dropout Reduces Underfitting, applied for the same fraction of training, isolates whether any capacity-regime benefit comes from gradient-noise reduction rather than from skipping layers.
Why not just…?
Why not just treat a layer-dropped model as a dense model with fewer FLOPs and read its loss off a standard compute scaling law? This is the most serious threat, and it is exactly hypothesis $\kappa = \bar\rho$. If it holds everywhere, our law collapses into existing compute scaling, and that would itself be a decisive and useful negative result. The CIFAR gains in Deep Networks with Stochastic Depth already contradict it in the low-data regime. Whether it holds at high $D/N$ in language models is precisely what the compute-matched-dense baseline settles.
Why not just sweep drop rates at small scale and reuse the best value? Our claim is that the best value depends on $D/N$, so a sweep at one ratio should transfer badly to another. We test this directly by transferring the best small-scale drop rate at fixed width, but different token budgets, and measuring the loss penalty. If transfer works, the law adds little, and we will say so.
Why not just use a tokens-per-parameter rule of thumb, dropping layers only when data is scarce? That rule is the zeroth-order version of our hypothesis, and it may be most of the answer. The added value lies in two things the rule of thumb cannot give: the continuous shape of $p_{\max}$ across the transition, and the per-domain vector $\kappa$, which would tell practitioners that knowledge-heavy mixtures tolerate less dropping than code-heavy ones at the same $D/N$. If the per-domain differences are within noise, that refinement fails.
Why not just condition drop rates on data difficulty, as curriculum layer-scaling suggests? Curriculum-Guided Layer Scaling gets its gains from pairing depth growth with an easy-to-hard curriculum. A time-annealed schedule over curriculum-ordered data would reproduce most of that coupling without per-example routing. We therefore treat difficulty conditioning as out of scope until the capacity law shows where dropping is affordable at all.
Proposed Experiments
Training corpora are DCLM-BASELINE from DataComp-LM for the single-epoch, high-ratio regime and the CAMEL mid-training five-domain corpus for per-domain $\kappa$. For the low-capacity end, where very small models already saturate, we use TinyStories: models of about one layer and a few million parameters produce coherent text there, which makes it cheap to push $D/N$ to extremes. Metrics are per-domain validation loss plus HellaSwag, ARC, LAMBADA and PIQA, as in Don't Drop Dropout, and MMLU, GSM8K and HumanEval for the 1B models.
The first decisive check is the sign change. $\kappa$ measured at the lowest $D/N$ (multi-epoch) should exceed 1, and $\kappa$ at the highest $D/N$ should fall to or below $\bar\rho$. Absence of any crossover across a hundredfold range of ratios would refute the central claim.
The second is prospective prediction: the $p_{\max}$ law fitted below 300M parameters must predict the 1B tolerance before those runs exist.
The third is the gap-closing test after continued dense training. Closure attributes the high-$D/N$ deficit to lost updates rather than to capacity, which answers the optimization-versus-expressiveness question for this setting.
The fourth reconciles our results with Dropout Reduces Underfitting. If early element-wise dropout helps at high $D/N$ while layer dropping hurts, the two findings are about different mechanisms and do not conflict. If both help, our capacity story is incomplete.
References
- Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
- Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
- Dropout Reduces Underfitting
- Deep Networks with Stochastic Depth
- Exploring the Limits of Weakly Supervised Pretraining
- Curriculum-Guided Layer Scaling for Language Model Pretraining
- Preparing Lessons for Progressive Training on Language Models
- Scaling depth capacity via zero/one-layer model expansion
- Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks
- DataComp-LM: In search of the next generation of training sets for language models
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?