Do Layers Close Their Critical Periods When Their Weight Spectra Settle?

Neural networks have critical learning periods: short windows early in training when a bad input, such as blurred images, permanently damages the final model, while the same problem introduced later does no lasting harm. A separate body of work reads a network's state directly from its trained weights. It looks at how the singular values of each weight matrix are distributed. Training pushes these from a random-noise shape toward a heavy-tailed one, meaning a few large singular values sit far above the rest. This proposal asks whether these are the same event seen from two sides. The hypothesis is that each layer's critical period closes at the moment that layer's singular-value spectrum stops changing shape. If so, the window can be read layer by layer from weights alone, with no data and no curvature computation, and that timing can be used to schedule when spectrum-guided regularizers should act.

This matters because two practical communities currently guess at the same quantity. Critical-period work locates the window with the trace of the Fisher Information, a measure of how sensitive the loss is to each weight, and treats the whole network as having one window. Spectral optimizers such as heavy-tail-guided layerwise learning rates already stop measuring after "the first 20% of training" as a cost-saving rule of thumb. A per-layer, weight-only clock would explain that rule of thumb, replace it with something principled, and test whether attention and feed-forward layers have different critical periods.

Core Claim

The claim has three parts.

  1. Timing. For each layer $k$, the end of its critical period coincides with the time its tail exponent stabilizes. We call this the spectral settling time $T^{\mathrm{spec}}_k$. Here $\alpha$ is the power-law exponent of the eigenvalue distribution of $W^\top W$, estimated with the Hill estimator, and smaller $\alpha$ means a heavier tail.
  2. Asynchrony. These settling times differ across layers in a structured way. In transformers, attention layers should settle before feed-forward layers, because attention develops heavy tails faster. A single global window therefore misdescribes the network.
  3. Usefulness. Restricting a spectral regularizer to each layer's own window should match or beat applying it for the whole run.

We define the settling time with a rolling-change rule:

$$T^{\mathrm{spec}}_k = \min\{t : |\alpha_k(t') - \alpha_k(t'-\Delta)| < \epsilon\ \text{for all}\ t' \in [t, t+H]\}$$

In this rule, $\alpha_k(t)$ is the Hill exponent of layer $k$ at step $t$, $\Delta$ is the comparison lag, $\epsilon$ is a tolerance, and $H$ is a hold length. All three are fixed once on a calibration architecture and then frozen.

The Hill estimator is meaningless while a spectrum still matches the Marchenko–Pastur shape, the bounded distribution that random matrices follow. We therefore pair $\alpha_k$ with a second, simpler signal: the number of eigenvalues above that random-matrix upper edge. Settling is declared only once both signals stop changing.

Where This Sits

Work that motivates the gap. Critical Learning Periods in Deep Networks established the phenomenon. On CIFAR-10, blur lasting beyond roughly epoch 40–60 raises final error from about 6% to over 18%, with sensitivity peaking near epoch 30. The window was traced with the global Fisher trace, not with the weights themselves. Time Matters in Regularizing Deep Networks showed that weight decay does its work early: removing it after epoch 75 of 200 keeps about 93% accuracy, while starting it at epoch 50 gives about 88%, close to the 87% with no decay at all. Early Period of Training Impacts Out-of-Distribution Generalization explicitly leaves automatic window detection as future work.

Work that supplies the method and makes the hypothesis plausible. From Spikes to Heavy Tails describes a phased progression: the spectrum starts random-like, a few eigenvalues "bleed out" of the random bulk, isolated spikes form above that bulk, the bulk decays, a heavy tail appears, and finally the matrix can collapse in rank. That paper finds this happening very early on clean data. It never introduces deficits and never asks whether the phases mark irreversibility. Crafting Heavy-Tails in Weight Matrix Spectrum without Gradient Noise shows that optimizer structure alone can produce these tails. This is a constraint on our work: we must check that settling is not just an Adam artifact.

The two spectral optimizers our application step reuses. One LR Doesn't Fit All assigns larger learning rates to layers with large $\alpha$ during the first 20% of training. AlphaDecay gives weaker weight decay to heavy-tailed modules throughout training. Neither paper ties its timing to critical periods.

The closest theory. Critical Learning Periods Emerge Even in Deep Linear Networks shows that depth-driven, winner-take-all dynamics among singular values permanently stunt the pathways that start out behind. This predicts a spectral scar from early deficits, but the analysis is for individual singular values in linear models, not for tail exponents in nonlinear networks.

Supporting evidence for layer-specific windows. On the special role of class-selective neurons in early training finds early-forming, layer-specific structure that matters only when suppressed early, which suggests that windows really are per-layer.

Research Program

The central object is a per-layer sensitivity curve, which serves as ground truth. For layer $k$ and onset time $t$, we freeze only that layer for a fixed window $[t, t+w]$, release it, train to completion, and record the final test-accuracy loss relative to an unperturbed run. The layer's empirical closure time $T^{\mathrm{sens}}_k$ is the earliest onset after which this loss stays within seed noise. We also run a second, softer deficit: injecting noise into the layer's updates during the window. This guards against freezing being a special case.

Global input deficits, specifically blur as in the original critical-period work, are also run. They serve to test the scar prediction: deficits applied before $T^{\mathrm{spec}}_k$ should leave layer $k$ with a persistently different tail exponent or spike count at convergence, and deficits applied after it should not.

Along every run we log several quantities per layer at matched intervals:

  • the Hill exponent $\alpha$;
  • the count of eigenvalues above the random-matrix edge;
  • stable rank, meaning squared Frobenius norm divided by squared spectral norm;
  • per-layer Fisher trace;
  • per-layer gradient norm;
  • relative weight change from the previous checkpoint.

The analysis question is which of these signals, turned into a settling time with an identical rule, best predicts $T^{\mathrm{sens}}_k$ across layers, seeds, and architectures. The prediction rules are calibrated on ResNet-18 and then evaluated without retuning on All-CNN, a small Vision Transformer, and a small LLaMA-style language model.

The application step then uses the winning clock to run windowed AlphaDecay. Each module receives AlphaDecay's spectrum-guided decay only until its own settling time, then plain weight decay. This is compared against full-duration AlphaDecay and against a single global 20% cutoff.

Baselines

Uniform reference. The naive point of comparison is a single global critical window, the assumption implicit in the original critical-period work. Operationally, every layer is assigned the network-wide Fisher-trace peak time as its closure. Matching or beating it requires the per-layer signal to explain variation that a single time cannot. If $T^{\mathrm{sens}}_k$ barely varies across layers, the asynchrony claim fails outright, and we will report that.

Strongest detector competitor. The strongest competitor for timing is per-layer Fisher-trace peak time. This is the measure used by Critical Learning Periods in Deep Networks and Early Period of Training Impacts Out-of-Distribution Generalization, localized to each layer. It is computed on the same checkpoints, with the same number of batches and the same settling rule, so the only difference is the signal. The spectral clock beats it if its rank correlation with $T^{\mathrm{sens}}_k$ is higher, or equal at lower cost and without data, with the gap exceeding the across-seed standard deviation on held-out architectures.

The file reports no prediction-accuracy numbers for Fisher-based window detection. No reported result was found on Lacuna for per-layer critical-period prediction, so this baseline will be measured fresh.

Strongest method competitor. For the application step, the competitor is full-duration AlphaDecay. On LLaMA-1B trained on C4 it reports validation perplexity of 14.55 versus 14.66 for uniform weight decay AlphaDecay. The second competitor is heavy-tail-guided layerwise learning rates, which report average zero-shot accuracy of 49.02% versus 47.09% on LLaMA-1B One LR Doesn't Fit All.

At our compute scale (60M–350M parameters), we re-run uniform decay, full AlphaDecay, global-20%-window AlphaDecay, and per-layer-windowed AlphaDecay. All four use the same tokens, the same base hyperparameters, and the same 500-step spectrum interval, and each gets an equal small tuning budget for the base decay. Beating the competitor means equal or lower perplexity than full AlphaDecay, with all spectral computation stopping early, and an improvement over the global-20% variant.

Ablations isolating the key ingredient. The first ablation replaces the spectral clock with a random permutation of the per-layer settling times. This keeps the distribution of window lengths but breaks the assignment of times to layers. The second ablation replaces the clock with a depth-ordered schedule, so that windows simply close in order of depth.

Why not just…?

Why not just use per-layer gradient norms or a cheap Fisher proxy, which cost almost nothing? This is a real threat. The diagonal empirical Fisher trace is close to a squared gradient norm, and the original Fisher-based signal might carry the same timing information. The spectral clock has two advantages. It needs no data, so it can be applied to released checkpoints. And under the scar hypothesis it can distinguish how a layer was damaged (failure to escape the random bulk versus collapse in rank), not just when. Neither advantage helps if gradient norms predict closure equally well. The head-to-head comparison in Experiment 1 decides this. If the gradient-based signal ties, we will report the spectral clock as an equivalent, data-free reading rather than a better detector.

Why not just track how far each layer's weights move from step to step? Relative weight change is also data-free and far cheaper than a singular value decomposition. It may simply reflect learning-rate decay, while the spectral shape can keep reorganizing at a constant weight norm. We test this directly by running constant-learning-rate controls. Critical Learning Periods in Deep Networks reports that critical periods persist under different schedules. If movement-based settling tracks $T^{\mathrm{sens}}_k$ under constant learning rate as well as spectral settling does, the spectral claim reduces to movement and we will say so.

Why not just assume lower layers close first? Depth ordering is a natural prior, and the low-level-first finding in the original critical-period paper supports it. It cannot, however, explain different closure times between attention and feed-forward blocks at the same depth. That divide is what the spectral optimizer papers observe. The depth-ordered ablation measures how much the spectral clock adds beyond depth. The Vision Transformer and language-model runs, where same-depth modules differ, are where we expect the gap.

Why not just keep the 20% cutoff that already works? If per-layer windows add nothing over a global 20% cutoff in the application step, the practical value lies only in the explanation, not in a better method. The comparison of the global-20% variant against the per-layer-windowed variant in Experiment 3 decides this.

Proposed Experiments

Experiment 1: does spectral settling predict closure? This is the decisive experiment. We run CIFAR-10 with ResNet-18 and All-CNN, and CIFAR-100 with a small Vision Transformer. Each condition uses five seeds, both the layer-freeze and update-noise deficits, and onsets every few epochs over a 200-epoch schedule. Success means the spectral settling time ranks per-layer closure with higher Spearman correlation than the global window, depth order, and relative weight change, and at least ties the per-layer Fisher peak, on architectures not used for calibration.

The key falsifier is constant-learning-rate training. If spectral settling and $T^{\mathrm{sens}}_k$ come apart under constant learning rate, the shared timing seen under standard schedules was produced by the schedule. As a further check, we repeat the core ResNet-18 runs with both SGD and Adam to rule out the optimizer-only origin of tails described in Crafting Heavy-Tails.

Experiment 2: spectral scars. We reproduce the blur deficit with onsets before and after each layer's settling time, aiming to match the reported jump from about 6% to over 18% error. We then test whether the converged spectra of early-blur and clean models differ in early layers but not late ones. The comparison uses tail exponent and edge-exceeding eigenvalue count, judged against seed variance. A data-free classifier distinguishing damaged from clean checkpoints from these spectral features, evaluated on held-out seeds, is the concrete test.

Experiment 3: windowed spectral regularization. We train LLaMA-style models of 60M and 350M parameters on C4 under the four AlphaDecay variants and score them on validation perplexity and the PIQA, HellaSwag, ARC-Easy, ARC-Challenge, and WinoGrande suite. Alongside, we check whether attention modules settle before feed-forward modules in these runs. The proposal's practical claim survives only if per-layer windows match full-duration AlphaDecay while stopping spectral computation early, and also beat the global 20% cutoff. Otherwise we report that the timing hypothesis explains but does not improve.