Does Weight Growth Close Critical Learning Periods?

Deep networks have "critical learning periods": short windows early in training when a defect in the input, such as blurred images, does permanent damage. The same defect applied later is harmless, and the damage from an early defect cannot be repaired by any amount of later clean training. Separately, optimization theory has shown that classification networks trained with cross-entropy usually have no finite best set of weights. Their weight norm keeps growing toward infinity while the direction of the weights settles down. This note proposes that the two facts are connected through one quantity. In layers followed by normalization, the output does not depend on the weight scale, so a fixed learning rate turns into an ever-smaller rotation of the weights as their norm grows. Our hypothesis is that the critical period closes when this effective step size has shrunk enough that the weight directions can no longer move. The clock that ends the window would then be weight growth, not elapsed epochs.

The claim is cheap to test and has real consequences. It makes three predictions. Holding weight norms fixed should stretch or abolish the window. Shrinking the norms after the window closes should partly reopen it. Damage curves from runs with very different weight decay, initialization scale and learning rate should line up once we plot them against how far the weights have actually rotated, rather than against epochs. If the predictions fail, that is also useful: it would show that representations consolidate in some way unrelated to weight magnitude, which the critical-period literature currently cannot tell apart from the mechanical explanation.

Core Claim

Call a weight tensor $W$ scale-invariant if a normalization layer follows it, so the network's output does not change when $W$ is multiplied by a positive constant. For such a tensor under plain gradient descent with learning rate $\eta$, the gradient scales as $1/\|W\|$. This was shown for weight normalization and weight standardization in Understanding the Disharmony between Weight Normalization Family and Weight Decay. It follows that the angle the weights turn per step scales as the effective learning rate

$$\eta_{\text{eff}}(t) = \frac{\eta}{\|W_t\|^2}.$$

We define the network's angular clock as the cumulative rotation

$$A(t) = \sum_{s<t} \angle(W_s, W_{s+1}),$$

summed over scale-invariant layers. The claim is that the critical period is a window in $A$, not in epochs. A deficit that ends at rotation $A$ causes the same permanent damage whatever schedule of norm growth produced that $A$. Restoring a large $\eta_{\text{eff}}$ after the window has closed should therefore restore at least part of the network's ability to recover.

For the final classifier layer, which is not scale-invariant, the matching story comes from directional convergence. Directional convergence and alignment in deep learning proves that homogeneous networks under cross-entropy diverge in norm while their normalized weights converge to a limit direction with finite path length. Finite path length means the direction can only travel a bounded distance in total. If an early deficit uses up much of that budget, the limit direction is effectively chosen during the deficit.

Relation to Prior Work

Three papers motivate the hypothesis. Critical Learning Periods in Deep Networks established the phenomenon on CIFAR-10. A blur deficit left uncorrected for the first 40–60 epochs raises final test error from about 6% to over 18%, with peak sensitivity around epoch 30. The paper tracked plasticity through the trace of the Fisher information, a measure of how strongly the outputs depend on the weights. It also reported that the effect survives changes of optimizer and learning-rate schedule. That finding is our main threat, and we return to it below. Time Matters in Regularizing Deep Networks showed that weight decay matters only early. For ResNet-18 on CIFAR-10 it reports about 93% accuracy when weight decay is removed after epoch 75, about 88% when weight decay starts only at epoch 50, and about 87% without weight decay. It recorded weight norms, but it never treated norm growth as the cause of anything. Implicit Regularization and Convergence for Weight Normalization proved a "direction first, scale later" dynamic for weight-normalized linear regression. That result suggests the same staging could govern deep classifiers, but it says nothing about deficits.

Two papers supply methods. The Disharmony paper provides the effective-learning-rate identity and Adaptive Weight Shrink, a weight decay whose strength is proportional to $\|W\|$. That method gives stable training across 1×, 20× and 100× initialization scales and reaches about 76.6% top-1 for WS-ResNet50 on ImageNet. We reuse it as a norm-control tool. Gradient descent follows the regularization path for general losses shows that unregularized gradient descent on linear predictors tracks the norm-constrained regularization path. This motivates treating the norm as a budget that the direction spends.

One result constrains us. Golatkar et al. found that late-onset weight decay fails. Late weight decay should eventually shrink norms and raise $\eta_{\text{eff}}$, so this is mild evidence that restoring the effective step size is not sufficient. Two explanations are possible: weight decay reaches its equilibrium norm slowly, or consolidation also has a component unrelated to the norm. We do not argue around this result. Instead we separate necessity from sufficiency and test them separately.

None of these papers measures deficit damage as a function of weight-norm trajectory, or tries to reopen a closed window by manipulating the norm. Early Period of Training Impacts Out-of-Distribution Generalization manipulates the early window through gradual unfreezing and tracks Fisher trace and sharpness. It treats parameters as bounded and is silent on growth.

Research Program

The first step is measurement. We train ResNet-18 and a VGG-style network with batch normalization on CIFAR-10 under the blur-deficit protocol of Achille et al. Deficit end-times are swept from 0 to 100 epochs, and each run continues to a fixed total of clean training. We log per-layer weight norms, $\eta_{\text{eff}}$, the angular clock $A(t)$, and a mini-batch estimate of the Fisher trace. Running this sweep under several weight-decay strengths, three initialization scales (1×, 20×, 100×, matching the Disharmony paper's range) and three base learning rates gives a family of "damage versus deficit-end" curves.

The second step tests necessity. We repeat the sweep with every scale-invariant weight projected back onto a sphere of fixed radius after each step. This freezes $\eta_{\text{eff}}$ at its initial value without changing the function the network computes. We run it once more with Adaptive Weight Shrink as a softer form of control.

The third step tests sufficiency. After a deficit that ends inside the closed window, for example at epoch 60, we rescale all scale-invariant weights back to their initialization norm in one step. This is function-preserving, because the scale-invariant layers ignore the scale. It also resets the momentum buffers and continues clean training. If the hypothesis is right, the network should partly recover.

The fourth step isolates the classifier head. We retrain only the last layer on frozen clean-trained features, and we separately freeze the head and reset only the backbone norms. Together these show whether the lasting damage sits in the features, where the effective-learning-rate story applies, or in the head's locked-in limit direction, where the directional-convergence story applies.

Baselines

The naive reference point is the standard deficit protocol with no intervention: blur for the first $T$ epochs, then clean training. Achille et al. report final test error rising from roughly 6% to over 18% when the deficit lasts 40–60 epochs. Every intervention is run on identical data, architecture and total epoch budget. Each gets the same small grid of learning rates, selected on a held-out split. Any method that measurably lowers the error after a closed-window deficit beats this reference.

The strongest existing competitor for recovering from a closed-window deficit is restarting the learning rate after the deficit, that is, re-warming to the peak and repeating the decay. The ingredient it targets, step size, is exactly ours. Plain SGD on scale-invariant weights is the one case where shrinking the norm by a factor $c$ is mathematically the same as multiplying the learning rate by $c^2$. Our comparison therefore matches the effective learning rate at the moment of intervention. The two interventions then differ only where the equivalence breaks: with momentum, with Adam, and on non-scale-invariant parameters such as biases and the classifier. To beat this baseline, norm rescaling must recover more accuracy at matched $\eta_{\text{eff}}$, or the angular clock must explain the timing of the window better than the raw learning rate does. For this exact recovery task there is no reported result on Lacuna for learning-rate restarts or any other recovery method.

For the role of norm control during training, the strongest existing method is Adaptive Weight Shrink from the Disharmony paper. We run it with its published proportional-decay rule, under the same deficit sweep. The weight-decay timing results of Golatkar et al. (about 93%, 88% and 87%) serve as anchors on the clean, no-deficit runs, to confirm that our pipeline reproduces them.

The ablation that isolates the key ingredient keeps sphere projection but compensates by shrinking the learning rate on the schedule that a growing norm would have produced. This restores the original $\eta_{\text{eff}}$ trajectory while the norm stays fixed. If the window comes back under this ablation, effective step size is the operative variable, and norm magnitude on its own does nothing.

Why not just…?

Why not just raise the learning rate after the deficit? For plain SGD on scale-invariant layers this is the same intervention, and we say so plainly. Our claim is not that norm rescaling is a better trick. The claim is that $\eta_{\text{eff}}$, which the norm controls, is the variable that times the window. Achille et al. report that constant or high learning rates did not rescue deficit-trained networks. Under constant raw learning rate, however, the norm still grows and $\eta_{\text{eff}}$ still collapses, so that finding is consistent with our hypothesis. It does not refute it. The matched-$\eta_{\text{eff}}$ comparison above, run with momentum SGD and Adam, settles whether anything beyond the raw learning rate matters. If the two interventions tie everywhere, we report that the window is set by effective step size however it is achieved.

Why not just keep tracking the Fisher trace? The Fisher trace is expensive at scale, and it is correlational: it falls when the window closes but cannot be manipulated directly. The weight norm costs nothing to compute and can be set by hand. If the clock prediction holds, the angular clock becomes a scalable stand-in for the Fisher trace that can also be used to intervene. To check this we test whether $A(t)$ predicts damage better than epoch count or Fisher trace across the full sweep.

Why not just use weight decay early, as Golatkar et al. recommend? That recipe describes when regularization matters but not why. It also does not address recovery after a deficit. Golatkar et al.'s late-onset failure is, in fact, the most direct existing evidence against a purely norm-based sufficiency claim. Our sufficiency experiment uses an instant rescaling rather than slow decay to separate the two explanations. A null result there, alongside a positive necessity result, would be a clean finding: norm growth closes the window, but reversing it cannot reopen the window.

Proposed Experiments and Decisive Checks

The first decisive check is curve collapse. We plot final error against deficit end-time in epochs and, separately, against $A$ at deficit end, across all weight-decay, initialization-scale and learning-rate settings, and measure the spread of the curves under each axis. The hypothesis predicts much tighter collapse in $A$. If the spread is no tighter in $A$ than in epochs, the claim fails.

The second check is necessity. Under sphere projection, a 60-epoch blur deficit should leave much less than the more-than-18% error reported for the unconstrained baseline. The compensated-learning-rate ablation should bring the damage back.

The third check is sufficiency. After a closed-window deficit, norm rescaling, and learning-rate restart at matched $\eta_{\text{eff}}$, should each recover some accuracy, measured against no intervention.

The fourth check decides where the damage lives. It compares head-only retraining with backbone-only rescaling, which tells us whether the effective-learning-rate mechanism or the directional-convergence mechanism of Ji and Telgarsky carries the deficit.

We run every experiment on CIFAR-10 and CIFAR-100. We then replicate the curve-collapse test once on ImageNet with WS-ResNet50, under Adaptive Weight Shrink and under standard weight decay with a shortened blur deficit. Recovery after a deficit is also scored on MNIST-C corruptions, so that the result connects to the out-of-distribution reading of the early window in Early Period of Training Impacts Out-of-Distribution Generalization.