Critical Learning Periods Emerge Even in Deep Linear Networks

Michael Kleinman, Alessandro Achille, Stefano Soatto

Abstract

Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems. This suggests that critical periods may be fundamental to learning and not an accident of biology. Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend on the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, we extend our analysis to multi-task learning to show that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks.

The Mystery of Critical Periods

In biology, a "critical period" is a window of high plasticity early in life where the brain is exceptionally sensitive to environmental input. A classic example is monocular deprivation: if a kitten’s eye is sutured shut during this window, it may never develop proper vision, even if the sutures are later removed. Traditionally, neuroscientists attributed this to biological "hardware" changes, such as a loss of synaptic plasticity.

However, recent research has shown that artificial Deep Neural Networks (DNNs) also exhibit critical periods. This raises a fundamental question: are critical periods caused by specific biological constraints or artificial "defects" (like vanishing gradients), or are they a fundamental property of learning in deep architectures? This paper seeks to isolate the core factors—specifically depth and data structure—that cause these phenomena to emerge.

Modeling Learning Dynamics

To investigate this, the authors focus on Deep Linear Networks (DLNs). While the input-output map of these networks is linear, their learning dynamics are highly non-linear and complex. The researchers employ two primary frameworks: multi-pathway DLNs to simulate competing sensors (like two eyes) and deep matrix factorization to study how early training on one task affects subsequent generalization.

Crucially, the authors derive Ordinary Differential Equations (ODEs) that characterize the evolution of the network's singular values. This allows them to analytically prove why certain features are learned early and why early deficits become permanent.

The Competitive Race to Learn

Learning in deep networks is a competitive process. Singular values, which represent learned features, undergo sigmoidal learning trajectories where larger, more prominent features are learned faster and earlier. In a multi-pathway system, if one pathway is blocked early, the other pathway "wins" the race to explain the data so decisively that the blocked pathway can almost never catch up.

Figure 2: Vector field analysis showing how increasing depth (D) and the timing of deficits (Early vs. Late) affect the learning trajectories of singular values in Path A and Path B.

As shown in Figure 2, depth acts as a powerful amplifier of this effect. In shallow networks, pathways can often recover from early imbalances. However, as depth increases, the dynamics become increasingly "winner-take-all." Once a pathway gains a significant lead, it suppresses the other, making early deficits nearly impossible to overcome.

Timing and Permanence

The most striking finding is the sensitivity to the onset of the deficit. Simulations show that a deficit in the first 150 epochs leads to a permanent loss of representation in the deprived pathway. Conversely, a deficit of the same duration applied late in training (e.g., epoch 750) has virtually no effect on the final outcome.

Figure 3: Comparison of learning trajectories for Early, Middle, and Late deficits. The dashed lines (ODE) perfectly match the experimental markers (SGD), demonstrating that early deficits lead to permanent representational loss.

Figure 3 illustrates this phenomenon. In the "Early Deficit" case, the path singular values for the deprived pathway never recover to the levels seen in the "Late Deficit" case. The mathematical alignment between the ODE predictions and the Stochastic Gradient Descent (SGD) results confirms that these critical periods are a direct consequence of the underlying gradient descent dynamics in deep architectures.

Conclusion

By stripping away the complexities of non-linear activations, this work demonstrates that critical learning periods are not biological accidents. Instead, they emerge naturally from the hierarchical nature of deep learning and the competitive dynamics of feature acquisition. These results suggest that critical periods might be an unavoidable consequence of how information is processed in any hierarchical system, biological or artificial. Understanding these dynamics could lead to better "curriculum learning" strategies or help identify the optimal timing for multi-modal pre-training to ensure one modality doesn't permanently "blind" another.

Summary updated

Abstract

Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems. This suggests that critical periods may be fundamental to learning and not an accident of biology. Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend on the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, we extend our analysis to multi-task learning to show that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks.
BibTeX
@inproceedings{kleinman2024critical,
  author = {Michael Kleinman and Alessandro Achille and Stefano Soatto},
  title = {{Critical Learning Periods Emerge Even in Deep Linear Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=Aq35gl2c1k},
}
Figure