Critical Learning Periods Emerge Even in Deep Linear Networks

Michael Kleinman, Alessandro Achille, Stefano Soatto

Overview

Critical learning periods—early developmental windows where sensory deficits can cause permanent representational damage—are not just a biological quirk. This paper demonstrates that these periods emerge naturally in simple, analytically tractable Deep Linear Networks (DLNs). The findings suggest that critical periods are a fundamental consequence of model depth and data structure, rather than an artifact of biological hardware or complex non-linearities.

The Mystery of Critical Periods

In biology, "critical periods" are well-documented: if a kitten’s eye is sutured shut early in life, it may never develop proper vision in that eye, even after the suture is removed. Surprisingly, similar phenomena have been observed in artificial Deep Neural Networks (DNNs).

The big question is why? Is it due to biological factors like changing neural plasticity? Or is it an artifact of artificial implementations, like ReLU units "dying" or specific learning rate schedules? To isolate the root cause, the authors look for the most minimal model possible that still exhibits this behavior. They turn to Deep Linear Networks—networks that lack non-linear activation functions but still possess the hierarchical structure of deep learning.

Analytical Methodology

The authors analyze two primary settings using DLNs:

  1. Multi-pathway Framework: They simulate a system with two competing input streams (analogous to two eyes) feeding into a shared output. They introduce "gating deficits" where one pathway is temporarily blocked from learning.
  2. Matrix Completion: They use deep matrix factorization to study how pre-training on one task (the "deficit" period) affects generalization on a subsequent task.

Crucially, because these networks are linear, the authors can derive exact differential equations (ODEs) to describe how the network's weights and singular values evolve over time. This allows them to move beyond mere observation and actually calculate the learning dynamics.

Key Insights and Dynamics

  • Depth is the Catalyst: Critical learning periods do not exist in shallow (depth-1) networks. As the network becomes deeper, the learning dynamics become increasingly "brittle." Depth creates a competitive "race" between features; once one feature or pathway gains an early lead, it can permanently suppress others.
  • Winner-Take-All Dynamics: In multi-pathway models, pathways compete to explain the output. An early deficit in one pathway allows the other to "win" the representation of those features. Because of the sigmoidal nature of learning in deep networks, the laggard pathway often cannot catch up once the leader has converged.
Phase portraits demonstrating that early deficits in one pathway lead to the other dominating feature learning. This winner-take-all effect intensifies significantly with network depth (D).
  • Early vs. Late Deficits: The timing of a deficit dictates its permanence. Early deficits have a permanent impact on what a pathway can learn. Conversely, late deficits (occurring after the "critical period" has closed) have almost no lasting effect on the network's final representations.
Evolution of path singular values under early, middle, and late deficits. Early deficits permanently stunt the affected pathway, while late deficits have minimal lasting impact.
  • Generalization as a Transfer Problem: In matrix completion, training on a "corrupted" or high-rank task early on can bake in brittle features that prevent the network from ever finding the simpler, low-rank solution required for the final task.
  • Biological Alignment: The model successfully replicates classical experiments in neuroscience, such as Guillery’s 1972 study on monocular deprivation and local lesions, showing that these simple mathematical models capture the essence of biological competition.

Limitations and Future Work

While DLNs provide incredible theoretical insight, they are still linear. They cannot represent the complex, non-linear functions that modern AI (like LLMs or Vision Transformers) can. Although the authors show that Tanh and ReLU networks exhibit similar qualitative behavior in their appendix, a full theoretical bridge to non-linear architectures remains open.

Future research could investigate whether these insights can be used to "re-open" critical periods in artificial systems—perhaps allowing models to unlearn early biases or adapt more flexibly to new, conflicting data distributions.

Summary updated

Abstract

Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems. This suggests that critical periods may be fundamental to learning and not an accident of biology. Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend on the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, we extend our analysis to multi-task learning to show that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks.
BibTeX
@inproceedings{kleinman2024critical,
  author = {Michael Kleinman and Alessandro Achille and Stefano Soatto},
  title = {{Critical Learning Periods Emerge Even in Deep Linear Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=Aq35gl2c1k},
}
Figure