Critical Learning Periods Emerge Even in Deep Linear Networks

Michael Kleinman, Alessandro Achille, Stefano Soatto

TL;DR

Researchers have discovered that "critical learning periods"—early developmental windows where sensory deficits cause permanent damage—emerge even in simple deep linear networks. This suggests that these periods are a fundamental property of deep learning dynamics driven by network depth and data structure, rather than just biological quirks like losing neural plasticity.

What Problem Does This Paper Address?

In biology, "critical periods" are well-known: if a kitten has one eye sutured shut early in life, it will never develop proper vision in that eye, even after the suture is removed. The traditional explanation is biological: as the brain ages, it loses "plasticity," making it harder to form new connections.

Surprisingly, modern deep neural networks (DNNs) exhibit similar behavior. If you train a DNN on blurred images early on, it often fails to reach peak performance even after switching to clear images. Since artificial networks don't have biological "plasticity loss," why does this happen? Is it just a bug in our optimization methods (like learning rate schedules or specific activation functions like ReLUs)?

This paper investigates whether critical periods are a fundamental consequence of the mathematics of deep learning itself.

Methods

The authors strip away the complexity of modern AI to study Deep Linear Networks (DLNs). These are networks without non-linear activation functions (like ReLU or Tanh) between layers. While the final input-output mapping of a DLN is just a simple linear transformation, the process of learning its weights is highly non-linear and complex.

They analyze two specific scenarios:

  1. Multi-pathway Models: Simulating two competing sensors (like two eyes) feeding into a single output. They introduce "gating deficits" by temporarily blocking updates to one pathway.
  2. Matrix Completion: A standard task for studying how networks generalize from partial data. Here, they "pre-train" the network on a different data distribution (a "deficit" task) before switching to the target task.

Crucially, the authors derive exact differential equations (ODEs) that characterize how the network's features (singular values) evolve over time, allowing for a rigorous analytical treatment of the learning dynamics.

Datasets

The study relies on structured synthetic data rather than standard benchmarks like ImageNet:

  • Hierarchical Synthetic Task: Used for multi-pathway experiments to simulate how different features are learned at different rates.
  • Low-Rank Matrices: For matrix completion, they generate ground-truth $100 \times 100$ matrices of specific ranks (e.g., rank-5 or rank-10) to precisely control task complexity and relationships.

Key Insights and Ideas

  • Depth is the Catalyst: Critical periods do not appear in shallow (one-layer) networks. As depth increases, the sensitivity to early deficits becomes much more pronounced. Depth creates a "winner-take-all" competitive dynamic.
  • Learning as a Race: In the multi-pathway model, pathways compete to explain the output. Because learning follows a sigmoidal (S-shaped) trajectory, a pathway that gets a head start "wins" the race for features, permanently suppressing the other pathway.
  • Brittle Features and Generalization: In matrix completion, pre-training on a "wrong" task (e.g., a higher-rank matrix) can bake in "brittle" features. If these features can somewhat explain the final task, the network gets stuck with them and fails to find the optimal, low-rank solution.
  • Fundamental Information Constraints: Because these effects emerge in linear networks, they cannot be blamed on biological plasticity, ReLU freezing, or vanishing gradients. They appear to be an inherent property of how deep architectures process information.

Key Results

  • Early vs. Late Deficits: In multi-pathway simulations, a deficit applied at the very beginning of training (epochs 0-150) causes the normal pathway to learn all features, leaving the deprived pathway permanently impaired. A deficit applied later in training has almost no effect.

    Figure 2: Early deficits (left) lead to one pathway winning the competition, while late deficits (right) allow both pathways to learn equally.
  • The Impact of Depth: Figure 1 shows that as depth ($D$) increases from 2 to 13, the "flow" of learning becomes increasingly biased toward a single pathway if an early deficit occurs.

    Figure 1: Deeper networks are more affected by a temporary early deficit. Phase portrait of pathway specific singular value Kaα and Kbα where Kaα + Kbα = σα, where σα = 10 is the singular value and denoted by the diagonal blue line in the plots. Black traces indicate a simulation for a particular initial condition (100 shown). Flow fields (purple arrows) are shown for systems without a deficit. (Top row) An early deficit to one pathway (“B”) leads to the other pathway learning more of the feature, and this effect becomes increasingly more pronounced in deeper networks (shown for depths ranging from 2 to 13). (Bottom row) In contrast, late deficits have a negligible effect on the final solution, where neither pathway dominates (on average) how a feature is learned.
  • Task Interference: In matrix completion, pre-training on a rank-10 task before switching to a rank-5 task significantly damages final performance, but only if the pre-training lasts through the "critical" early phase of learning.

    Figure 5: Sensitivity to initial deficits in matrix completion depends on the rank of the initial task and the number of observations.

Limitations and Future Work

  • Linearity vs. Reality: While the authors show in the appendix that Tanh and ReLU networks behave similarly, the core analytical results are for linear networks. Real-world DNNs have much more complex non-linearities.
  • Simple Tasks: The synthetic tasks used are far simpler than real-world computer vision or natural language processing.
  • Future Directions: The authors suggest that understanding these dynamics could lead to better "curriculum learning" strategies (deciding the best order to present tasks to a network) or methods to "re-open" critical periods in AI to allow for more flexible lifelong learning.

Summary updated

Abstract

Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems. This suggests that critical periods may be fundamental to learning and not an accident of biology. Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend on the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, we extend our analysis to multi-task learning to show that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks.
BibTeX
@inproceedings{kleinman2024critical,
  author = {Michael Kleinman and Alessandro Achille and Stefano Soatto},
  title = {{Critical Learning Periods Emerge Even in Deep Linear Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=Aq35gl2c1k},
}
Figure