Understanding and Leveraging the Learning Phases of Neural Networks

Johannes Schneider, Mohit Prabhushankar

Abstract

This study characterizes the learning dynamics of neural networks through the lens of reconstruction loss. We identify a consistent three-phase process: an initial plateau, a fitting phase where input features are learned, and a final forgetting phase where task-irrelevant information is discarded. Our findings challenge traditional Information Bottleneck theory by providing a more measurable metric. Crucially, we demonstrate that the forgetting phase can be detrimental to transfer learning, suggesting that pre-training should be terminated at the point of minimum reconstruction loss rather than peak accuracy.

Introduction

Understanding the internal journey of a neural network from random initialization to high-performance classification remains a central challenge in deep learning. While the Information Bottleneck (IB) theory provides a conceptual framework—suggesting networks first fit data and then compress it—it is difficult to verify empirically due to the complexities of calculating mutual information. This paper proposes reconstruction loss as a rigorous and measurable alternative to track how much information a layer retains about its input.

Methodology

The authors evaluate standard architectures, including VGG-11 and ResNet-10, across benchmark datasets such as MNIST, FashionMNIST, CIFAR-10, and CIFAR-100. At each training iteration, hidden layer activations are frozen, and a separate decoder is trained to reconstruct the original input. This reconstruction error serves as a proxy for information retention. Additionally, the authors provide a mathematical proof of these phases for linear classifiers using a structured data model.

The Three-Phase Learning Model

Empirical observations reveal a characteristic U-shaped curve in reconstruction loss over time:

  1. Phase I (Plateau): The network has not yet begun to represent significant input features.
  2. Phase II (Fitting): Reconstruction loss decreases as the network learns to represent the input.
  3. Phase III (Forgetting): Reconstruction loss increases as the network specializes, discarding input details that do not contribute to classification accuracy.
Reconstruction loss and classification accuracy over training iterations for a linear classifier.

Results and Discussion

The three-phase behavior is consistent across various architectures and datasets. In deeper networks, the forgetting phase is most pronounced in the final layers, while earlier layers maintain more generic representations.

Learning dynamics in a ResNet-10 architecture across different layers and datasets.

A key practical insight involves transfer learning. Experiments transferring from CIFAR-100 to CIFAR-10 show that the best downstream performance is achieved by stopping pre-training near the point of minimum reconstruction loss. Training until the source-task accuracy plateaus often leads the model into Phase III, where it discards general features that would have been beneficial for the target task.

Conclusion and Future Work

This work provides a tangible metric for observing the "fitting" and "compression" cycles of neural networks. While the theoretical bounds are currently limited to linear models, the empirical evidence across deep architectures is robust. Future research will focus on extending the mathematical framework to non-linear regimes, exploring how different loss functions affect these phases, and investigating automated early-stopping strategies for large-scale foundation models.

Summary updated

BibTeX
@inproceedings{schneider2024understanding,
  author = {Johannes Schneider and Mohit Prabhushankar},
  title = {{Understanding and Leveraging the Learning Phases of Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i13.29408},
}
Figure