Understanding and Leveraging the Learning Phases of Neural Networks

Johannes Schneider, Mohit Prabhushankar

Abstract

Recent research into the internal dynamics of deep neural networks suggests a three-phase learning process: stability, fitting, and compression. By monitoring reconstruction loss—the ability to recreate inputs from hidden layer activations—this study provides a measurable framework for these phases. The findings reveal a critical trade-off: as networks reach peak classification accuracy, they discard descriptive input information, which can paradoxically degrade performance in transfer learning tasks.

Introduction

Despite the widespread success of deep learning, the evolution of internal representations during training remains poorly understood. The Information Bottleneck (IB) theory posits that networks first fit data and then compress it by discarding irrelevant information. However, empirical validation of IB theory is hampered by the difficulty of measuring mutual information in high-dimensional spaces. This paper addresses this gap by using reconstruction loss as a proxy for information retention, offering a rigorous way to map learning phases across various architectures and datasets.

Methodology

The authors employed a combination of empirical tracking and theoretical modeling:

  1. Empirical Tracking: Standard architectures (VGG-11, ResNet-10, and linear networks) were trained on MNIST, FashionMNIST, and CIFAR datasets. At various training stages, "decoder" networks were trained to reconstruct original inputs from frozen hidden layer activations.
  2. Reconstruction vs. Accuracy: The relationship between classification accuracy and reconstruction error was analyzed to observe how "forgetting" correlates with "learning."
  3. Theoretical Modeling: A mathematical model of a linear classifier was developed to prove that the observed three-phase reconstruction loss is a necessary consequence of optimization in softmax-based classifiers.
  4. Transfer Learning Probe: The study investigated whether the "compression" phase, where information is discarded, impacts the efficacy of fine-tuning for downstream tasks.

Key Insights: The Three-Phase U-Curve

The central discovery is that reconstruction loss follows a distinct "U-shaped" trajectory over the course of training.

Figure 1: The U-shaped reconstruction loss curve alongside rising classification accuracy, illustrating the transition from fitting to compression.
  • Phase 1 (Stability): Initial loss remains constant as the network moves away from random initialization.
  • Phase 2 (Fitting): Loss decreases rapidly as the network learns features necessary to represent the input data.
  • Phase 3 (Compression): Loss begins to increase. As the network approaches 100% accuracy, it becomes a specialist, discarding "nuanced differences" and non-discriminative features to focus solely on the labels.

Key Results and Transfer Learning

The U-shaped curve was consistently observed across all architectures and datasets, confirming that compression is an inherent part of the optimization process.

Figure 6: Consistent U-shaped reconstruction curves across different datasets (MNIST, FashionMNIST) and layers in a ResNet-10 architecture.

A significant finding for practitioners is the "Transfer Learning Twist." The authors discovered that peak pre-training accuracy is often a "trap." Downstream performance on new tasks actually peaks much earlier—specifically at the point of minimum reconstruction loss (the bottom of the U-curve). In VGG experiments transferring between CIFAR-100 and CIFAR-10, stopping pre-training at this point yielded better final accuracy than using a fully converged model.

Conclusion

This work provides a measurable alternative to abstract information-theoretic metrics for understanding neural network dynamics. By identifying the reconstruction "U-curve," the authors offer a practical guide for early stopping in transfer learning. While the mathematical proofs currently focus on linear layers, the empirical evidence across deep architectures suggests a universal phenomenon. Future research will aim to extend these theoretical bounds to multi-layer non-linear dynamics and complex architectures like Transformers.

Summary updated

BibTeX
@inproceedings{schneider2024understanding,
  author = {Johannes Schneider and Mohit Prabhushankar},
  title = {{Understanding and Leveraging the Learning Phases of Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i13.29408},
}
Figure