On the Theory of Continual Learning with Gradient Descent for Neural Networks

Hossein Taheri, Avishek Ghosh, Arya Mazumdar

TL;DR

This paper provides the first closed-form theoretical guarantees for catastrophic forgetting in neural networks trained via gradient descent. By analyzing a two-layer network on sequential, non-linear XOR-cluster tasks, the authors reveal how network width, training iterations, and—surprisingly—the sample size of subsequent tasks interact to control forgetting dynamics.

What Problem Does This Paper Address?

Deep neural networks excel at learning from massive datasets, but they struggle when data arrives sequentially. When trained on a new task, they notoriously suffer from catastrophic forgetting—rapidly overwriting the knowledge required to perform previously learned tasks.

While a growing body of literature provides mathematical guarantees for continual learning in simple linear or generalized linear models, extending these theories to non-linear neural networks has proven incredibly difficult. Prior works on neural networks often lacked explicit, closed-form bounds or failed to account for how optimization algorithms and data properties jointly influence what a network remembers. This paper bridges that gap by establishing a rigorous theoretical framework for tracking forgetting dynamics in non-linear architectures.

Methods

The authors analyze a one-hidden-layer neural network with quadratic activations ($\phi(t) = t^2/2$) trained via empirical risk minimization (ERM) using gradient descent. They operate within the lazy/kernel regime (Neural Tangent Kernel regime), where the network width is large enough that the features remain relatively constant and weights stay close to their initialization.

To analyze test-time forgetting, they introduce a clean mathematical decomposition:

  1. Train-time forgetting: The explicit increase in training loss on an old task caused by updating weights on subsequent tasks.
  2. Delayed generalization gap: How learning intermediate tasks impacts the generalization error of previous tasks, which they bound using an algorithmic stability framework tailored for continual learning.

Additionally, the paper analyzes regularized continual learning (where an $L_2$ penalty discourages weights from moving too far from the previous task's solution). They prove a surprising structural equivalence: in the linearized regime, this regularized approach is mathematically identical to unregularized training with a rescaled, time-varying step size.

Datasets

The paper validates its theoretical findings using both synthetic and real-world datasets:

  • XOR-Cluster Dataset with Gaussian Noise: A canonical, non-linearly separable synthetic dataset used for the core mathematical proofs. Tasks are structured as clusters where different tasks correspond to mutually orthogonal means.
  • Split-MNIST & Split-FashionMNIST: Standard real-world image benchmarks adapted for continual learning by splitting digits or clothing categories into a sequence of binary classification tasks.
  • Gaussian-Mixture Data: Used in conjunction with a small attention-based Transformer architecture to test if the paper's insights generalize to richer models.

Key Insights and Ideas

  • The Power of Future Data: Counterintuitively, the data size of later tasks directly suppresses the forgetting of earlier tasks. Even though subsequent tasks are independent and orthogonal, larger future datasets reduce sampling noise and stabilize the shared weight trajectory.
  • No Single Silver Bullet: Forgetting cannot be solved by over-parameterization (network width) alone, nor by large sample sizes alone. Eliminating forgetting requires a delicate, joint scaling of network width, sample size, and early stopping.
  • Regularization Doesn't Change the Scaling Laws: Because weight-regularized continual learning in the kernel regime is equivalent to unregularized gradient descent with a modified step size, standard $L_2$ regularization does not fundamentally alter or improve the underlying sample or model complexity requirements.

Key Results

  • Closed-Form Forgetting Bounds: The authors derive an explicit upper bound for train-time forgetting after learning $K$ tasks. For a $d$-dimensional dataset, the forgetting error scales as: $$\tilde{O}\left(\eta T \frac{\sqrt{K-k}}{d\sqrt{n}} + \eta T \frac{\sqrt{K-k}}{d^2 \text{polylog}(d)} + \eta^2 T^2 \frac{K^2}{\sqrt{m}}\right)$$ where $n$ is the sample size per task, $m$ is the hidden-layer width, $T$ is the number of iterations, and $\eta$ is the learning rate.

  • Sufficient Conditions for Zero Forgetting: To achieve uniformly small training loss and vanishing forgetting across all tasks, it is sufficient to scale problem parameters as:

  • Samples per task: $n = \tilde{\Theta}(d^2 K)$
  • Network width: $m = \tilde{\Theta}(d^8 K^4)$
  • Training iterations: $T = \tilde{\Theta}(d^2)$
  • Empirical Corroboration: Experiments show that these theoretical scaling trends hold true across various loss functions (linear, hinge, logistic) and standard activations like ReLU and GELU.

Limitations and Future Work

  • The Lazy Regime Constraint: The analysis is restricted to the lazy/kernel regime, where weights do not travel far from initialization. Extending this theory to the feature-learning regime—where networks learn highly non-linear representations and step sizes are large—remains an open and challenging problem.
  • Orthogonality Assumption: The clean closed-form bounds rely on the assumption that task means are orthogonal. Real-world tasks often have overlapping features, which would introduce complex cross-task correlations not captured here.
  • Simplified Architectures: While the authors provide preliminary experiments on small Transformers, the rigorous proofs are limited to two-layer quadratic networks and full-batch gradient descent rather than mini-batch Stochastic Gradient Descent (SGD).

Summary updated

Abstract

Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting earlier ones, is a central goal of artificial intelligence. To better understand its underlying mechanisms, we study the limitations of continual learning in a tractable yet representative setting. Specifically, we analyze one-hidden-layer quadratic neural networks trained by gradient descent on a sequence of XOR-cluster datasets with Gaussian noise, where different tasks correspond to clusters with orthogonal means. Our analysis is based on a tight characterization of gradient descent dynamics for the training loss, which yields explicit bounds on the rate of train-time forgetting as functions of the number of iterations, sample size, number of tasks, and hidden-layer width. We then leverage an algorithmic stability framework to bound the generalization gap, leading to corresponding guarantees on test-time forgetting. Together, our results provide the first closed-form guarantees for forgetting in continual learning with neural networks and show how key problem parameters jointly govern forgetting dynamics. Numerical experiments corroborate our theoretical results.
BibTeX
@inproceedings{taheri2026theory,
  author = {Hossein Taheri and Avishek Ghosh and Arya Mazumdar},
  title = {{On the Theory of Continual Learning with Gradient Descent for Neural Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=l35QweVxgn},
}
Figure