Parabolic Continual Learning

Haoming Yang, Ali Hasan, Vahid Tarokh

Abstract

Regularizing continual learning techniques is important for anticipating algorithmic behavior under new realizations of data. We introduce a new approach to continual learning by imposing the properties of a parabolic partial differential equation (PDE) to regularize the expected behavior of the loss over time. This class of parabolic PDEs has a number of favorable properties that allow us to analyze the error incurred through forgetting and the error induced through generalization. Specifically, we do this through imposing boundary conditions where the boundary is given by a memory buffer. By using the memory buffer as a boundary, we can enforce long term dependencies by bounding the expected error by the boundary loss. Finally, we illustrate the empirical performance of the method on a series of continual learning tasks.

Introduction

Artificial intelligence models frequently encounter the challenge of catastrophic forgetting, where training on a sequence of new tasks leads to an abrupt loss of performance on previously learned ones. Existing strategies typically involve either weight regularization or experience replay using a memory buffer. However, these methods often lack a unified mathematical framework to predict and control the evolution of the model's loss profile over time.

A robust continual learner must achieve two primary objectives: knowledge retention (minimizing loss spikes on old tasks) and generalization (rapidly calibrating to new data distributions). This paper proposes framing continual learning as a physical process governed by parabolic partial differential equations (PDEs), enabling the use of classical physics-based mathematics to bound learning errors.

Methodology: Parabolic Continual Learning (PCL)

The core innovation of this work is the Parabolic Continual Learner (PCL), which treats the evolution of the loss function $u(x, t)$ as a diffusive process similar to heat flow.

The PDE Analogy and Boundary Conditions

By modeling the loss profile using a variation of the heat equation, the authors define the "boundary" of the data space using the memory buffer—a small set of saved past samples. Enforcing the PDE ensures that the loss at any interior point in the data domain is mathematically constrained by the loss at these boundary points. This provides a theoretical guarantee that the error on previous tasks remains bounded by the error at the current time.

Conceptual framework of Parabolic Continual Learning (PCL).

Practical Implementation via Feynman-Kac

Solving high-dimensional PDEs directly is computationally prohibitive for deep learning. To overcome this, PCL utilizes the Feynman-Kac formula, which establishes a link between PDEs and stochastic processes. Instead of a direct solution, the model samples Brownian bridges—stochastic paths connecting new data points to samples in the memory buffer. Optimization is then performed by minimizing the expectation over these paths, effectively smoothing the loss landscape. This diffusive property ensures more robust learning in regions between data points compared to linear interpolation methods like Mixup.

Experimental Evaluation

PCL was evaluated on standard continual learning benchmarks, including Sequential CIFAR-10, Sequential CIFAR-100, and Sequential TinyImageNet. These were tested under challenging conditions such as online class-incremental learning (one-pass training), label corruption (50% noise), and data imbalance.

Results

PCL demonstrated superior or competitive performance against state-of-the-art (SOTA) methods like DER++, CLSER, and VRMCL. Notably, on Seq-CIFAR100, PCL achieved a final accuracy of 20.54%, surpassing the previous SOTA of 19.33%.

Performance comparison of PCL against state-of-the-art methods across standard benchmarks.

Key findings include:

  • Efficiency: PCL is approximately five times faster than meta-learning competitors like VRMCL, as it enforces physical constraints rather than simulating future tasks.
  • Robustness: The method remains stable under high label noise (50% corruption), avoiding the "memorization" of noise common in other replay-based methods.
  • Memory Efficiency: PCL excels even with extremely small buffers (e.g., 50 samples), significantly outperforming other methods in anytime accuracy.

Conclusion

Parabolic Continual Learning provides a rigorous mathematical framework for balancing stability and plasticity in neural networks. By leveraging the properties of parabolic PDEs, PCL offers theoretical error bounds and state-of-the-art empirical performance with high computational efficiency. Future work will focus on understanding the evolution of these bounds during parameter optimization and extending the framework to multimodal domains and stochastic control problems.

Summary updated

Abstract

Regularizing continual learning techniques is important for anticipating algorithmic behavior under new realizations of data. We introduce a new approach to continual learning by imposing the properties of a parabolic partial differential equation (PDE) to regularize the expected behavior of the loss over time. This class of parabolic PDEs has a number of favorable properties that allow us to analyze the error incurred through forgetting and the error induced through generalization. Specifically, we do this through imposing boundary conditions where the boundary is given by a memory buffer. By using the memory buffer as a boundary, we can enforce long term dependencies by bounding the expected error by the boundary loss. Finally, we illustrate the empirical performance of the method on a series of continual learning tasks.
BibTeX
@inproceedings{yang2025parabolic,
  author = {Haoming Yang and Ali Hasan and Vahid Tarokh},
  title = {{Parabolic Continual Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=7gausJk3jB},
}
Figure