Abstract
An important problem in machine learning is the ability to learn tasks in a sequential manner. If trained with standard first-order methods most models forget previously learned tasks when trained on a new task, which is often referred to as catastrophic forgetting. A popular approach to overcome forgetting is to regularize the loss function by penalizing models that perform poorly on previous tasks. For example, elastic weight consolidation (EWC) regularizes with a quadratic form involving a diagonal matrix build based on past data. While EWC works very well for some setups, we show that, even under otherwise ideal conditions, it can provably suffer catastrophic forgetting if the diagonal matrix is a poor approximation of the Hessian matrix of previous tasks. We propose a simple approach to overcome this: Regularizing training of a new task with sketches of the Jacobian matrix of past data. This provably enables overcoming catastrophic forgetting for linear models and for wide neural networks, at the cost of memory. The overarching goal of this paper is to provided insights on when regularization-based continual learning algorithms work and under what memory costs.
Introduction
Continual learning—the ability of a model to learn tasks sequentially without forgetting previous ones—remains a significant challenge in machine learning. Standard first-order training methods often lead to "catastrophic forgetting," where new task updates overwrite parameters critical for past performance. A popular mitigation strategy is regularization, such as Elastic Weight Consolidation (EWC), which penalizes parameter shifts based on their importance to previous tasks. However, EWC relies on a diagonal approximation of the Hessian matrix, which can fail significantly in certain regimes. This paper introduces Regularization with Sketched Jacobians (RSJ), a method that provides provable guarantees for continual learning by more accurately approximating the loss landscape of past tasks.
From Diagonal Approximations to Sketched Jacobians
Methods like EWC attempt to preserve past knowledge by adding a quadratic penalty to the loss function. To remain computationally efficient, EWC approximates the Hessian (or Fisher Information) matrix using only its diagonal elements. This simplification assumes that model parameters are independent, effectively ignoring the complex interactions between weights.
While this diagonal approximation works empirically on some benchmarks, it lacks theoretical robustness. This paper demonstrates that even under ideal conditions, EWC can suffer from catastrophic forgetting if the diagonal approximation is poor. In contrast, the proposed RSJ method uses random projections (Gaussian sketches) to compress the Jacobian matrix. This approach captures weight interactions more effectively than a diagonal matrix, bridging the gap between the impractical exact Jacobian and the overly simplistic EWC.
Empirical Evaluation
The researchers evaluated RSJ against EWC and L2 regularization on two primary benchmarks: the MNIST Permutation Problem and Incremental MNIST.
In the Permuted MNIST task, where pixels are randomly shuffled for each new task, EWC often performs surprisingly well. The paper provides a theoretical explanation for this "EWC Paradox": randomly permuted tasks are nearly orthogonal. In such cases, even simple L2 regularization can approach Bayes optimality. However, the Incremental MNIST task (class-incremental learning) presents a much greater challenge. Here, the model learns to classify pairs of digits sequentially, and EWC's performance can collapse as the number of tasks increases.
As illustrated in Figure 2, RSJ significantly outperforms EWC on Incremental MNIST. With a sketch dimension of $s=100$, RSJ maintains performance levels close to the "all data" baseline, whereas EWC's accuracy drops precipitously. This demonstrates that capturing the structural interactions of the Jacobian is critical for non-orthogonal tasks.
Key Insights and Theoretical Guarantees
The core contribution of this work is the mathematical foundation for RSJ. The authors prove that by using a random sketch of the Jacobian, RSJ enables successful continual learning for linear models and infinitely wide neural networks (within the Neural Tangent Kernel regime).
The study also highlights a fundamental trade-off: Memory vs. Performance. Higher-fidelity approximations of the Jacobian (larger sketch sizes $s$) lead to better retention of past knowledge but require more storage. Specifically, RSJ requires storing a sketched matrix of size $s \times p$, where $p$ is the number of parameters.
Conclusion
This research provides a clearer understanding of when regularization-based continual learning succeeds and why it fails. While RSJ offers provable guarantees and strong empirical performance on difficult benchmarks like Incremental MNIST, its memory requirements are a notable constraint. For very large models, the storage cost of the sketched Jacobian may approach that of replay-based methods. Future work must continue to explore the balance between the memory footprint of structural approximations and the efficacy of sequential learning.
Summary updated