Introduction
Continual Learning (CL) is a cornerstone of developing artificial intelligence that mirrors human-like adaptability. The primary obstacle in this field is Catastrophic Forgetting (CF), where neural networks overwrite existing weights when learning new tasks. Orthogonal Projection methods, such as Orthogonal Weight Modification (OWM), offer a promising solution by restricting updates to subspaces that do not interfere with prior knowledge. However, OWM's reliance on fixed hyperparameters introduces estimation errors that accumulate over time. This paper presents Adaptive Orthogonal Projection (AOP), a novel framework that dynamically adjusts projection parameters based on data statistics, achieving state-of-the-art results in both batch and online continual learning.
The Limitation of Fixed Regularization
The efficacy of OWM relies on calculating a projection matrix that defines the "safe" subspace for weight updates. This calculation involves a matrix inversion that typically requires Tikhonov regularization—adding a small constant, $\alpha$, to the diagonal to ensure invertibility.
The authors demonstrate that OWM's use of a fixed, arbitrary hyperparameter for $\alpha$ introduces a persistent estimation error. This "drift" from the actual input space of previous tasks accumulates as the model encounters more tasks, eventually leading to significant forgetting. This issue is particularly acute in Class-Incremental Learning (Class-IL), where the model must distinguish between all classes across all tasks without task-specific identifiers during inference.
Adaptive Orthogonal Projection (AOP)
AOP addresses the drift problem by making the projection matrix data-dependent. Instead of a static constant, AOP dynamically calculates an optimal $\alpha$ based on the statistical properties of the current data batch. Key components of the AOP framework include:
- Statistical Estimation: Utilizing the Woodbury matrix identity and the rule of expectation, AOP estimates the projection matrix more accurately by incorporating the mean and covariance of the input data.
- Adaptive $\alpha$ Calculation: The regularization term $\alpha$ is derived from the average value of the expected correlation matrix of the current batch, aligning the projection with the actual geometry of the data.
- Recursive Updates: By integrating these improvements into a Recursive Least Squares (RLS) framework, AOP allows for efficient, incremental updates to the projection matrix.
- Online Suitability: AOP is the first gradient-based orthogonal method capable of Online Continual Learning, where data is processed in a single pass without the need for storage or rehearsal of old samples.
Experimental Evaluation
The performance of AOP was evaluated against several benchmarks, including MNIST, EMNIST-47, CIFAR-10, and CIFAR-100. The experiments covered both Batch CL (multiple epochs per task) and Online CL (single pass).
Results in Class-Incremental Learning
AOP demonstrated superior performance in Class-IL scenarios, particularly as the number of tasks increased. On the EMNIST-47 benchmark (47 tasks), AOP achieved an accuracy of 79.4%, outperforming OWM's 77.5%. In the most challenging CIFAR-100 scenario (100 tasks), AOP maintained a significant lead over traditional methods like EWC and LwF, which often struggle to exceed single-digit accuracy in such high-task settings.
Performance in Online Continual Learning
In the high-speed Online CL setting, AOP's adaptive mechanism provided substantial gains. For the CIFAR-100 (10 tasks) online stream, AOP reached 58.0% accuracy, a nearly 10% improvement over OWM's 48.5%. Notably, AOP achieved these results without a replay buffer, whereas many competing methods rely heavily on stored data to maintain performance.
Efficiency and Privacy
Beyond accuracy, AOP offers significant advantages in terms of computational efficiency and data privacy.
- Speed: AOP is highly efficient, particularly in online settings. It is approximately 35x faster than replay-based methods like GSS, which incur high overhead from managing and querying data buffers.
- Privacy: Because AOP does not store any raw data from previous tasks—only the learned projection matrix—it is inherently suited for applications with strict privacy requirements, such as medical or proprietary datasets.
Conclusion
Adaptive Orthogonal Projection (AOP) provides a mathematically grounded fix to the estimation errors found in previous orthogonal projection methods. By making the projection adaptive to data statistics, AOP significantly reduces catastrophic forgetting in both batch and online learning. Its combination of high accuracy, computational efficiency, and privacy-preserving nature makes it a compelling framework for real-world continual learning applications.
Summary updated