A Simple, Fast Algorithm for Continual Learning from High-Dimensional Data

Neil Ashtekar, Vasant G HonavarOpenReview 2023 · Submission to Tiny Papers @ ICLR · OpenReview

TL;DR

This paper introduces a fast, lightweight alternative to resource-heavy deep learning for continual learning. By combining Adaptive Resonance Theory (ART) with Incremental Principal Component Analysis (IPCA), the authors create an algorithm that avoids catastrophic forgetting and handles high-dimensional data efficiently, performing competitively with complex deep learning models like iCaRL.

The Challenge of Continual Learning

In continual learning, a model learns a sequence of tasks one after another. The ultimate goal is to learn new information without losing what was previously learned—a phenomenon known as catastrophic forgetting.

Most modern solutions rely on deep learning, which comes with significant baggage: high computational costs, massive memory requirements, and poor performance when task identifiers are missing at test time (single-headed settings). Older, non-neural approaches like Adaptive Resonance Theory (ART) have mathematical guarantees against forgetting but struggle with the "curse of dimensionality" when dealing with high-resolution data like images. This paper seeks to bridge that gap, providing the stability of ART with the efficiency needed for high-dimensional inputs.

Methods: Integrating ART with IPCA

The authors propose a matching-based algorithm that integrates Incremental PCA (IPCA) into the ART framework. The process unfolds in four main steps:

  1. Dimensionality Reduction: As new tasks arrive, the model updates an IPCA matrix to reduce incoming data from $d$ dimensions to a more manageable $k$ dimensions.
  2. Original Space Weights: Crucially, the model stores its learned "category weights" in the original high-dimensional space rather than the compressed space.
  3. The Matching Process: To classify a new sample, both the sample and the stored weights are projected into the current IPCA-reduced space. The model then looks for the "best match" based on a similarity function.
  4. Vigilance & Adaptation: If the best match is "close enough" (determined by a vigilance parameter) and has the correct label, the weight is updated in the original space. If no good match exists, the model simply creates a new category weight.

The core theoretical contribution is Proposition 1: the authors prove that because the weights are linear combinations of training samples, storing them in the original space allows the model to "re-project" them whenever the IPCA matrix updates. This ensures that old knowledge stays relevant even as the model's view of the data changes.

Datasets and Evaluation

The model was evaluated on two standard benchmarks for continual learning:

  • Split MNIST: The classic handwritten digit dataset, split into five tasks (e.g., 0 vs 1, 2 vs 3).
  • Split Fashion-MNIST (F-MNIST): A more challenging dataset of clothing items, similarly split into five tasks.

The authors tested these in both single-headed (the model doesn't know which task it's performing at test time) and multi-headed (task ID is provided) settings.

Key Insights

  • Linearity as a Shield: The linearity of PCA allows the model to bypass the need for re-training. By storing weights in the original space, the model adapts to a shifting feature space simply by re-applying the new projection matrix to old weights.
  • Competition Prevents Forgetting: By using ART’s "winner-take-all" competition, the model only modifies weights that are highly similar to the new data. This localized updating naturally protects unrelated, previously learned knowledge.
  • Simplicity Wins: For many continual learning benchmarks, a well-tuned linear dimensionality reduction tool combined with a stable clustering algorithm can match the accuracy of deep neural networks that are orders of magnitude more complex.

Key Results

The proposed IPCA-based model was compared against deep learning baselines (EWC, iCaRL, DEN) and variations of itself (No PCA, Static PCA).

Single-headed accuracy results on Split MNIST and Split F-MNIST
  • Accuracy: On Split MNIST in the challenging single-headed setting, the IPCA model achieved 94.03% accuracy. This drastically outperformed deep learning methods like EWC (20.01%) and DEN (56.95%), and performed nearly identically to the much heavier iCaRL (94.57%).
  • PCA Benefit: Using IPCA significantly boosted performance over a "No PCA" version (which scored 92.99% on MNIST), proving that dimensionality reduction is vital for these matching-based algorithms to succeed on image data.
  • Efficiency: Unlike deep learning methods that require backpropagation and complex regularization, this algorithm uses simple linear transformations and similarity matches, making it significantly faster to train.

Limitations and Future Work

  • The Matching Assumption: Proposition 1 assumes that a sample which matches a weight in one version of the IPCA space will still match it in a later version. While experiments show this is a good approximation, it isn't guaranteed to hold perfectly in all scenarios.
  • Linearity Constraints: PCA is a linear technique. For extremely complex, non-linear data, future work could explore non-linear incremental methods or hierarchical versions of IPCA.
  • Deep Learning Integration: The authors suggest that the "competition" mechanism of ART could be retrofitted into deep learning architectures to help them avoid forgetting without the high overhead of current rehearsal-based methods.

Summary updated

Abstract

As an alternative to resource-intensive deep learning approaches to the continual learning problem, we propose a simple, fast algorithm inspired by adaptive resonance theory (ART). To cope with the curse of dimensionality and avoid catastrophic forgetting, we apply incremental principal component analysis (IPCA) to the model's previously learned weights. Experiments show that this approach approximates the performance achieved using static PCA and is competitive with continual deep learning methods. Our implementation is available on https://github.com/neil-ash/ART-IPCA
BibTeX
@misc{ashtekar2023simple,
  author = {Neil Ashtekar and Vasant G Honavar},
  title = {{A Simple, Fast Algorithm for Continual Learning from High-Dimensional Data}},
  year = {2023},
  url = {https://openreview.net/forum?id=TPTbHxeR6U},
  howpublished = {OpenReview},
  note = {Submitted to Tiny Papers @ ICLR},
}
Figure