Abstract
Continual lifelong learning is a machine learning framework inspired by human learning, where learners are trained to continuously acquire new knowledge in a sequential manner. However, the non-stationary nature of streaming training data poses a significant challenge known as catastrophic forgetting, which refers to the rapid forgetting of previously learned knowledge when new tasks are introduced. While some approaches, such as experience replay (ER), have been proposed to mitigate this issue, their performance remains limited, particularly in the class-incremental scenario which is considered natural and highly challenging. In this paper, we present a novel algorithm, called adaptive-experience replay (AdaER), to address the challenge of continual lifelong learning. AdaER consists of two stages: memory replay and memory update. In the memory replay stage, AdaER introduces a contextually-cued memory recall (C-CMR) strategy, which selectively replays memories that are most conflicting with the current input data in terms of both data and task. Additionally, AdaER incorporates an entropy-balanced reservoir sampling (E-BRS) strategy to enhance the performance of the memory buffer by maximizing information entropy. The results demonstrate that AdaER outperforms existing continual lifelong learning baselines, highlighting its efficacy in mitigating catastrophic forgetting and improving learning performance.
Introduction
Continual Lifelong Learning (CLL) aims to develop models that can learn a sequence of tasks over time without forgetting previous knowledge. A major hurdle in this field is catastrophic forgetting, where a neural network's weights, optimized for a new task, abruptly overwrite the parameters necessary for previous tasks.
While Experience Replay (ER)—storing a subset of past data in a buffer for periodic rehearsal—is a standard mitigation strategy, it often relies on random sampling. This approach is inefficient and particularly struggles in Class-Incremental Learning (Class-IL). In Class-IL, the model must distinguish between all classes encountered so far without task-specific identifiers during inference. Random sampling often fails to retain the most informative samples or leads to imbalanced buffers, especially when the data stream is non-uniform.
The AdaER Framework
The proposed AdaER (Adaptive Experience Replay) algorithm improves upon standard ER through two primary innovations: Contextually-Cued Memory Recall (C-CMR) and Entropy-Balanced Reservoir Sampling (E-BRS).
Contextually-Cued Memory Recall (C-CMR)
Rather than replaying random samples, C-CMR identifies "conflicting" memories—those most likely to be forgotten due to the current task's updates. AdaER employs a "virtual classifier," a temporary model trained solely on the incoming data batch, to calculate interference scores for buffered samples. It then retrieves a combination of:
- Example-interfered samples: Specific data points that show the highest drop in performance when evaluated by the virtual classifier.
- Task-associated samples: Data from previous tasks that are historically more susceptible to interference when learning the current data distribution.
Entropy-Balanced Reservoir Sampling (E-BRS)
To manage the memory buffer, AdaER replaces standard random reservoir sampling with E-BRS. This strategy aims to maximize the information entropy of the buffer by maintaining a strict class balance. When the buffer reaches capacity, E-BRS identifies the majority class and replaces its "least important" sample (determined by C-CMR scores) with new data. This ensures the model retains a diverse representation of all previously seen classes, even in imbalanced streams.
Experimental Evaluation
The effectiveness of AdaER was evaluated on several standard benchmarks, including Split-MNIST, Split-FMNIST, Split-CIFAR10, and Split-CIFAR100. These datasets are partitioned into sequential tasks to simulate the lifelong learning environment.
Performance Results
AdaER consistently outperformed state-of-the-art baselines such as Maximally Interfered Retrieval (MIR) and Gradient Episodic Memory (GEM):
- Accuracy: On Split-MNIST, AdaER achieved 89.6% accuracy, surpassing standard ER (86.4%). On Split-FMNIST, it reached 74.0%, a significant 6.3% improvement over ER.
- Forgetting and Transfer: In Split-CIFAR10 tests, AdaER demonstrated lower forgetting rates and achieved a positive Backward Transfer of 4.4, indicating that learning new tasks actually enhanced its performance on previous ones.
- Buffer Efficiency: AdaER proved highly robust to limited memory constraints. While competing methods saw sharp performance declines with smaller buffer sizes, AdaER maintained stable accuracy.
Conclusion
AdaER addresses the stability-plasticity dilemma in continual learning by making memory management more intelligent. By focusing on conflicting memories through C-CMR and maintaining informational diversity via E-BRS, AdaER effectively mitigates catastrophic forgetting in challenging Class-IL scenarios. Future work will explore automating hyperparameter tuning and extending the framework to domain-incremental learning environments.
Summary updated