Using Hindsight to Anchor Past Knowledge in Continual Learning

Arslan Chaudhry, Albert Gordo, Puneet K. Dokania, Philip H. S. Torr, David Lopez-Paz

Abstract

In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a new objective that we call anchoring, where the learner uses bilevel optimization to update its knowledge on the current task, while keeping intact the predictions on some anchor points of past tasks. These anchor points are learned using gradient-based optimization to maximize forgetting, which is approximated by fine-tuning the currently trained model on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.

Introduction

In the ideal world of machine learning, models have access to all data at once. In the real world—and specifically in Continual Learning (CL)—data arrives in a stream of sequential tasks. The core challenge here is catastrophic forgetting: as a neural network learns Task B, it often overwrites the weights necessary to perform Task A. Current state-of-the-art methods use Experience Replay (ER), where a small "episodic memory" (a buffer) of past examples is stored and interleaved with new data. However, these buffers are often tiny (e.g., 1-5 examples per class). Because the buffer is so sparse, the model’s performance becomes highly sensitive to which specific samples were saved, and the model can still "drift" away from the correct decision boundaries for past tasks.

Methodology

The authors propose a new objective called anchoring, implemented through an algorithm called Hindsight Anchor Learning (HAL). The approach consists of two main components:

The Anchoring Objective (Bilevel Optimization)

Instead of a standard gradient update, HAL uses a two-step process. First, it calculates a "temporary" update based on the current data and the memory buffer. Then, it performs the actual update by minimizing the current loss while ensuring the model’s predictions on specific "anchor points" remain unchanged. This is a form of nested optimization that forces the model to be "conservative" regarding its most critical past knowledge.

Learning Anchors in Hindsight

The "anchors" are synthetic points (one per class, per task) optimized to be at the "front lines" of forgetting. Since future tasks are unknown, HAL simulates the future by fine-tuning the current model on the existing episodic memory. It then uses gradient ascent in the image space to find points that would be most forgotten during that fine-tuning. To ensure these synthetic anchors stay realistic, a "mean embedding loss" keeps them near the average feature representation of the actual data.

Experiments and Results

HAL was evaluated on four standard supervised learning benchmarks: Permuted MNIST, Rotated MNIST, Split CIFAR, and Split miniImageNet. HAL consistently outperformed a wide array of baselines, including EWC, GEM, and standard Experience Replay.

Performance comparison of HAL against various baselines across four benchmarks: Permuted MNIST, Rotated MNIST, Split CIFAR, and Split miniImageNet.

Key results include:

  • Accuracy Boost: On Split CIFAR, HAL achieved ~60.4% accuracy compared to ~56.2% for the next best memory-based method (ER-Ring).
  • Reduced Forgetting: HAL reduced the "forgetting metric" by nearly 23% compared to standard experience replay baselines.
  • Efficiency with Tiny Memory: Performance gains were most notable when memory was extremely restricted (e.g., only 1 sample per class).

Key Insights

A critical finding of this work is that forgetting happens at the edges of the decision space. Through t-SNE visualizations, the authors show that learned anchors naturally migrate toward the decision boundaries of the classifier. Points near the boundary are the easiest to "flip" (and thus forget) when weights shift. By anchoring these specific points, the model protects the most fragile parts of its knowledge.

t-SNE visualization of the feature space showing that learned anchors (e_i) migrate toward the decision boundaries between classes.

Surprisingly, HAL performs better using one synthetic anchor per class than it does if you simply double the size of the memory buffer with real data. This suggests that where you protect the decision space is more important than how much data you store.

Conclusion

HAL introduces a powerful way to anchor past knowledge using synthetic points optimized in hindsight. While it involves some computational overhead due to anchor construction and nested optimization, it is substantially faster than other meta-learning approaches like MER. Future work includes adapting HAL to "task-free" settings where data distributions change smoothly without explicit task markers.

Summary updated

Abstract

In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a new objective that we call anchoring, where the learner uses bilevel optimization to update its knowledge on the current task, while keeping intact the predictions on some anchor points of past tasks. These anchor points are learned using gradient-based optimization to maximize forgetting, which is approximated by fine-tuning the currently trained model on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.
BibTeX
@inproceedings{chaudhry2021using,
  author = {Arslan Chaudhry and Albert Gordo and Puneet K. Dokania and Philip H. S. Torr and David Lopez-Paz},
  title = {{Using Hindsight to Anchor Past Knowledge in Continual Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=tYfaMCDO3bB},
}
Figure