Abstract
We study continual learning in the large scale setting where tasks in the input sequence are not limited to classification, and the outputs can be of high dimension. Among multiple state-of-the-art methods, we found vanilla experience replay (ER) still very competitive in terms of both performance and scalability, despite its simplicity. However, a degraded performance is observed for ER with small memory. A further visualization of the feature space reveals that the intermediate representation undergoes a distributional drift. While existing methods usually replay only the input-output pairs, we hypothesize that their regularization effect is inadequate for complex deep models and diverse tasks with small replay buffer size. Following this observation, we propose to replay the activation of the intermediate layers in addition to the input-output pairs. Considering that saving raw activation maps can dramatically increase memory and compute cost, we propose the Compressed Activation Replay technique, where compressed representations of layer activation are saved to the replay buffer. We show that this approach can achieve superior regularization effect while adding negligible memory overhead to replay method. Experiments on both the large-scale Taskonomy benchmark with a diverse set of tasks and standard common datasets (Split-CIFAR and Split-miniImageNet) demonstrate the effectiveness of the proposed method.
Introduction
Deep neural networks are notorious for catastrophic forgetting: the tendency to overwrite knowledge of earlier tasks when trained on new ones. While much research has focused on small-scale classification (e.g., MNIST), real-world applications demand scalability and task diversity, including regression, depth estimation, and surface normal prediction. This paper investigates continual learning at scale, identifying that even when output performance is maintained, internal feature representations suffer from "distributional drift."
Methods: Compressed Activation Replay (CAR)
To address internal drift, we propose Compressed Activation Replay (CAR). While vanilla Experience Replay (ER) stores only input-output pairs $(x, y)$, CAR also stores "snapshots" of intermediate layer activations to anchor the model's internal state.
The framework consists of several key components:
- Vanilla Replay: A buffer stores past samples to be mixed into current training batches.
- Activation Matching: The buffer stores compressed feature maps produced when a data point was first learned.
- Compression: To minimize memory overhead, raw activation maps are reduced via average pooling across spatial and channel dimensions into small vectors.
- Loss Function: During training, an activation matching loss penalizes the model if current activations for replayed samples deviate from their stored counterparts, effectively "anchoring" the feature space and preventing drift.
Internal Drift and Results
Visualizing the latent space using t-SNE reveals the core issue: as the model learns new tasks, the features associated with previous tasks physically shift in the latent space. This distributional drift is the primary cause of forgetting in deep networks even when replaying final outputs.
Key Findings
- Scalability of ER: In large-scale settings like the Taskonomy benchmark, simple ER often outperforms complex state-of-the-art methods due to its efficiency and lack of architectural expansion.
- CAR Performance: On Taskonomy, CAR reduced the forgetting rate of vanilla ER by approximately 28% when using a small memory buffer ($m=64$).
- Standard Benchmarks: CAR achieved state-of-the-art results on Split-CIFAR and Split-miniImageNet, surpassing established methods like iCaRL and A-GEM.
- Efficiency: Because activations are heavily compressed, the additional memory required is negligible compared to the storage already needed for raw images in the replay buffer.
Conclusion
Compressed Activation Replay (CAR) provides a powerful, scalable regularization mechanism for continual learning. By stabilizing internal representations, it mitigates forgetting in complex, high-dimensional task sequences without significant memory overhead. Future work will explore the optimal layers for anchoring across emerging architectures like Transformers.
Summary updated