TL;DR
This paper introduces Backward Feature Projection (BFP), a novel method for continual learning that combats catastrophic forgetting by preserving the linear separability of previously learned classes in the feature space. BFP achieves a better balance between learning new information (plasticity) and retaining old knowledge (stability) by allowing new features to emerge while ensuring old class distinctions remain clear.
What Problem Does This Paper Address?
Deep neural networks are notoriously prone to "catastrophic forgetting" when trained sequentially on new tasks without access to old data. This means their performance on previously learned tasks degrades significantly. Continual learning (CL) aims to address this, but a key challenge is the "stability-plasticity dilemma": how to learn new things without forgetting old ones. Existing methods, particularly those using feature distillation, often force new features to be too similar to old ones, hindering the model's ability to adapt to new tasks. This paper argues that forgetting is often linked to the loss of linear separability in the feature space, where distinct classes become mixed.
Methods
The core contribution is the Backward Feature Projection (BFP) loss. BFP works by learning a linear transformation matrix, denoted by $A$, that projects the current task's features ($z$) back to match the features from the previous task's checkpoint ($z'$). The loss function is $L_{BFP} = ||Az - z'||^2$. This approach doesn't force exact feature equality but rather preserves the linear separability of old classes. This means that if two classes were separable by a hyperplane in the old feature space, they remain separable in the new space, albeit potentially along a transformed direction ($A^T w$). This allows new classes to be learned along "unused" feature directions, promoting plasticity. BFP is designed to be integrated with existing experience replay methods, acting as an additional regularization term.
Datasets
The paper evaluates its method on several standard continual learning benchmarks:
- Split CIFAR-10: Divided into 5 tasks, each with 2 classes.
- Split CIFAR-100: Divided into 10 tasks, each with 10 classes.
- Split TinyImageNet: Divided into 10 tasks, each with 20 classes.
Key Insights and Ideas
- Linear Separability is Key: The paper posits that catastrophic forgetting is largely due to the loss of linear separability in the feature space. When old classes become mixed, performance degrades.
- Feature Space Evolution: As models learn new tasks, they need to utilize more dimensions in the feature space to maintain class separability. However, a significant portion of these dimensions might be "unused" by older tasks, offering room for new learning.
- The Power of Linear Transformation: A simple linear transformation ($A$) is sufficient to preserve linear separability while allowing for plasticity. This is a more flexible approach than forcing exact feature matching.
- BFP as a Stability-Plasticity Bridge: BFP effectively balances stability (preserving old class distinctions) and plasticity (learning new classes) by allowing new features to emerge in directions not occupied by old classes.
Key Results
- Significant Accuracy Boost: When combined with experience replay methods like ER and DER++, BFP consistently improved Final Average Accuracy (FAA) in the challenging Class-IL setting, with improvements up to 6-8% on datasets like Split CIFAR-10 and Split CIFAR-100.
- State-of-the-Art Performance: DER++ with BFP outperformed all other baselines in the Class-IL setting on the tested datasets.
- Improved Feature Space Quality: Linear probing experiments showed that BFP leads to a more linearly separable feature space, where even a simple finetuning baseline (FT w/ BFP) achieved performance comparable to more complex methods like DER++.
- Reduced Forgetting: BFP significantly reduced Final Forgetting (FF), indicating better stability.
- Scalability: The method showed promising results on the larger Split TinyImageNet dataset.
Limitations and Future Work
- Hyperparameter Sensitivity: While the paper states that BFP's performance is not highly sensitive to its main hyperparameter ($\gamma$), further extensive hyperparameter tuning for specific datasets or buffer sizes might yield even better results.
- Computational Overhead: While BFP is integrated with experience replay, it still adds a linear transformation layer and its associated computation during training. The paper mentions that methods like PCA-based pruning can have significant computational overhead, implying BFP is more efficient in comparison.
- Future Work: The paper suggests exploring the effect of different projection layers (linear vs. non-linear) and further investigating the trade-off between plasticity and stability by analyzing the impact of the regularization weight $\gamma$. They also mention interest in online CL settings.
Summary updated