TL;DR
This paper introduces Focal Neural Collapse Contrastive (FNC2) and Hardness-Softness Distillation (HSD) to address catastrophic forgetting in continual learning. Their method effectively balances "soft" (inter-sample) and "hard" (sample-prototype) relationships, achieving state-of-the-art results, particularly in memory-free settings, making it ideal for data-privacy-sensitive applications.
What Problem Does This Paper Address?
Continual learning (CL) aims to enable models to learn sequentially from new data without forgetting previously acquired knowledge. A major challenge in CL is "catastrophic forgetting," where learning new tasks overwrites knowledge from older ones. Existing methods often rely on memory buffers to store past data, which raises privacy concerns. Furthermore, many contrastive learning approaches focus on "soft" relationships between samples, which can lead to representation overlap and forgetting. While "hard" relationships, inspired by Neural Collapse (NC), can improve class separation, they might reduce intra-class variability and still cause forgetting by pulling old representations towards new ones. This paper tackles the need for memory-efficient CL that effectively balances both soft and hard relationships to mitigate forgetting.
Methods
The proposed method, called Focal Neural Collapse Contrastive (FNC2), is a novel representation learning loss that combines two key ideas:
- Focal Neural Collapse Contrastive (FNC2) for Plasticity: This loss function aims to improve the model's ability to learn new tasks. It integrates both "soft" (inter-sample) and "hard" (sample-prototype) relationships. The "soft" component uses a supervised contrastive loss (similar to SupCon) to pull positive samples together and push negative samples apart. The "hard" component leverages fixed, equidistant prototypes (derived from Neural Collapse principles) to align sample representations with their corresponding class centers. A focal loss-inspired mechanism is used to emphasize "hard" samples (those that are positive but far from the anchor or prototype), allowing for more robust learning and better class separation. This approach implicitly incorporates cross-task consolidation by using old prototypes as negative points, acting as a form of "pseudo-replay."
- Hardness-Softness Distillation (HSD) for Stability: This loss function focuses on preserving knowledge from previous tasks. It combines two distillation strategies:
- LIRD (Soft Stability): This is a standard instance-wise relation distillation loss that preserves inter-sample relationships.
- LS-PRD (Hard Stability): This novel component preserves sample-prototype relationships, directly leveraging the NC concept.
HSD uses a weighted combination of LIRD and LS-PRD, with weights that dynamically shift over training. Initially, LIRD is more emphasized to support plasticity, and as training progresses, LS-PRD becomes more dominant to enhance stability and knowledge preservation. This dual approach aims to effectively balance learning new information with retaining old knowledge.
Datasets
The paper evaluates its method on three standard CL benchmarks:
- Seq-Cifar-10
- Seq-Cifar-100
- Seq-Tiny-ImageNet
These datasets are split into multiple tasks for both class-incremental (Class-IL) and task-incremental (Task-IL) learning settings.
Key Insights and Ideas
- The critical role of both "soft" and "hard" relationships: The paper argues that effectively balancing inter-sample (soft) and sample-prototype (hard) relationships is crucial for robust continual learning, especially in the context of Neural Collapse.
- Neural Collapse as a foundation for memory-efficient CL: By leveraging the structured feature representations and fixed prototypes associated with Neural Collapse, the authors demonstrate a path towards reducing reliance on memory buffers.
- Pseudo-replay with prototypes: Using fixed prototypes from previous tasks as negative points in the current task's learning process acts as an effective form of "pseudo-replay," allowing for knowledge consolidation without storing actual past samples.
- Dynamic distillation for stability: The Hardness-Softness Distillation (HSD) loss, with its time-varying weights, effectively balances the preservation of both soft and hard knowledge over time, leading to improved stability.
- Memory-free performance is key: The paper highlights that their method excels in memory-free settings, which is a significant advantage for applications with strict data privacy requirements.
Key Results
- State-of-the-art performance: The proposed method achieves state-of-the-art results across various CL settings, datasets, and buffer sizes.
- Superior memory-free performance: In memory-free scenarios, the method significantly outperforms existing replay-based methods with memory buffers of size 200. For instance, on Seq-Cifar-10, it shows improvements of approximately 10% in Class-IL and 7.76% in Task-IL compared to Co²L.
- Competitive with limited memory: Even with limited memory (buffer size 200), the method is competitive and often surpasses other baselines.
- Effective prototype utilization: Ablation studies confirm the effectiveness of using pseudo-replay prototypes, showing performance gains comparable to using a buffer of 200 without them.
Limitations and Future Work
- Predefined prototypes: The current approach requires predefining prototypes, which can be impractical if the number of classes is unknown.
- Future directions: The authors suggest exploring methods to pre-define a maximum number of prototypes and learn their distributions, allowing for more flexible associations. They also aim to investigate alternative memory-free evaluation methods and focus on distilling only the most crucial knowledge for easily forgotten samples.
Summary updated