Introduction
Online Task-Free Class Incremental Learning (OTFCIL) is a challenging frontier in machine learning where models must learn from a continuous data stream without explicit task boundaries. The primary obstacle is catastrophic forgetting: as new classes are introduced, the model's feature space typically shifts, causing it to lose previously acquired knowledge. Traditional solutions often rely on large memory buffers to "replay" old samples, which is inefficient and often impractical due to storage or privacy constraints.
The DYSON Paradigm: Compute-and-Align
DYSON (DYnamic feature space Self-OrganizatioN) introduces a novel "compute-and-align" paradigm. Instead of allowing the feature space to drift unpredictably as new data arrives, DYSON pre-computes an optimal geometric structure based on Neural Collapse theory and forces the network to align its features to this stable target.
The framework consists of three core components:
- Dynamic Neural Collapse (DNC): Leveraging the phenomenon where deep networks naturally map classes to symmetric geometric shapes (Simplex Equiangular Tight Frames), DNC mathematically calculates where class prototypes should be. It expands this geometry as new classes arrive while providing a mathematical guarantee on the stability of old prototypes.
- Dynamic Feature-Geometry Alignment (DFGA): DYSON utilizes a frozen, pre-trained backbone (such as ResNet or ViT) and trains only a small projection layer to map these features onto the fixed, optimal coordinates defined by DNC.
- Prototype Placeholders: These reserve space in the geometry for future classes, ensuring long-term stability and preventing "jitter" when the model has only seen a few classes.
Experimental Results
DYSON was evaluated on several standard computer vision benchmarks, including CIFAR-10, CIFAR-100, CUB-200, and CoRe50. The model consistently outperformed state-of-the-art (SOTA) methods across various backbones and incremental steps.
Key results include:
- Significant Accuracy Gains: On CIFAR-100 using a ViT backbone, DYSON outperformed the previous best method (DSDM) by a massive 16.6% in accuracy in the 1-step setting.
- Memory-Free Efficiency: DYSON achieves SOTA results with zero stored samples, outperforming many methods that rely on large memory buffers for data replay.
- Backbone Agnostic: The method demonstrated robust performance whether using traditional ResNet50 or modern Vision Transformer (ViT) architectures.
Conclusion and Future Work
DYSON demonstrates that shifting from learning a feature space to computing one can effectively mitigate catastrophic forgetting in online learning scenarios. By pre-reserving slots in an optimal geometric layout, the model maintains stability without the need for raw data replay. Future research may explore adapting the backbone itself through parameter-efficient fine-tuning (PEFT) to further improve performance in domains far removed from the pre-training data.
Summary updated