Abstract
Few-Shot Class-Incremental Learning (FSCIL) requires models to learn new classes from sparse data while maintaining performance on previously learned base classes. A primary failure mode in FSCIL is the "crowding" of the feature space, leading to catastrophic forgetting as new class embeddings overlap with old ones. This paper introduces Expanding Hyperspherical Space (EHS), a framework that utilizes the full geometric range of the hypersphere by adopting symmetric activations and explicitly partitioning the space to reserve room for future, unseen classes.
The Geometric Bottleneck of ReLU
Most modern computer vision models utilize the ReLU activation function. However, because ReLU outputs are strictly non-negative, the resulting feature vectors are confined to the positive orthant of the embedding space. In a high-dimensional space of dimension $d$, this region represents only $1/2^d$ of the total available hypersphere. This mathematical constraint creates a significant bottleneck: as incremental classes are added, they are forced into an increasingly congested sub-region, causing confusion between old and new categories.
The EHS Framework
The proposed EHS framework addresses this overcrowding through three primary mechanisms:
- Odd-Symmetric Activation: By replacing ReLU with
tanh, the model generates both positive and negative feature values, effectively expanding the usable embedding space to the entire hypersphere. - Space Partitioning: The hypersphere is bisected into two halves using a defined "base vector." The upper hemisphere is allocated for base classes, while the lower hemisphere is strictly reserved for future incremental classes.
- Proactive Reservation: Unlike reactive methods that adjust after new data arrives, EHS uses a dual regularization loss ($\\mathcal{L}_B$ and $\\mathcal{L}_P$) during initial training. $\\mathcal{L}_B$ constrains base classes to the upper hemisphere, while $\\mathcal{L}_P$ uses synthetic "pseudo-instances" to occupy the lower hemisphere, preparing the model for future data.
Experimental Evaluation
The researchers evaluated EHS on the CIFAR-100 and miniImageNet benchmarks. The primary metric was the Performance Dropping (PD) rate, which measures the decline in accuracy from the initial session to the final incremental session.
- CIFAR-100: EHS achieved a PD of 21.84%, outperforming previous state-of-the-art methods like FACT (22.50%) and SAVC (25.68%).
- miniImageNet: EHS recorded a PD of 21.76%, surpassing FACT (22.07%) and ALICE (24.90%).
Ablation studies highlighted that the transition to tanh alone accounted for a significant portion of the performance gain, validating the importance of geometric expansion in mitigating catastrophic forgetting.
Conclusion
Expanding Hyperspherical Space (EHS) offers a novel geometric perspective on incremental learning. By recognizing and resolving the spatial limitations of standard activation functions, EHS provides a robust foundation for models to grow their knowledge base without the typical "crowding" effects that lead to forgetting. Future research may explore hybrid activations to balance the geometric benefits of tanh with the optimization stability of ReLU.
Summary updated