Sculpting Margin Penalty: Intra-Task Adapter Merging and Classifier Calibration for Few-Shot Class-Incremental Learning

Liang Bai, Hong Song, Jinfu Li, Yucong Lin, Jingfan Fan, Tianyu Fu, Danni Ai, Deqiang Xiao, Jian YangarXiv 2025 · DOI · arXiv

Few-Shot Class-Incremental Learning (FSCIL) aims to develop models capable of mastering a set of "base" concepts and subsequently acquiring new classes from limited examples without forgetting previous knowledge. However, a fundamental trade-off exists: optimizing for high discriminability among base classes often compromises the model's ability to generalize to novel classes.

The Margin Dilemma in FSCIL

A critical observation in FSCIL is the "margin dilemma." When training on base classes, applying a positive margin penalty (e.g., using ArcFace or similar losses) forces classes further apart in the feature space. While this enhances discriminability for base classes, it "squeezes" the available feature space, leaving little room for new classes to be effectively integrated. Conversely, training without a margin penalty preserves feature space flexibility for generalization but leads to fuzzy, less accurate decision boundaries for the initial classes.

The negative correlation between base-class discriminability and new-class generalization as margin size varies.

As shown in the figure above, there is a clear negative correlation: increasing the margin penalty improves base-class accuracy but significantly degrades performance on new classes.

The SMP Framework

To address this trade-off, the researchers propose Sculpting Margin Penalty (SMP), a two-stage framework that balances discriminability and generalization through parameter-efficient fine-tuning and strategic margin application.

Overview of the SMP framework, featuring Margin-aware Intra-task Adapter Merging (MIAM) and Margin Penalty-based Classifier Calibration (MPCC).

1. Margin-aware Intra-task Adapter Merging (MIAM)

Instead of choosing a single margin strategy, SMP trains two sets of low-rank adapters (LoRA) on the base task:

  • Discriminative Set ($R^d$): Trained with a large margin penalty ($m=0.2$) to maximize base-class separation.
  • Generalization Set ($R^g$): Trained with standard classification loss (no margin) to maintain feature space openness.

These adapters are then merged using Fisher Information Statistics. This adaptive merging "sculpts" a final set of parameters that inherits the strengths of both paths—sharp boundaries for base classes and flexibility for new ones.

2. Margin Penalty-based Classifier Calibration (MPCC)

During incremental stages, the model's decision boundaries can drift due to the data imbalance between base and new classes. SMP calibrates the classifier using:

  • Real few-shot samples for new classes.
  • Synthetic pseudo-features for old classes, generated from stored class statistics.
  • A large margin penalty during this calibration phase to ensure clear separation between new and existing classes.

Experimental Results

The SMP method was evaluated on standard benchmarks including CIFAR100, ImageNet-R, and CUB200. It consistently outperformed state-of-the-art methods across various metrics:

  • Superior Accuracy: SMP achieved significant gains in final accuracy, outperforming the next best methods by 2.0% on CIFAR100 and 1.7% on ImageNet-R.
  • Balanced Performance: On the Harmonic Accuracy (HAcc) metric, which measures the balance between base and new class performance, SMP reached 87.1% on CIFAR100, a notable improvement over previous benchmarks.
  • Resistance to Forgetting: The accuracy decay across incremental sessions was markedly slower, demonstrating the model's stability.

Conclusion

The "Sculpting Margin Penalty" approach demonstrates that the trade-off between discriminability and generalization in FSCIL can be effectively managed. By merging adapters with complementary objectives and employing margin-aware calibration, SMP provides a robust solution for continuous learning from limited data. Future work may extend this "sculpting" philosophy to multi-modal learning environments.

Summary updated

BibTeX
@misc{bai2025sculpting,
  author = {Liang Bai and Hong Song and Jinfu Li and Yucong Lin and Jingfan Fan and Tianyu Fu and Danni Ai and Deqiang Xiao and Jian Yang},
  title = {{Sculpting Margin Penalty: Intra-Task Adapter Merging and Classifier Calibration for Few-Shot Class-Incremental Learning}},
  year = {2025},
  eprint = {2508.05094},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.05094},
}
Figure