Federated Few-Shot Class-Incremental Learning

Muhammad Anwar Ma'sum, Mahardhika Pratama, Lin Liu, H Habibullah, Ryszard KowalczykICLR 2025 · OpenReview

TL;DR

This paper introduces UOPP (Unified Optimized Prototype Prompt), a novel framework for the challenging task of Federated Few-Shot Class-Incremental Learning (FFSCIL). By combining task-specific prompts, an adaptive dual-head classifier, and a Neural ODE-based prototype rectification process, UOPP achieves massive performance gains—up to 76% higher accuracy than previous state-of-the-art methods—while maintaining strict data privacy and low communication costs.

What Problem Does This Paper Address?

The authors tackle a "triple threat" machine learning scenario: Federated Few-Shot Class-Incremental Learning (FFSCIL).

  1. Federated Learning (FL): Data stays on local devices (clients) for privacy; only model updates are shared.
  2. Few-Shot Learning (FSL): New classes arrive with only a handful of examples (e.g., 1 or 5 images).
  3. Class-Incremental Learning (CIL): The model must learn new classes over time without "forgetting" the old ones (catastrophic forgetting).

Existing methods fail here because they are "data-hungry." When clients only have a few samples, they suffer from prototype bias—the mathematical average (prototype) of those few samples doesn't accurately represent the true distribution of the class. Furthermore, sharing entire model backbones in a federated setting is computationally expensive and risks leaking private data.

Methods

The Unified Optimized Prototype Prompt (UOPP) model uses several clever architectural choices to balance "stability" (remembering old tasks) and "plasticity" (learning new ones):

UOPP Architecture
  • Task-wise Prompt Learning: Instead of updating the entire model, UOPP freezes a pre-trained Vision Transformer (ViT) backbone. It only trains tiny, task-specific "prompts." This prevents the backbone from drifting (forgetting) and drastically reduces the amount of data sent over the network.
  • Prototype Rectification via Neural ODE: To fix "prototype bias," the authors treat the refinement of a class prototype as a continuous flow problem. They use a Gradient Network (GradNet) and an Ordinary Differential Equation (ODE) solver to iteratively "nudge" the biased local prototypes toward their true population means.
  • Unified Static-Dynamic Prototypes: The model maintains "static" prototypes for old classes and "dynamic" ones for new, incoming classes, ensuring the model can distinguish between everything it has ever seen.
  • Adaptive Dual Heads: The model uses two "heads" for prediction: an MLP-based head (optimized for the data-rich base classes) and a prototype-based head (optimized for the data-scarce few-shot classes). A "head selector" automatically decides which one to use based on the input.

Datasets

The authors evaluated UOPP on three standard computer vision benchmarks, adapted for the federated incremental setting:

  • Split CIFAR100: 100 classes of general objects.
  • Split MiniImageNet: A subset of ImageNet with 100 classes.
  • Split CUB200: A fine-grained dataset of 200 bird species.

In these tests, the "base task" contained 60–100 classes with abundant data, while subsequent "few-shot tasks" introduced 5–10 classes with only 1 or 5 samples per class.

Key Insights and Ideas

  • Freezing is a Feature: By freezing the ViT backbone, the authors essentially eliminate catastrophic forgetting at the feature-extraction level. The "learning" happens entirely in the prompts and the classifier heads.
  • Rectification is Essential: The "aha moment" is that simply averaging local prototypes across clients isn't enough to fix bias. You need an active mechanism (the Neural ODE) to move those prototypes to the correct locations in the embedding space.
Prototype Rectification Process
  • Communication Efficiency: Because UOPP only shares prompts and prototypes (kilobytes) rather than full model weights (hundreds of megabytes), it is significantly more practical for real-world edge devices like smartphones or IoT sensors.

Key Results

UOPP didn't just beat the competition; it redefined the baseline for this sub-field:

  • Accuracy Boost: UOPP outperformed state-of-the-art (SOTA) methods by 5% to 76% in average accuracy across the three datasets.
  • Stability-Plasticity Balance: On the Harmonic Mean (H-mean) metric—which penalizes models that are good at new tasks but forget old ones—UOPP showed improvements of up to 90%.
  • Robustness: The model remained stable even when the number of participating clients fluctuated or when the data was highly "non-i.i.d." (meaning different clients had very different types of images).

Limitations and Future Work

The authors identify several areas for improvement:

  • Vision Only: The current model only uses visual data. Future work could incorporate Multi-Modality (e.g., using text descriptions of classes to help guide the prototypes).
  • Fixed Prompt Size: Every client uses the same prompt size regardless of their local data complexity. An "evolving prompt" that grows or shrinks based on need could be more efficient.
  • Overfitting in Extreme Scenarios: While UOPP handles few-shot data well, it can still suffer from overfitting in the very last tasks of a long sequence. The authors suggest exploring "early stopping" or more aggressive learning rate decay to mitigate this.

Summary updated

Abstract

This study proposes a challenging yet practical Federated Few-Shot Class-Incremental Learning (FFSCIL) problem, where clients only hold very few samples for new classes. We develop a novel Unified Optimized Prototype Prompt (UOPP) model to simultaneously handle catastrophic forgetting, over-fitting, and prototype bias in FFSCIL. UOPP utilizes task-wise prompt learning to mitigate task interference and over-fitting, unified static-dynamic prototypes to achieve a stability-plasticity balance, and adaptive dual heads for enhanced inferences. Dynamic prototypes represent new classes in the current few-shot task and are rectified to deal with prototype bias. Our comprehensive experimental results show that UOPP significantly outperforms state-of-the-art (SOTA) methods on three datasets with improvements up to 76% on average accuracy and 90% on harmonic mean accuracy respectively. Our extensive analysis shows UOPP robustness in various numbers of local clients and global rounds, low communication costs, and moderate running time. The source code of UOPP is publicly available at https://github.com/anwarmaxsum/FFSCIL.
BibTeX
@inproceedings{masum2025federated,
  author = {Muhammad Anwar Ma'sum and Mahardhika Pratama and Lin Liu and H Habibullah and Ryszard Kowalczyk},
  title = {{Federated Few-Shot Class-Incremental Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=ZiPoAlKf9Y},
}
Figure