Asynchronous Federated Continual Learning

Donald Shenaj, Marco Toldo, Alberto Rigon, Pietro Zanuttigh

Abstract

The standard class-incremental continual learning setting assumes a set of tasks seen one after the other in a fixed and pre-defined order. This is not very realistic in federated learning environments where each client works independently in an asynchronous manner getting data for the different tasks in time-frames and orders totally uncorrelated with the other ones. We introduce a novel federated learning setting (AFCL) where the continual learning of multiple tasks happens at each client with different orderings and in asynchronous time slots. We tackle this novel task using prototype-based learning, a representation loss, fractal pre-training, and a modified aggregation policy. Our approach, called FedSpace, effectively tackles this task as shown by the results on the CIFAR-100 dataset using 3 different federated splits with 50, 100, and 500 clients, respectively.

1. Introduction

Standard Federated Learning (FL) typically assumes a static environment where clients possess a fixed dataset and participate in synchronized training rounds. Conversely, Continual Learning (CL) focuses on a single agent learning a sequence of tasks over time. However, real-world applications—such as autonomous vehicle fleets or smartphone networks—operate in a dynamic, asynchronous environment.

In this work, we define Asynchronous Federated Continual Learning (AFCL). In AFCL, decentralized clients encounter new tasks on their own schedules and in potentially different orders. This setting introduces a "double whammy" of challenges: catastrophic forgetting (losing knowledge of previous tasks) and client drift (divergent model updates due to heterogeneous task progression).

Overview of the Asynchronous Federated Continual Learning (AFCL) setting, where diverse clients encounter local task streams asynchronously.

2. The FedSpace Framework

To address the complexities of AFCL, we propose FedSpace (Federated learning System with Prototype Aggregation for Continual rEpresentation). FedSpace is built upon four primary components designed to stabilize learning and preserve knowledge across the federation.

2.1 Server Fractal Pre-training

To establish a robust initial feature space without compromising privacy or introducing data bias, the server is pre-trained on mathematically generated fractal images. This provides the model with a foundational understanding of shapes and textures before any real-world client data is processed, avoiding early bias toward specific objects.

2.2 Prototype Aggregation and Augmentation

Rather than storing raw images—which is memory-intensive and poses privacy risks—FedSpace utilizes "prototypes." A prototype is the mean feature vector for a specific class. Clients share these prototypes with the server, which aggregates them into global prototypes. During local training, clients use these global prototypes, augmented with Gaussian noise, to "re-imagine" old classes and prevent forgetting. This approach is significantly more privacy-preserving and memory-efficient than saving exemplar images.

2.3 Contrastive Representation Loss

FedSpace incorporates a contrastive representation loss ($L_r$) to ensure that feature vectors of the same class cluster together while distinct classes are pushed apart. This ensures that internal representations remain organized even as new tasks are added to the stream.

2.4 Modified Server Aggregation

Standard aggregation methods like FedAvg can be unstable in AFCL due to highly divergent client updates. FedSpace employs a modified aggregation policy using a weighted moving average ($\rho=0.5$) between the current aggregate and the previous global model. This "momentum" at the server level acts as a buffer against the noise of asynchronous task progression.

The FedSpace architecture, illustrating local training with representation loss, prototype aggregation, and server-side model aggregation.

3. Experimental Evaluation

We evaluated FedSpace on the CIFAR-100 dataset, partitioned into three federated splits with 50, 100, and 500 clients. Each client was assigned 10 tasks, with each task containing 10 classes. To simulate real-world heterogeneity, data was distributed using a Dirichlet distribution ($\alpha=3$), creating a non-IID environment where clients had unbalanced and diverse data samples.

4. Results and Discussion

Our experiments demonstrate that traditional FL and CL methods struggle significantly in the AFCL setting.

  • FedAvg (Baseline): Failed to achieve meaningful learning, with accuracy stagnating around 2-3% due to severe forgetting and misaligned updates.
  • PASS + FedAvg: A state-of-the-art CL method adapted to FL reached ~29% accuracy with 50 clients but saw performance drop to 13.39% as the number of clients scaled to 500.
  • FedSpace (Ours): Achieved 37.18% accuracy with 50 clients and maintained a much stronger 28.42% with 500 clients.

FedSpace shows superior scalability and robustness. While the absolute accuracy reflects the extreme difficulty of the AFCL task, FedSpace more than doubles the performance of its closest competitor in the 500-client scenario.

5. Conclusion

FedSpace introduces a viable path forward for learning in decentralized, asynchronous, and non-stationary environments. By combining fractal pre-training, prototype-based memory, and structured representation learning, it effectively bridges the gap between Federated and Continual Learning. Future work will explore more advanced server-side aggregation strategies and the application of the framework to even larger-scale or more complex datasets beyond CIFAR-100.

Summary updated

Abstract

The standard class-incremental continual learning setting assumes a set of tasks seen one after the other in a fixed and pre-defined order. This is not very realistic in federated learning environments where each client works independently in an asynchronous manner getting data for the different tasks in time-frames and orders totally uncorrelated with the other ones. We introduce a novel federated learning setting (AFCL) where the continual learning of multiple tasks happens at each client with different orderings and in asynchronous time slots. We tackle this novel task using prototype-based learning, a representation loss, fractal pre-training, and a modified aggregation policy. Our approach, called FedSpace, effectively tackles this task as shown by the results on the CIFAR-100 dataset using 3 different federated splits with 50, 100, and 500 clients, respectively. The code and federated splits are available at https://github.com/LTTM/FedSpace.
BibTeX
@misc{shenaj2023asynchronous,
  author = {Donald Shenaj and Marco Toldo and Alberto Rigon and Pietro Zanuttigh},
  title = {{Asynchronous Federated Continual Learning}},
  year = {2023},
  eprint = {2304.03626},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.03626},
}
Figure