DOLFIN: Balancing Stability and Plasticity in Federated Continual Learning

Omayma Moussadek, Riccardo Salami, Simone Calderara

Abstract

Federated Continual Learning (FCL) presents a unique challenge in modern machine learning: models must learn from a sequential stream of decentralized data without succumbing to catastrophic forgetting or violating user privacy. This paper introduces DOLFIN (Distributed Online LoRA for Federated INcremental learning), a novel framework that leverages orthogonal low-rank adapters (LoRA) within Vision Transformers (ViT). By mathematically constraining model updates to subspaces orthogonal to previous tasks, DOLFIN achieves state-of-the-art performance across multiple benchmarks without the need for privacy-compromising data rehearsal buffers.

Introduction

The intersection of Continual Learning (CL) and Federated Learning (FL) creates a complex optimization landscape. In CL, models must maintain a balance between plasticity (the ability to acquire new knowledge) and stability (the ability to retain past knowledge). In FL, this must be achieved across a decentralized network where data cannot be centralized due to privacy constraints.

Traditional CL methods often rely on rehearsal buffers—storing a subset of old data to replay during new tasks. However, in a federated setting, sharing or storing raw client data is often a violation of privacy protocols. Furthermore, the communication overhead of transmitting full model updates across a network is frequently prohibitive. DOLFIN addresses these issues by introducing a parameter-efficient, buffer-free approach to FCL.

Methodology

DOLFIN utilizes a frozen Vision Transformer (ViT) backbone and introduces a specialized implementation of Parameter-Efficient Fine-Tuning (PEFT). The architecture is built upon two primary pillars:

1. Task-Specific LoRA Modules

Instead of fine-tuning the entire backbone, DOLFIN injects low-rank matrices ($A$ and $B$) into the key and value projections of the ViT. This significantly reduces the number of trainable parameters and the resulting communication overhead between clients and the central server.

2. Orthogonal Subspace Updates via DualGPM

To prevent catastrophic forgetting, DOLFIN ensures that updates for a new task do not interfere with the representations learned for previous tasks. This is achieved through Dual Gradient Projection Memory (DualGPM).

For each new task $t$, the server broadcasts a frozen matrix $A_t$ that defines a "safe" orthogonal direction for updates. Clients then train a corresponding matrix $B_t$ locally on their private data. By forcing the trainable LoRA matrix to project through this frozen, orthogonal basis, the model provides a structural guarantee against interference with prior knowledge. After local training, clients compute the next orthogonal basis ($A_{t+1}$) and send the updates back to the server for aggregation.

Experimental Evaluation

Datasets and Heterogeneity

The authors evaluated DOLFIN on four standard image classification benchmarks, each split into 10 incremental tasks:

  • CIFAR-100
  • ImageNet-R
  • ImageNet-A
  • CUB-200

To simulate realistic federated environments, data was distributed across 10 clients using a Dirichlet distribution (parameterized by $\beta$). Lower values of $\beta$ (e.g., 0.1) represent high data heterogeneity (non-IID), where class distributions vary significantly across different clients.

Baselines

DOLFIN was compared against six prominent baselines, including regularization-based methods (EWC, LwF), prompt-based methods (L2P, CODA-Prompt), and other PEFT approaches (PILoRA).

Results and Discussion

The primary metric for evaluation was Final Average Accuracy (FAA), representing the mean accuracy across all tasks after the full incremental sequence was completed.

  • Superior Accuracy: DOLFIN achieved state-of-the-art results on the majority of benchmarks. On CIFAR-100 ($\beta=1.0$), it reached 86.58%, significantly outperforming PILoRA (75.75%). On the challenging ImageNet-A dataset, DOLFIN showed a massive improvement, reaching 35.75% compared to the next best baseline, L2P, at 20.14%.
  • Robustness to Heterogeneity: The method proved highly resilient to non-IID data distributions, maintaining high performance even as the Dirichlet parameter $\beta$ decreased.
  • Privacy and Efficiency: Unlike many high-performing CL methods, DOLFIN requires no rehearsal buffers. The "memory" of past tasks is encoded geometrically within the orthogonal constraints of the $A$ matrices, preserving client privacy while minimizing the communication footprint.

Conclusion

DOLFIN demonstrates that orthogonal low-rank adapters provide a scalable and effective solution for Federated Continual Learning. By decoupling plasticity and stability through task-specific subspaces, the framework achieves high accuracy without the privacy risks associated with data rehearsal. While the local computation of Singular Value Decomposition (SVD) for orthogonal projections may present a minor bottleneck for resource-constrained edge devices, DOLFIN represents a significant step forward in building robust, private, and incremental decentralized models.

Summary updated

BibTeX
@inproceedings{moussadek2025dolfin,
  author = {Omayma Moussadek and Riccardo Salami and Simone Calderara},
  title = {{DOLFIN: Balancing Stability and Plasticity in Federated Continual Learning}},
  booktitle = {ICIAP (Workshops 2)},
  year = {2025},
  doi = {10.1007/978-3-032-11381-8\_15},
}
Figure