Abstract
Federated Continual Learning (FCL) presents a unique challenge in modern machine learning: models must learn from a sequential stream of decentralized data without succumbing to catastrophic forgetting or violating user privacy. This paper introduces DOLFIN (Distributed Online LoRA for Federated INcremental learning), a novel framework that leverages orthogonal low-rank adapters (LoRA) within Vision Transformers (ViT). By mathematically constraining model updates to subspaces orthogonal to previous tasks, DOLFIN achieves state-of-the-art performance across multiple benchmarks without the need for privacy-compromising data rehearsal buffers.
Introduction
The intersection of Continual Learning (CL) and Federated Learning (FL) creates a complex optimization landscape. In CL, models must maintain a balance between plasticity (the ability to acquire new knowledge) and stability (the ability to retain past knowledge). In FL, this must be achieved across a decentralized network where data cannot be centralized due to privacy constraints.
Traditional CL methods often rely on rehearsal buffers—storing a subset of old data to replay during new tasks. However, in a federated setting, sharing or storing raw client data is often a violation of privacy protocols. Furthermore, the communication overhead of transmitting full model updates across a network is frequently prohibitive. DOLFIN addresses these issues by introducing a parameter-efficient, buffer-free approach to FCL.
Methodology
DOLFIN utilizes a frozen Vision Transformer (ViT) backbone and introduces a specialized implementation of Parameter-Efficient Fine-Tuning (PEFT). The architecture is built upon two primary pillars:
1. Task-Specific LoRA Modules
Instead of fine-tuning the entire backbone, DOLFIN injects low-rank matrices ($A$ and $B$) into the key and value projections of the ViT. This significantly reduces the number of trainable parameters and the resulting communication overhead between clients and the central server.
2. Orthogonal Subspace Updates via DualGPM
To prevent catastrophic forgetting, DOLFIN ensures that updates for a new task do not interfere with the representations learned for previous tasks. This is achieved through Dual Gradient Projection Memory (DualGPM).
For each new task $t$, the server broadcasts a frozen matrix $A_t$ that defines a "safe" orthogonal direction for updates. Clients then train a corresponding matrix $B_t$ locally on their private data. By forcing the trainable LoRA matrix to project through this frozen, orthogonal basis, the model provides a structural guarantee against interference with prior knowledge. After local training, clients compute the next orthogonal basis ($A_{t+1}$) and send the updates back to the server for aggregation.
Experimental Evaluation
Datasets and Heterogeneity
The authors evaluated DOLFIN on four standard image classification benchmarks, each split into 10 incremental tasks:
- CIFAR-100
- ImageNet-R
- ImageNet-A
- CUB-200
To simulate realistic federated environments, data was distributed across 10 clients using a Dirichlet distribution (parameterized by $\beta$). Lower values of $\beta$ (e.g., 0.1) represent high data heterogeneity (non-IID), where class distributions vary significantly across different clients.
Baselines
DOLFIN was compared against six prominent baselines, including regularization-based methods (EWC, LwF), prompt-based methods (L2P, CODA-Prompt), and other PEFT approaches (PILoRA).
Results and Discussion
The primary metric for evaluation was Final Average Accuracy (FAA), representing the mean accuracy across all tasks after the full incremental sequence was completed.
- Superior Accuracy: DOLFIN achieved state-of-the-art results on the majority of benchmarks. On CIFAR-100 ($\beta=1.0$), it reached 86.58%, significantly outperforming PILoRA (75.75%). On the challenging ImageNet-A dataset, DOLFIN showed a massive improvement, reaching 35.75% compared to the next best baseline, L2P, at 20.14%.
- Robustness to Heterogeneity: The method proved highly resilient to non-IID data distributions, maintaining high performance even as the Dirichlet parameter $\beta$ decreased.
- Privacy and Efficiency: Unlike many high-performing CL methods, DOLFIN requires no rehearsal buffers. The "memory" of past tasks is encoded geometrically within the orthogonal constraints of the $A$ matrices, preserving client privacy while minimizing the communication footprint.
Conclusion
DOLFIN demonstrates that orthogonal low-rank adapters provide a scalable and effective solution for Federated Continual Learning. By decoupling plasticity and stability through task-specific subspaces, the framework achieves high accuracy without the privacy risks associated with data rehearsal. While the local computation of Singular Value Decomposition (SVD) for orthogonal projections may present a minor bottleneck for resource-constrained edge devices, DOLFIN represents a significant step forward in building robust, private, and incremental decentralized models.
Summary updated