Introduction
Federated Learning (FL) has emerged as a vital paradigm for training machine learning models across decentralized clients—such as mobile devices or medical institutions—without compromising data privacy. However, the inherent diversity in local data distributions, known as data heterogeneity, often prevents a single global model from performing optimally for all participants. Personalized Federated Learning (PFL) addresses this by tailoring models to individual client needs.
Recent advancements in PFL have leveraged Parameter-Efficient Fine-Tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA), to minimize communication overhead. Despite their success, existing PFL methods typically exhibit "short-sightedness," relying solely on the most recent client updates to generate personalized parameters. This approach ignores the rich information embedded in a client's historical learning trajectory, making the personalization process susceptible to noise and local fluctuations.
The pFedSeq Framework
To overcome these limitations, we introduce pFedSeq, a novel framework that treats the sequence of client updates as a time-series problem. By analyzing the historical evolution of client models, pFedSeq identifies consistent learning trends to generate more robust and accurate personalized adapters.
Sequential Modeling with Mamba
At the heart of pFedSeq is a server-side sequential learner based on the Selective State Space Model (SSM), specifically the Mamba architecture. The server maintains a history of updates ($\Delta$) from each client. Instead of simple averaging, these sequences are fed into the Mamba-based learner, which acts as a hypernetwork.
The choice of Mamba is strategic. Traditional sequence models like RNNs or LSTMs often struggle with long-term dependencies or incur high computational costs. Mamba provides log-linear time complexity, allowing the server to process long communication histories efficiently without becoming a bottleneck.
Cross-Client and Cross-Step Dynamics
The sequential learner is designed to capture two critical dimensions of information:
- Cross-Step Relations: It models how a specific client's requirements evolve over time, effectively using historical "momentum" to navigate the optimization landscape.
- Cross-Client Relations: It identifies similarities between different clients, allowing the model to calibrate a specific client's adapter using relevant knowledge gained from others.
The output of this learner is a set of personalized "calibrations." These are small, client-specific adjustments that are added to the globally aggregated adapter, ensuring that each client receives a model that is both globally informed and locally specialized.
Experimental Evaluation
The researchers evaluated pFedSeq against ten state-of-the-art PFL baselines, including pFedHN, FedRep, and PerAda, across four diverse benchmarks:
- CIFAR-100 & Tiny-ImageNet: Benchmarks for label-skew, where clients possess different classes of data.
- DomainNet: A test for feature-skew, involving the same objects depicted in varying styles (e.g., sketches vs. photographs).
- Omniglot: A real-world heterogeneous dataset featuring handwritten characters from 50 different alphabets.
Key Results
pFedSeq demonstrated superior performance across all tested scenarios:
- Accuracy Improvements: The framework achieved an accuracy boost of up to 5.39% over existing methods. The most significant gains were observed in complex heterogeneous environments like DomainNet (+2.21%) and Omniglot (+2.27%).
- Convergence Speed: While pFedSeq requires a brief "warm-up" period (typically 10 rounds) to accumulate sufficient history, its performance accelerates rapidly thereafter. Within 15 rounds following the warm-up, pFedSeq consistently surpassed the accuracy of all baseline methods.
- Robustness: By leveraging historical trajectories, the model proved more resilient to the noise inherent in local training, capturing "consistent trends" that single-round updates often miss.
Limitations and Future Work
Despite its advantages, pFedSeq introduces specific considerations for deployment. The warm-up requirement means that personalization benefits are not immediate; clients initially rely on a standard global model until the sequential learner has enough data to provide meaningful calibrations. Additionally, to manage server-side memory, the framework employs sequence capping, which limits the history length ($L$). While this maintains efficiency, it potentially discards older updates that might still hold value.
Future research may explore the application of this sequential approach to other PEFT methods beyond LoRA and investigate strategies to reduce the warm-up period or optimize long-term memory retention.
Conclusion
pFedSeq represents a significant shift in Personalized Federated Learning by moving from instantaneous updates to historical trajectory analysis. By utilizing the Mamba architecture to model the evolution of client parameters, the framework provides a scalable and effective solution to the challenges of data heterogeneity, setting a new benchmark for personalized adapter tuning.
Summary updated