TL;DR
pFedDC is a novel personalized federated learning framework that addresses severe data heterogeneity by optimizing both vision and text prompts simultaneously. By decoupling prompts into global consensus and local client-specific traits and adaptively combining them via cross-attention, it achieves state-of-the-art accuracy on complex tasks featuring concurrent label and domain shifts.
What Problem Does This Paper Address?
Federated Learning (FL) allows multiple decentralized clients to collaboratively train a model without sharing their raw, private data. However, real-world deployments suffer from severe data heterogeneity, primarily in two forms:
- Label Shift: Different clients collect entirely different distributions of classes (e.g., Client A mostly has pictures of cats, while Client B mostly has dogs).
- Domain Shift: Even for the same class, the visual style varies drastically across clients (e.g., one client has photos, another has sketches, and a third has oil paintings).
Recently, researchers have turned to pre-trained Vision-Language Models (VLMs) like CLIP, using parameter-efficient prompt tuning to adapt models locally without massive communication overheads. However, existing federated prompt-learning methods focus almost exclusively on tuning text prompts, leaving the vision side fixed. This creates a bottleneck when clients face simultaneous label and domain shifts, as text prompts alone cannot adequately reconcile severe visual misalignments across different local domains.
Methods
To bridge this gap, the authors introduce pFedDC (Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion). The framework introduces three core innovations:
- Dual-Prompt Learning: Instead of only tuning text, pFedDC introduces learnable prompt tokens into both the text encoder and the image encoder of a frozen CLIP model. Text prompts capture category-level semantics, while vision prompts handle domain-specific feature adaptations.
- Global-Local Prompt Decoupling: Each client maintains two sets of dual prompts:
- Global Prompts: Shared with the central server and aggregated via standard FedAvg to capture cross-client consensus.
- Local Prompts: Kept strictly on the client device to preserve unique local domain and label characteristics.
- Cross-Attention Fusion Module: Rather than simply adding or concatenating these prompts, pFedDC employs local text and vision cross-attention modules. These modules dynamically query and fuse global and local prompts, generating personalized representations tailored to each client's specific data distribution.
Datasets
The authors rigorously evaluated pFedDC across nine public benchmark datasets categorized by their type of heterogeneity:
- Label Shift Benchmarks: Caltech101, Flowers102, OxfordPets, DTD, CUB, UCF101, and EuroSAT (partitioned using Dirichlet distributions with $\beta=0.1$ and $\beta=0.05$ to simulate severe class imbalances).
- Domain Shift & Joint Shift Benchmarks:
- DomainNet: Features 6 distinct domains (Clipart, Infograph, Painting, Quickdraw, Real, Sketch).
- Office-Caltech10: Features 4 domains (Amazon, Caltech, DSLR, Webcam).
Key Insights and Ideas
- Tuning Text Alone is Suboptimal: The paper demonstrates that keeping the image encoder fixed during federated prompt tuning severely limits the model's ability to handle domain shifts. Visual feature alignment is crucial when the underlying data distributions look fundamentally different.
- Decomposition Prevents Drift: Separating the optimization into global and local prompt spaces ensures that local personalization doesn't overwrite or corrupt the shared global knowledge base, mitigating client divergence.
- Adaptive Fusion Matters: Static aggregation methods fail to balance global consensus and local nuances effectively. Utilizing a cross-attention mechanism allows the model to dynamically decide how much global context versus local domain knowledge is required for a given sample.
Key Results
pFedDC consistently outperformed state-of-the-art federated prompt learning methods (such as PromptFL and FedOTP) across all settings:
- Label Shift: Under severe label skew ($\beta=0.1$), pFedDC achieved an average accuracy of 94.43% across seven datasets, outperforming the closest competitor, FedOTP (93.06%).
- Joint Label & Domain Shift: In the most challenging scenarios where both shifts occur simultaneously, pFedDC demonstrated substantial improvements:
- DomainNet: Reached an average accuracy of 47.80% (outperforming FedOTP's 46.49%).
- Office-Caltech10: Reached 37.33% (outperforming FedOTP's 34.13%).
- Ablation Studies: Disabling either the text or vision cross-attention modules resulted in a noticeable performance drop, confirming that multi-modality prompt fusion is key to handling complex data heterogeneity.
Limitations and Future Work
While pFedDC delivers strong empirical performance, the authors note that a rigorous theoretical analysis of the framework is still missing. Future work will focus on analyzing the convergence bounds of this dual-prompt cross-attention architecture, as well as exploring its implications on data privacy, fairness, and robustness against adversarial clients.
Summary updated