TL;DR
Many personalized federated learning (PFL) algorithms quietly fail a basic sanity check: they perform worse than isolated local training without any collaboration. To fix this, FEDMOSAIC introduces an adaptive federated co-training framework where clients share hard predictions and confidence scores over a shared unlabeled public dataset rather than exchanging model weights. By deciding when to trust the global consensus and whose expertise to trust per example, FEDMOSAIC consistently outperforms both local training and standard PFL baselines across extreme non-IID conditions.
What Problem Does This Paper Address?
Federated learning (FL) trains machine learning models across distributed clients without centralizing private data. However, real-world data across clients is rarely identically distributed (non-IID). When clients hold drastically different data distributions (e.g., label skews or domain shifts), classical federated methods like FedAvg suffer from "client drift" and perform poorly.
Personalized Federated Learning (PFL) aims to address this by training customized models for each client. But the authors highlight a troubling trend: in many heterogeneous settings, popular PFL methods perform worse than standalone local training (training purely on local data with zero communication). Parameter averaging often dilutes local specialization with irrelevant or conflicting updates. The core challenge is building a collaboration mechanism that extracts useful knowledge from peers on an example-by-example basis without corrupting specialized local models.
Methods
Rather than averaging model parameters or soft probabilities, FEDMOSAIC adopts federated co-training over a small, shared, unlabeled public dataset $U$. Collaboration is governed by two complementary mechanisms:
-
Dynamic Loss Weighting (Deciding When to Trust):
Each client optimizes a joint loss $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{local}} + \lambda_i^t \mathcal{L}_{\text{pseudo}}$, where the adaptive weight $\lambda_i^t$ depends on the discrepancy between local loss and public pseudo-label loss: $$\lambda_i^t = \exp\left(-\frac{\mathcal{L}_{\text{pseudo}} - \mathcal{L}_{\text{local}}}{\mathcal{L}_{\text{local}}}\right)$$ If consensus pseudo-labels conflict with local knowledge ($\mathcal{L}_{\text{pseudo}} \gg \mathcal{L}_{\text{local}}$), $\lambda_i^t \to 0$, providing an automatic fail-safe that defaults back to local training. If the consensus aligns or provides clean signal, $\lambda_i^t \ge 1$. -
Confidence-Based Aggregation (Deciding Whose Predictions to Trust):
Instead of uniform majority voting, each client submits both a hard prediction and an expertise/confidence vector $E_i^t$ over $U$. The server aggregates predictions by weighting each client's vote by its confidence. The authors evaluate two confidence heuristics: -
Class-frequency-based (FEDMOSAIC-W): Higher weight if the client has seen more local instances of that predicted class.
- Uncertainty-based (FEDMOSAIC-U): Derived from the predictive entropy of the client's model.
Theoretical & Privacy Guarantees:
The authors prove that FEDMOSAIC converges to an approximate stationary point at an $\mathcal{O}(1/T)$ rate under standard smoothness, bounded variance, and bounded drift assumptions. Client-level differential privacy (DP) is supported via an XOR mechanism on discrete predictions and a Gaussian mechanism on expertise scores.
Datasets
The authors benchmark across three heterogeneity regimes:
- Label Skew: Fashion-MNIST and CIFAR-10 under both "pathological" non-IID (2 classes per client) and "practical" Dirichlet label partitions.
- Feature Shift: Office-10 (4 domains: Amazon, Caltech, DSLR, Webcam) and DomainNet (top 10 categories across 6 domains: Clipart, Infograph, Painting, Quickdraw, Real, Sketch).
- Hybrid (Label Skew + Feature Shift): Office-10 (20 clients) and DomainNet (30 clients) split across domains with only 2 classes per client.
- Larger Benchmark: Tiny-ImageNet (200 classes) evaluated with a ResNet-18 backbone under practical Dirichlet label skew.
Key Insights and Ideas
- Local Training Must Be the Ground Truth Baseline: If a federated model cannot beat local training, collaboration is actively detrimental. PFL methods must have an automatic opt-out when peer data is harmful or uninformative.
- Granular, Example-Level Trust Beats Parameter Blending: Sharing predictions on public data allows clients to contribute strictly within their domain of expertise (e.g., an expert on cats won't corrupt consensus on cars).
- Simplicity Wins: In extreme label-skew settings, a simple class-frequency counter serves as an expertise proxy that performs just as well as predictive entropy.
- Minimal Public Data Needed: Ablations show that reducing the public dataset size by over 90% (from 3,000 down to 250 samples on CIFAR-10) results in negligible accuracy drops.
- High Uplink Efficiency: Communicating class indices and scalar confidence scores reduces per-round communication by up to $\approx 177\times$ compared to transferring multi-megabyte neural network weights.
Key Results
- Hybrid Heterogeneity (Label Skew + Feature Shift): On Office-10, FEDMOSAIC achieves 89.43% accuracy, outperforming local training (86.79%), FedAvg (37.25%), and the strongest PFL baseline, Ditto (80.63%). On DomainNet, it reaches 88.36% vs. local training's 84.64% and Ditto's 81.47%.
- Tiny-ImageNet: Under practical label skew with ResNet-18, FEDMOSAIC reaches 41.90%, surpassing local training (36.75%), FedAvg (19.80%), and all PFL baselines (best was FedPHP at 35.63%), closely approaching centralized upper bounds (42.20%).
- Efficiency: On Fashion-MNIST, FEDMOSAIC achieved 75% target test accuracy in 28.8 minutes compared to 53.6 minutes for FedAvg.
Limitations and Future Work
- Requirement of Public Data: The approach assumes access to an unlabeled public dataset $U$ roughly aligned with the target problem space, which may not always exist in highly sensitive or niche domains.
- Modest Margins in Pure Pathological Skew: When local client data is almost completely disjoint, the margin of improvement over local training is relatively narrow (e.g., 88.03% vs. 88.01% on CIFAR-10), indicating that collaborative gain is inherently limited in such regimes.
- Future Directions: The authors suggest exploring generative methods to synthesize public datasets privacy-preservingly or leveraging foundation models to distill public representations.
Summary updated