Whom to Trust? Adaptive Collaboration in Personalized Federated Learning

Amr Abourayya, Jens Kleesiek, Bharat Rao, Michael KampSubmission to ICLR 2026 · Rejected

TL;DR

Many personalized federated learning (PFL) algorithms quietly fail a basic sanity check: they perform worse than isolated local training without any collaboration. To fix this, FEDMOSAIC introduces an adaptive federated co-training framework where clients share hard predictions and confidence scores over a shared unlabeled public dataset rather than exchanging model weights. By deciding when to trust the global consensus and whose expertise to trust per example, FEDMOSAIC consistently outperforms both local training and standard PFL baselines across extreme non-IID conditions.

What Problem Does This Paper Address?

Federated learning (FL) trains machine learning models across distributed clients without centralizing private data. However, real-world data across clients is rarely identically distributed (non-IID). When clients hold drastically different data distributions (e.g., label skews or domain shifts), classical federated methods like FedAvg suffer from "client drift" and perform poorly.

Personalized Federated Learning (PFL) aims to address this by training customized models for each client. But the authors highlight a troubling trend: in many heterogeneous settings, popular PFL methods perform worse than standalone local training (training purely on local data with zero communication). Parameter averaging often dilutes local specialization with irrelevant or conflicting updates. The core challenge is building a collaboration mechanism that extracts useful knowledge from peers on an example-by-example basis without corrupting specialized local models.

Methods

Rather than averaging model parameters or soft probabilities, FEDMOSAIC adopts federated co-training over a small, shared, unlabeled public dataset $U$. Collaboration is governed by two complementary mechanisms:

  1. Dynamic Loss Weighting (Deciding When to Trust):
    Each client optimizes a joint loss $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{local}} + \lambda_i^t \mathcal{L}_{\text{pseudo}}$, where the adaptive weight $\lambda_i^t$ depends on the discrepancy between local loss and public pseudo-label loss: $$\lambda_i^t = \exp\left(-\frac{\mathcal{L}_{\text{pseudo}} - \mathcal{L}_{\text{local}}}{\mathcal{L}_{\text{local}}}\right)$$ If consensus pseudo-labels conflict with local knowledge ($\mathcal{L}_{\text{pseudo}} \gg \mathcal{L}_{\text{local}}$), $\lambda_i^t \to 0$, providing an automatic fail-safe that defaults back to local training. If the consensus aligns or provides clean signal, $\lambda_i^t \ge 1$.

  2. Confidence-Based Aggregation (Deciding Whose Predictions to Trust):
    Instead of uniform majority voting, each client submits both a hard prediction and an expertise/confidence vector $E_i^t$ over $U$. The server aggregates predictions by weighting each client's vote by its confidence. The authors evaluate two confidence heuristics:

  3. Class-frequency-based (FEDMOSAIC-W): Higher weight if the client has seen more local instances of that predicted class.

  4. Uncertainty-based (FEDMOSAIC-U): Derived from the predictive entropy of the client's model.

Theoretical & Privacy Guarantees:
The authors prove that FEDMOSAIC converges to an approximate stationary point at an $\mathcal{O}(1/T)$ rate under standard smoothness, bounded variance, and bounded drift assumptions. Client-level differential privacy (DP) is supported via an XOR mechanism on discrete predictions and a Gaussian mechanism on expertise scores.

Datasets

The authors benchmark across three heterogeneity regimes:

  • Label Skew: Fashion-MNIST and CIFAR-10 under both "pathological" non-IID (2 classes per client) and "practical" Dirichlet label partitions.
  • Feature Shift: Office-10 (4 domains: Amazon, Caltech, DSLR, Webcam) and DomainNet (top 10 categories across 6 domains: Clipart, Infograph, Painting, Quickdraw, Real, Sketch).
  • Hybrid (Label Skew + Feature Shift): Office-10 (20 clients) and DomainNet (30 clients) split across domains with only 2 classes per client.
  • Larger Benchmark: Tiny-ImageNet (200 classes) evaluated with a ResNet-18 backbone under practical Dirichlet label skew.

Key Insights and Ideas

  • Local Training Must Be the Ground Truth Baseline: If a federated model cannot beat local training, collaboration is actively detrimental. PFL methods must have an automatic opt-out when peer data is harmful or uninformative.
  • Granular, Example-Level Trust Beats Parameter Blending: Sharing predictions on public data allows clients to contribute strictly within their domain of expertise (e.g., an expert on cats won't corrupt consensus on cars).
  • Simplicity Wins: In extreme label-skew settings, a simple class-frequency counter serves as an expertise proxy that performs just as well as predictive entropy.
  • Minimal Public Data Needed: Ablations show that reducing the public dataset size by over 90% (from 3,000 down to 250 samples on CIFAR-10) results in negligible accuracy drops.
  • High Uplink Efficiency: Communicating class indices and scalar confidence scores reduces per-round communication by up to $\approx 177\times$ compared to transferring multi-megabyte neural network weights.

Key Results

  • Hybrid Heterogeneity (Label Skew + Feature Shift): On Office-10, FEDMOSAIC achieves 89.43% accuracy, outperforming local training (86.79%), FedAvg (37.25%), and the strongest PFL baseline, Ditto (80.63%). On DomainNet, it reaches 88.36% vs. local training's 84.64% and Ditto's 81.47%.
  • Tiny-ImageNet: Under practical label skew with ResNet-18, FEDMOSAIC reaches 41.90%, surpassing local training (36.75%), FedAvg (19.80%), and all PFL baselines (best was FedPHP at 35.63%), closely approaching centralized upper bounds (42.20%).
  • Efficiency: On Fashion-MNIST, FEDMOSAIC achieved 75% target test accuracy in 28.8 minutes compared to 53.6 minutes for FedAvg.

Limitations and Future Work

  • Requirement of Public Data: The approach assumes access to an unlabeled public dataset $U$ roughly aligned with the target problem space, which may not always exist in highly sensitive or niche domains.
  • Modest Margins in Pure Pathological Skew: When local client data is almost completely disjoint, the margin of improvement over local training is relatively narrow (e.g., 88.03% vs. 88.01% on CIFAR-10), indicating that collaborative gain is inherently limited in such regimes.
  • Future Directions: The authors suggest exploring generative methods to synthesize public datasets privacy-preservingly or leveraging foundation models to distill public representations.

Summary updated

Abstract

Data heterogeneity poses a fundamental challenge in federated learning (FL), especially when clients differ not only in distribution but also in the reliability of their predictions across individual examples. While personalized FL (PFL) aims to address this, we observe that many PFL methods fail to outperform two necessary baselines, local training and centralized training. This suggests that meaningful personalization only emerges in a narrow regime, where global models are insufficient, but collaboration across clients still holds value. Our empirical findings point to two key ingredients for success in this regime: adaptivity in collaboration and fine-grained trust, at the level of individual examples. We show that these properties can be achieved within federated semi-supervised learning, where clients exchange predictions over a shared unlabeled dataset. This enables each client to align with public consensus when it is helpful, and disregard it when it is not, without sharing model parameters or raw data. As a concrete realization of this idea, we develop FEDMOSAIC, a personalized co-training method where clients reweight their loss and their contribution to pseudo-labels based on per-example agreement and confidence. FEDMOSAIC consistently outperforms strong FL and PFL baselines across a range of non-IID settings, and we prove convergence under standard smoothness, bounded-variance, and drift assumptions. In contrast to many of these baselines, it also consistently outperforms local and centralized training. These results clarify when federated personalization can be effective, and how fine-grained, trust-aware collaboration enables it.
BibTeX
@misc{abourayya2025whom,
  author = {Amr Abourayya and Jens Kleesiek and Bharat Rao and Michael Kamp},
  title = {{Whom to Trust? Adaptive Collaboration in Personalized Federated Learning}},
  year = {2025},
  eprint = {2507.00259},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.00259},
}
Figure