Communication-Efficient Distributionally Robust Federated Averaging with Multi-Source Data

Yingzhu Liu, Zhongkui Li, Pengcheng You, Ashish CherukuriarXiv 2026 · arXiv

TL;DR

This paper introduces BiDRO-FedAvg, a federated learning framework that jointly handles uncertainty in local client data and shifts in the global client population. By constructing a global "ambiguity set" using Wasserstein metrics, the authors provide a communication-efficient algorithm that remains robust even when the deployment environment looks significantly different from the training setup.

What Problem Does This Paper Address?

Federated Learning (FL) allows a central server to train a model across many clients (like smartphones or hospitals) without ever seeing their raw data. However, two major statistical hurdles often break these models in the real world:

  1. Within-Client Ambiguity: A single client might only have a few data points. If a model overfits to this small sample, it won't generalize well to that client's true underlying distribution.
  2. Cross-Client Mixture Uncertainty: During training, we might have 60% data from "Client A" and 40% from "Client B." But at deployment, the model might be used 90% of the time by "Client B." If the clients are heterogeneous (e.g., different demographics or geographic regions), this "mixture shift" can cause performance to crater.

While previous research has tackled these issues individually, this paper argues that they must be treated jointly. Existing robust methods often struggle with communication efficiency or require all clients to participate in every round—a luxury rarely available in real-world FL.

Methods

The authors propose a framework based on Distributionally Robust Optimization (DRO). Their approach involves several key components:

  • The Global Ambiguity Set: They define a set of "plausible" distributions that is a mixture of local Wasserstein balls. Each ball represents the uncertainty around a specific client's data. This construction allows for client-dependent radii, meaning we can be more "cautious" with clients who have very little data.
  • Tractable Reformulation: Optimization over probability distributions is usually a nightmare. The authors prove that their complex global objective can be decomposed into client-wise subproblems, which is essential for the decentralized nature of FL.
  • BiDRO-FedAvg Algorithm: To make this practical, they develop a penalty-based version of the objective. The algorithm supports:
  • Partial Participation: Only a subset of clients needs to be active in any round.
  • Multiple Local Steps: Clients perform several updates locally before communicating, drastically reducing the communication bottleneck.
  • Snapshotting: A clever trick to update the "mixture weights" (the global importance of each client) without requiring extra communication rounds.

Datasets

The framework is validated using a Federated Market-Demand Prediction Simulation.

  • Setup: Six clients represent different market stages (Emerging, Early-growth, Growth, Mature, Saturated, and Volatile late markets).
  • The Shift: The training environment is "early-market-heavy" (mostly data from emerging markets), while the deployment environment is "late-market-heavy" (mostly data from volatile/mature markets).
  • Features: The model predicts demand based on market maturity, price, and advertising intensity.

Key Insights and Ideas

  • Separability is King: The most significant "aha" moment is the proof that a global ambiguity set—which considers all possible mixtures of all possible local perturbations—can still be solved by having clients work on their own local robust objectives independently.
  • Robustness is a Certificate: The authors provide an "out-of-sample" guarantee. They prove that the value the algorithm optimizes is a mathematically sound upper bound on the actual risk the model will face when deployed on unseen data.
  • The Induced Radius: By using a penalty-based approach (Lagrangian relaxation), the authors show that each client is effectively optimizing against a Wasserstein ball whose radius is "induced" by the penalty parameter. This bridges the gap between easy-to-solve penalized problems and hard-to-solve constrained problems.

Key Results

  • Superior Generalization: In the market-demand simulation, BiDRO-FedAvg achieved a deployment Mean Squared Error (MSE) of 0.0133, significantly outperforming standard FedAvg (0.0321) and even specialized robust methods like AFL (0.0157) and DRFA (0.0146).
  • Communication Efficiency: The advantage of the proposed method was maintained even when clients performed multiple local steps and only a fraction of clients participated per round.
  • Theoretical Convergence: The authors established a convergence rate of $O(T^{-3/8} + \epsilon)$, where $T$ is the number of iterations and $\epsilon$ is the internal approximation error. Experimental results closely tracked this theoretical decay (showing a slope of -0.388 on a log-log plot).

Limitations and Future Work

  • Simulation vs. Real World: While the market simulation is sophisticated, the framework has not yet been tested on massive-scale real-world benchmarks like LEAF or Federated EMNIST.
  • Hyperparameter Sensitivity: The performance depends on choosing the right penalty parameters ($\rho_i$) and ambiguity radii ($\epsilon_i$). Automating the selection of these parameters based on local sample sizes is a logical next step.
  • Computational Cost: Solving the "inner" maximization (the adversarial distribution) at each client adds a computational burden compared to simple SGD, though the authors mitigate this by using efficient KKT-based solvers.

For more details, you can find the full paper under the title "Communication-Efficient Distributionally Robust Federated Averaging with Multi-Source Data" on arXiv (2611.20060v1).

Summary updated

Abstract

Federated learning trains a shared model from distributed client data. The deployment risk is typically specified through two approximations: each unknown local distribution is replaced by its empirical distribution, and the unknown deployment mixture across clients is replaced by empirical or prescribed training weights. Existing robust methods usually address one approximation at a time through local distributional ambiguity sets or worst-case client mixtures. However, joint treatment of both approximations remains underexplored. To address this gap, we construct a global ambiguity set formed by admissible mixtures of client-wise Wasserstein ambiguity sets, yielding a distributionally robust optimization formulation. The construction allows client-dependent ambiguity radii, preserves client-wise separability of the objective function, and yields a high-probability out-of-sample guarantee for the resulting robust problem. We then introduce a penalty-based approximation of this formulation and develop a communication-efficient algorithm called BiDRO-FedAvg, for which we establish a convergence guarantee. We validate the proposed framework through a market-demand prediction simulation.
BibTeX
@misc{liu2026communication,
  author = {Yingzhu Liu and Zhongkui Li and Pengcheng You and Ashish Cherukuri},
  title = {{Communication-Efficient Distributionally Robust Federated Averaging with Multi-Source Data}},
  year = {2026},
  eprint = {2611.20060},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20060},
}
Figure