TL;DR
FedCDA introduces a novel aggregation strategy for Federated Learning (FL) that breaks the traditional "single-round" mold. Instead of only averaging local models from the current communication round, it selectively pulls from a cache of a client's recent historical models to find the combination that minimizes divergence. This effectively preserves local data knowledge even in the face of extreme statistical heterogeneity.
What Problem Does This Paper Address?
In Federated Learning, a central server aggregates models trained locally by various clients without accessing their private data. The gold standard, FedAvg, performs a simple coordinate-wise weighted average of these models. However, when clients have highly diverse data (non-IID settings), their local models drift toward vastly different local optima.
The authors argue that averaging these divergent models results in a global model that sits in a "no-man's land"—far from any individual client's local optimum. This leads to a global model that fails to retain the specific knowledge learned from any single client's private data. Traditional fixes often involve complex re-ordering of parameters or adaptive weighting, but they still typically limit themselves to the models produced in the most recent communication round.
Methods
The core innovation is Cross-round Divergence-aware Aggregation (FedCDA). The intuition is that because deep neural networks are non-convex, a client might converge to different, equally valid local optima in different rounds depending on the global starting point it receives.
Key components of the FedCDA framework include:
- Model Caching: The server maintains a cache of the $K$ most recent local models for each client.
- Divergence-Aware Selection: Instead of just taking the latest model, the server solves an optimization problem to select one model from each client's cache. The goal is to pick a set of models that are "closest" to each other (minimizing divergence), ensuring the resulting average remains representative of all selected local states.
- Combinatorial Approximation: Finding the absolute optimal combination across $N$ clients with $K$ models each is computationally expensive ($O(K^N)$). To solve this, the authors propose a batch-based selection strategy. They group clients into batches and greedily select models batch-by-batch, reducing complexity to a manageable level.
- Warmup Phase: The algorithm typically starts with standard FedAvg for a set number of rounds to allow models to reach a baseline level of convergence before the cross-round selection kicks in.
Datasets
The authors evaluated FedCDA using three standard computer vision benchmarks:
- Fashion-MNIST: Tested with a simple CNN.
- CIFAR-10: Tested with ResNet-18.
- CIFAR-100: Tested with ResNet-18.
To simulate real-world heterogeneity, they used two partitioning strategies: Shards (assigning a fixed number of classes per client) and Dirichlet distribution (using a parameter $\alpha$ to control the degree of label imbalance).
Key Insights and Ideas
- History Matters: A client's model from two rounds ago might actually be a better "fit" for the current global consensus than its most recent update. By looking back, the server can find a path through the non-convex loss landscape that satisfies more clients simultaneously.
- The Divergence-Knowledge Link: The paper establishes a theoretical link between model divergence and loss. By minimizing the variance (divergence) among the aggregated models, FedCDA effectively minimizes the upper bound of the global loss function.
- Stability over Speed: FedCDA produces much smoother, less oscillatory convergence curves compared to FedAvg. This suggests that cross-round aggregation acts as a stabilizer against the "noise" introduced by non-IID data.
Key Results
FedCDA consistently outperformed state-of-the-art (SOTA) methods like FedProx, FedMA, and FedLAW across almost all settings:
- CIFAR-100 (Dirichlet $\alpha=0.1$): FedCDA achieved 47.38% accuracy, a significant jump over the best baseline, FedDF (41.04%).
- CIFAR-10 (Dirichlet $\alpha=0.1$): It reached 62.46% accuracy compared to FedExP's 60.63%.
- Scalability: The method showed even greater gains as the model complexity and dataset size increased, suggesting it is well-suited for modern DNN architectures.
- Memory Sensitivity: Increasing the cache size ($K$) generally improves performance, as it provides more "options" for the divergence-aware selection process.
Limitations and Future Work
- Equal Smoothness Assumption: The current theoretical proof assumes that the loss functions of all clients share the same smoothness constant ($L$). In reality, different local datasets might result in very different loss landscapes.
- Sharpness Neglect: The authors acknowledge that they currently treat all local models equally. They suggest that future work could incorporate "sharpness-aware" metrics, as flatter local optima tend to generalize better when aggregated.
- Storage Overhead: While the authors argue that large storage drives are common on the edge, caching multiple copies of large models (like ResNets) for hundreds of clients still imposes a non-trivial storage requirement on the server.
Summary updated