TL;DR
Starting Federated Learning (FL) from a pre-trained model instead of random weights drastically improves final accuracy (by up to 40%) and speeds up convergence. More importantly, pre-training acts as a powerful "heterogeneity buffer," significantly reducing the negative impact of non-IID data and making complex optimization methods designed for heterogeneity less necessary.
What Problem Does This Paper Address?
Federated Learning faces two major hurdles: data heterogeneity (clients have very different data distributions) and system heterogeneity (clients have varying hardware capabilities). These issues often lead to "client drift," where local updates pull the global model in conflicting directions, slowing down or even preventing convergence.
While researchers have developed numerous specialized optimizers to combat these issues, most empirical evaluations start training from scratch (random initialization). However, in many real-world scenarios, a server might have access to public "proxy" data to pre-train a model before starting the federated process. This paper asks: Does where we start training change how FL algorithms behave and how much heterogeneity matters?
Methods
The authors conducted an extensive empirical study comparing 15 variations of federated optimization methods across four benchmark datasets. They utilized the FedOpt framework, which allows for different combinations of client-side and server-side optimizers:
- Client Optimizers: Standard SGD, Proximal (FedProx), Normalized Averaging (FedNova), and MimeLite.
- Server Optimizers: SGD (FedAvg), SGD with Momentum (FedAvgM), and Adam (FedAdam).
They compared two initialization strategies:
- Random Initialization: The standard research baseline.
- Supervised Pre-training: Using ImageNet for computer vision tasks and WikiText-103 or DistilGPT2 for natural language processing tasks.
Datasets
The study used four standard FL benchmarks, covering both vision and text:
- CIFAR-10: Artificially partitioned using a Dirichlet distribution to simulate varying levels of non-IID data.
- FEMNIST: A naturally heterogeneous dataset of handwritten characters partitioned by the writer.
- Stack Overflow: A large-scale language modeling task based on user questions and answers.
- Reddit: Next-word prediction using comments from the social network.
Key Insights and Ideas
- The Ranking Shift: The "best" algorithm changes depending on initialization. While specialized optimizers like FedProx might help when starting from scratch, their advantage often disappears with pre-training. In the pre-trained setting, FedAdam (server-side) with simple SGD (client-side) emerged as a consistently strong and simple performer.
- Heterogeneity Mitigation: Pre-training significantly closes the performance gap between IID (identically distributed) and non-IID data. It essentially "pre-aligns" the model, so client updates are more similar to each other from the start.
- Smoother Landscapes: By examining the Hessian matrix, the authors found that pre-trained models start in regions of the loss landscape with much smaller eigenvalues (i.e., the surface is "smoother" or less curved). This makes the optimization problem fundamentally easier.
- Communication Efficiency: With pre-training, you can perform more local training epochs per round without degrading the final model quality. This is a huge win for cross-device FL, where communication is often the primary bottleneck.
Key Results
- Accuracy Boost: Pre-training enabled models to reach up to 40% higher accuracy than those started from random initialization on the same number of communication rounds.
- Faster Convergence: Pre-trained models reached target error rates in significantly fewer rounds.
- Client Alignment: The "gradient diversity" (a measure of how much client updates differ) was significantly lower in pre-trained models. Consecutive global updates also pointed more consistently in the same direction, indicating a more stable path to the optimum.
- System Robustness: In simulations of system heterogeneity (where clients perform different amounts of work), FedAdam with pre-training consistently outperformed methods specifically designed to handle varying local updates (like FedNova).
Limitations and Future Work
- Proxy Data Availability: The biggest limitation is the assumption that relevant public proxy data exists. In some highly niche or sensitive domains, this might not be the case.
- Pre-training Bias: Pre-training on public data might introduce biases if the public distribution differs significantly from the private user data. The authors suggest further study into detecting and mitigating such biases.
- Alternative Warm-starts: This study focused on supervised pre-training. Future work could explore the impact of self-supervised pre-training or meta-learning as alternative ways to find a good starting point for federated optimization.
- Theoretical Gaps: The authors noted that existing FL convergence theory doesn't fully explain their findings—for instance, pre-training helps even when the initial loss isn't lower than random initialization. This suggests a need for new theoretical frameworks that account for the local geometry of the loss landscape.
Summary updated