Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning

John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, Michael Rabbat

TL;DR

This paper demonstrates that starting Federated Learning (FL) from a pre-trained model rather than random initialization drastically accelerates convergence and boosts final accuracy by up to 40%. Crucially, pre-training fundamentally alters optimization dynamics by smoothing the loss landscape, which effectively neutralizes the negative impacts of both data and system heterogeneity and changes which FL optimizers perform best.

What Problem Does This Paper Address?

Federated Learning (FL) allows a central server to train a machine learning model collaboratively across millions of edge devices (like smartphones) without raw data ever leaving those devices. However, FL faces two massive real-world bottlenecks:

  1. Data Heterogeneity: Different users have vastly different habits, meaning their local data distributions are non-IID (not identically and independently distributed). This causes "client drift," where local updates pull the global model in conflicting directions.
  2. System Heterogeneity: Client devices have varying computational power and network bandwidth, meaning some can perform more local training epochs than others in a given timeframe.

To combat these issues, researchers have designed complex, specialized federated optimization algorithms. However, almost all academic benchmarks evaluate these algorithms starting from random initialization. In practice, a server often has access to public proxy data (e.g., ImageNet or public text corpora) that can be used to pre-train a model before federated training begins. This paper investigates a simple but overlooked question: How does starting from a pre-trained model change the behavior and efficacy of federated optimization methods?

Methods

The authors conduct an extensive empirical study comparing 15 variations of federated optimization methods under two initialization strategies: standard random initialization and supervised pre-training.

They utilize the unified FedOpt framework, which decouples the optimization into two parts:

  • Client Optimizer (CLIENTOPT): What the edge devices use locally (e.g., standard SGD, FedProx to penalize drift, FedNova to handle data imbalance, or MIMELITE to track server momentum).
  • Server Optimizer (SERVEROPT): How the server aggregates client updates (e.g., standard FedAvg/SGD, FedAvgM with momentum, or FedAdam for adaptive learning rates).

For pre-training, they use standard public models or datasets: SqueezeNet and ResNet18 pre-trained on ImageNet for vision tasks, and DistilGPT2 and CharLM pre-trained on public text corpora (WikiText-103) for language tasks.

Datasets

The methods were evaluated across four standard FL benchmarks representing both vision and text applications:

  • CIFAR-10: Artificially partitioned across 50 users using a Dirichlet allocation to simulate non-IID data.
  • Federated EMNIST-62 (FEMNIST): Image classification dataset naturally partitioned by the writer of the characters.
  • Stack Overflow: Text dataset consisting of real questions and answers, naturally partitioned by user.
  • Reddit (pushshift.io): Next-word prediction task using a sub-sampled dataset of real Reddit comments partitioned by user.

Key Insights and Ideas

  • The Optimizer Leaderboard Flips: Algorithms explicitly engineered to fight data heterogeneity (like FedProx or FedNova) lose their competitive edge when training starts from a pre-trained model. Instead, a simple combination of local SGD at the client and an adaptive optimizer like FedAdam at the server consistently yields the best results.
  • Pre-training Erases the Non-IID Penalty: Remarkably, starting from a pre-trained model significantly shrinks the performance gap between training on perfectly IID data versus highly heterogeneous non-IID data.
  • Client Drift is Suppressed: When initialized randomly, increasing the number of local epochs causes the model to diverge due to client drift. With pre-training, you can aggressively increase local epochs to save communication rounds without destroying final accuracy.
  • A Smoother Loss Landscape: Why does pre-training help so much? The authors analyzed the Hessian matrix (which measures the curvature of the loss surface). Pre-trained models consistently show a significantly smaller maximum Hessian eigenvalue at initialization. This means the loss surface is much smoother ("well-behaved"), allowing client updates to align more closely (higher cosine similarity) rather than fighting each other.

Key Results

  • Massive Accuracy Boosts: Pre-training enables models to achieve up to 40% higher final accuracy compared to starting from scratch on the same federated data.
  • Faster Convergence: Pre-trained models reach target accuracy thresholds in a fraction of the communication rounds, translating directly to massive savings in network bandwidth.
Figure 1: The test set accuracy on four datasets with random and pre-trained weights. We represent SGD by solid lines, PROXIMAL by dashed lines, and MIMELITE by dotted lines. See Section 3.3 and Table 4 in Appendix B for comparison characteristics of each method.
  • Overcoming System Heterogeneity: In simulations where 30% of clients performed variable local epochs (simulating system lag), FedAdam-SGD starting from a pre-trained model consistently outperformed algorithms specifically designed to handle system heterogeneity.
Pre-training consistently accelerates convergence and achieves higher final accuracy across vision and language tasks.

Limitations and Future Work

  • Proxy Data Availability & Bias: Public data for pre-training is not always available for every niche task. Furthermore, if the proxy data is heavily biased or misaligned with the actual client data distribution, it could introduce unwanted downstream biases.
  • Scope of Pre-training: This study focused exclusively on supervised pre-training. The authors note that exploring self-supervised pre-training (e.g., contrastive learning) or meta-learning approaches remains an open and highly promising direction for future FL research.

Summary updated

Abstract

An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very different distributions. \emph{System heterogeneity} refers to the fact that client devices have different system capabilities. A considerable number of federated optimization methods address this challenge. In the literature, empirical evaluations usually start federated training from random initialization. However, in many practical applications of federated learning, the server has access to proxy data for the training task that can be used to pre-train a model before starting federated training. We empirically study the impact of starting from a pre-trained model in federated learning using four standard federated learning benchmark datasets. Unsurprisingly, starting from a pre-trained model reduces the training time required to reach a target error rate and enables the training of more accurate models (up to 40\%) than is possible when starting from random initialization. Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity. We recommend that future work proposing and evaluating federated optimization methods evaluate the performance when starting from random and pre-trained initializations. We also believe this study raises several questions for further work on understanding the role of heterogeneity in federated optimization.
BibTeX
@inproceedings{nguyen2023where,
  author = {John Nguyen and Jianyu Wang and Kshitiz Malik and Maziar Sanjabi and Michael Rabbat},
  title = {{Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=Mpa3tRJFBb},
}
Figure