Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning

John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, Michael Rabbat

TL;DR

Starting Federated Learning (FL) from a pre-trained model instead of random weights drastically improves final accuracy (by up to 40%) and speeds up convergence. More importantly, pre-training acts as a powerful "heterogeneity buffer," significantly reducing the negative impact of non-IID data and making complex optimization methods designed for heterogeneity less necessary.

What Problem Does This Paper Address?

Federated Learning faces two major hurdles: data heterogeneity (clients have very different data distributions) and system heterogeneity (clients have varying hardware capabilities). These issues often lead to "client drift," where local updates pull the global model in conflicting directions, slowing down or even preventing convergence.

While researchers have developed numerous specialized optimizers to combat these issues, most empirical evaluations start training from scratch (random initialization). However, in many real-world scenarios, a server might have access to public "proxy" data to pre-train a model before starting the federated process. This paper asks: Does where we start training change how FL algorithms behave and how much heterogeneity matters?

Methods

The authors conducted an extensive empirical study comparing 15 variations of federated optimization methods across four benchmark datasets. They utilized the FedOpt framework, which allows for different combinations of client-side and server-side optimizers:

  • Client Optimizers: Standard SGD, Proximal (FedProx), Normalized Averaging (FedNova), and MimeLite.
  • Server Optimizers: SGD (FedAvg), SGD with Momentum (FedAvgM), and Adam (FedAdam).

They compared two initialization strategies:

  1. Random Initialization: The standard research baseline.
  2. Supervised Pre-training: Using ImageNet for computer vision tasks and WikiText-103 or DistilGPT2 for natural language processing tasks.

Datasets

The study used four standard FL benchmarks, covering both vision and text:

  • CIFAR-10: Artificially partitioned using a Dirichlet distribution to simulate varying levels of non-IID data.
  • FEMNIST: A naturally heterogeneous dataset of handwritten characters partitioned by the writer.
  • Stack Overflow: A large-scale language modeling task based on user questions and answers.
  • Reddit: Next-word prediction using comments from the social network.

Key Insights and Ideas

  • The Ranking Shift: The "best" algorithm changes depending on initialization. While specialized optimizers like FedProx might help when starting from scratch, their advantage often disappears with pre-training. In the pre-trained setting, FedAdam (server-side) with simple SGD (client-side) emerged as a consistently strong and simple performer.
Figure 1: We tested the accuracy of four datasets using random and pre-trained weights. We used solid lines for SGD, dashed lines for PROXIMAL, and dotted lines for MIMELITE. The graph shows how the algorithm rankings changed between the random and pre-trained models. Although no single method is the best for all tasks, FEDADAM with SGD for CLIENTOPT performed consistently well when starting from a pre-trained model, especially for the larger language modeling tasks of Stack Overflow and Reddit.
  • Heterogeneity Mitigation: Pre-training significantly closes the performance gap between IID (identically distributed) and non-IID data. It essentially "pre-aligns" the model, so client updates are more similar to each other from the start.
  • Smoother Landscapes: By examining the Hessian matrix, the authors found that pre-trained models start in regions of the loss landscape with much smaller eigenvalues (i.e., the surface is "smoother" or less curved). This makes the optimization problem fundamentally easier.
  • Communication Efficiency: With pre-training, you can perform more local training epochs per round without degrading the final model quality. This is a huge win for cross-device FL, where communication is often the primary bottleneck.

Key Results

  • Accuracy Boost: Pre-training enabled models to reach up to 40% higher accuracy than those started from random initialization on the same number of communication rounds.
  • Faster Convergence: Pre-trained models reached target error rates in significantly fewer rounds.
  • Client Alignment: The "gradient diversity" (a measure of how much client updates differ) was significantly lower in pre-trained models. Consecutive global updates also pointed more consistently in the same direction, indicating a more stable path to the optimum.
  • System Robustness: In simulations of system heterogeneity (where clients perform different amounts of work), FedAdam with pre-training consistently outperformed methods specifically designed to handle varying local updates (like FedNova).

Limitations and Future Work

  • Proxy Data Availability: The biggest limitation is the assumption that relevant public proxy data exists. In some highly niche or sensitive domains, this might not be the case.
  • Pre-training Bias: Pre-training on public data might introduce biases if the public distribution differs significantly from the private user data. The authors suggest further study into detecting and mitigating such biases.
  • Alternative Warm-starts: This study focused on supervised pre-training. Future work could explore the impact of self-supervised pre-training or meta-learning as alternative ways to find a good starting point for federated optimization.
  • Theoretical Gaps: The authors noted that existing FL convergence theory doesn't fully explain their findings—for instance, pre-training helps even when the initial loss isn't lower than random initialization. This suggests a need for new theoretical frameworks that account for the local geometry of the loss landscape.

Summary updated

Abstract

An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very different distributions. \emph{System heterogeneity} refers to client devices having different system capabilities. A considerable number of federated optimization methods address this challenge. In the literature, empirical evaluations usually start federated training from random initialization. However, in many practical applications of federated learning, the server has access to proxy data for the training task that can be used to pre-train a model before starting federated training. Using four standard federated learning benchmark datasets, we empirically study the impact of starting from a pre-trained model in federated learning. Unsurprisingly, starting from a pre-trained model reduces the training time required to reach a target error rate and enables the training of more accurate models (up to 40\%) than is possible when starting from random initialization. Surprisingly, we also find that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity. We recommend future work proposing and evaluating federated optimization methods to evaluate the performance when starting from random and pre-trained initializations. This study raises several questions for further work on understanding the role of heterogeneity in federated optimization. \footnote{Our code is available at: \url{https://github.com/facebookresearch/where_to_begin}}
BibTeX
@inproceedings{nguyen2023where,
  author = {John Nguyen and Jianyu Wang and Kshitiz Malik and Maziar Sanjabi and Michael Rabbat},
  title = {{Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=Mpa3tRJFBb},
}
Figure